Changes between Version 23 and Version 24 of Docs/PostProcessingAppArcher2Setup


Ignore:
Timestamp:
15/01/21 15:34:57 (3 months ago)
Author:
ros
Comment:

Update for ARCHER2

Legend:

Unmodified
Added
Removed
Modified

  • Docs/PostProcessingAppArcher2Setup

    v23 v24  
    1 = ARCHER specific setup instructions for data transfer to JASMIN = 
     1= ARCHER2 specific setup instructions for data transfer to JASMIN = 
    22 
    33Instructions depend on whether your suite has the PP Transfer option already available and also if this is your first time using the Transfer App. 
    … …  
    1010 
    1111 * [#addpptransfertask Add PP Transfer task to the suite] 
    12  * [#sshpumatodtn Setup ssh-key to connect from PUMA to ARCHER Data Transfer Node] 
    13  * [#sshdtntojasmin Setup ssh-key to connect from ARCHER Data Transfer Node to JASMIN] 
     12 * [#ssharcher2tojasmin Setup ssh-key to connect from ARCHER2 to JASMIN] 
    1413 
    15 === Scenario 2: PP Transfer already in suite and you have never used the Transfer app before. === 
     14=== Scenario 2: PP Transfer already in suite and you have either never used the Transfer app before or it is configured for ARCHER. === 
    1615 
    17  * [#sshpumatodtn Setup ssh-key to connect from PUMA to ARCHER Data Transfer Node] 
    18  * [#sshdtntojasmin Setup ssh-key to connect from ARCHER Data Transfer Node to JASMIN] 
     16 * [#ssharcher2tojasmin Setup ssh-key to connect from ARCHER2 to JASMIN] 
    1917 
    2018 
    … …  
    9492    [[postproc]] 
    9593        inherit = ... 
    96         pre-script = "module load nco/4.5.0; module load anaconda/2.2.0-python2;  module list 2> &1; export PYTHONPATH=$PYTHONPATH:$UMDIR/lib/python2.7; ulimit -s unlimited" 
     94        pre-script = "module restore /work/y07/shared/umshared/modulefiles/postproc/2020.12.11; module list 2>&1; ulimit -s unlimited" 
    9795        ... 
    98 }}} 
    99  
    100  Note: If your are running the coupled model and using netcdf module `cray-netcdf/4.4.1.1` then the pre-script line will need to be: 
    101  
    102 {{{ 
    103         pre-script = "module load nco/4.6.8; module load anaconda; module list 2>&1; export PYTHONPATH=$PYTHONPATH:$UMDIR/lib/python2.7; ulimit -s unlimited" 
    10496}}} 
    10597 
    … …  
    10799{{{ 
    108100    [[PPTRANSFER]] 
    109         [[[remote]]] 
    110             host = dtn02.rdf.ac.uk 
     101        inherit = HPC_SERIAL 
     102        [[[job]]] 
     103            batch system = background 
    111104        [[[environment]]] 
    112105            UMDIR=~um 
    … …  
    124117    [[pptransfer]] 
    125118        inherit = PPTRANSFER 
    126         pre-script = "module load anaconda" 
     119        pre-script = "module load cray-python" 
    127120        [[[environment]]] 
    128121            CYCLEPERIOD = $( rose date $CYLC_TASK_CYCLE_POINT $CYLC_TASK_CYCLE_POINT --calendar {{CALENDAR}} --offset2 {{FMT}} -f y,m,d,h,M,s ) 
    … …  
    133126'''Note:''' `LINUX_UM` may be called something different (e.g `EXTRACT_RESOURCE`) depending on the suite. 
    134127 
    135 == Set up ssh-key to connect from PUMA to Data Transfer Node == #sshpumatodtn 
     128== Set up ssh-key to connect from ARCHER2 to JASMIN == #ssharcher2tojasmin 
    136129 
    137 These instructions assume you already have an ssh-agent running on PUMA which you would already have needed in order to run a UM suite on ARCHER.  If this is not the case then please [http://cms.ncas.ac.uk/documents/training/November2016/UM_practicals/getting-setup.html#set-up-an-ssh-connection-from-puma-to-archer setup ssh-agent from PUMA to ARCHER] before continuing. 
     130This section setups up connection from ARCHER2 login node to JASMIN. 
    138131 
    139 1. Login to dtn02.rdf.ac.uk.  Assuming you have not already setup ssh-agent to access the Data Transfer Nodes you will be prompted for your ARCHER password.  If you are logged on with no password/passphrase prompt then you are already setup and can move onto the next section "Set up ssh-key to connect from Data Transfer Node to JASMIN" below. 
    140  
    141 2. Create `.ssh` directory: 
    142 {{{ 
    143 $ mkdir ~/.ssh 
    144 }}} 
    145  
    146 3. Logout of dtn02. 
    147  
    148 4. Login to ARCHER: `ssh <username>@login.archer.ac.uk` 
    149  
    150 5. Copy your `authorized_keys` file from ARCHER home disk to `/nerc` disk: 
    151 {{{ 
    152 $ cat ~/.ssh/authorized_keys >> /nerc/n02/n02/<username>/.ssh/authorized_keys 
    153 }}} 
    154  where `<username>` is your ARCHER username. 
    155  
    156 6. Logout of ARCHER. 
    157  
    158 7. Add the following to your `~/.ssh/config` file on PUMA: 
    159 {{{ 
    160 Host dtn02.rdf.ac.uk 
    161 User <username> 
    162 ForwardAgent no 
    163 }}} 
    164  where `<username>` is your ARCHER username. 
    165  
    166 8. Login to dtn02 from PUMA: `ssh dtn02.rdf.ac.uk`.  \\ 
    167  You should now be logged straight in to dtn02 without prompt for password/passphrase. 
    168  
    169 == Set up ssh-key to connect from Data Transfer Node to JASMIN == #sshdtntojasmin 
    170  
    171 This section setups up connection from Archer Data Transfer Node (dtn02) to JASMIN as well as setting up your environment for Rose & Cylc to work. 
    172  
    173 1. Login to dtn02.rdf.ac.uk. 
     1321. Login to login.archer2.ac.uk (Note: this needs to be from somewhere other than PUMA)  
    174133 
    1751342. Add the following lines to your profile.  This will be either `~/.profile` or `~/.bash_profile`. (Create one if it doesn't already exist): 
    176135{{{ 
    177 export PATH=/general/y07/umshared/software/bin:$PATH 
    178  
    179136# ssh-agent setup 
    180137. ~/.ssh/ssh-setup 
    … …  
    1831403. Copy the `~/.ssh/ssh-setup` script. 
    184141{{{ 
    185 $ cp /nerc/n02/n02/ros/software/bin/ssh-setup ~/.ssh 
     142$ cp /work/y07/shared/umshared/um-training/ssh-setup ~/.ssh 
    186143}}} 
    187144 
    188 4. Copy the ssh-key you use to access JASMIN to `~/.ssh` directory. 
     1454. Copy the ssh-key you use to access JASMIN to `~/.ssh` directory (e.g. id_rsa_jasmin). 
    189146 
    1901475. Add the following to your `~/.ssh/config` file (create one if it doesn't already exist): 
    … …  
    194151IdentityFile ~/.ssh/<jasmin_key> 
    195152ForwardAgent no 
    196  
    197 Host * 
    198 ForwardAgent no 
    199 ForwardX11 yes 
    200 ForwardX11Trusted yes 
    201153}}} 
    202154 
    … …  
    205157 '''Note:''' in order to use `hpxfer1.jasmin.ac.uk` you need to have requested access to the High Performance Data Transfer service via the JASMIN accounts portal. 
    206158 
    207 6. Logout of dtn02 and then log back in again to start up your ssh-agent. 
     1596. Logout of ARCHER2 and then log back in again to start up your ssh-agent. 
    208160 
    2091617. Run `ssh-add ~/.ssh/<jasmin_key>` where <jasmin_key> is the name of your JASMIN ssh-key E.g. id_rsa_jasmin. (This is the key you generated when you applied for access to JASMIN). Type in your passphrase when prompted to do so.