Changes between Version 20 and Version 21 of Archer/Transition2020/PPTransfer


Ignore:
Timestamp:
10/06/20 09:48:43 (2 months ago)
Author:
ros
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • Archer/Transition2020/PPTransfer

    v20 v21  
    1 = Configuring pptransfer app and UM suite to pull data on `/work` from JASMIN = 
     1= Configuring pptransfer app and UM suite to push to JASMIN from ARCHER serial nodes = 
    22 
    3 The `pptransfer` task used to run on the ARCHER Data Transfer Node (dtn02) which we no longer have access to from February 19th. The instructions below modify a suite to run the `pptransfer` task on the JASMIN transfer nodes and pull the data across from ARCHER `/work` disk.   
     3The `pptransfer` task used to run on the ARCHER Data Transfer Node (dtn02) which we no longer have access to. The instructions show how to modify a suite to run the `pptransfer` task on the ARCHER serial nodes and push the data across to JASMIN from ARCHER `/work` disk.   
    44 
    55Note:  Instructions cannot cover all possible suite setup combinations so you may need to adjust them accordingly. For example tasks may be named slightly differently or inherit differently.  
     
    1212 
    1313 * In panel ''"JASMIN Transfer"'' change: 
    14     * `transfer_type` to '''Pull''' 
    15     * `remote_host` to '''login.archer.ac.uk''' 
    1614    * Set `gridftp` to '''false''' 
    1715 
    1816In the `~/roses/<SUITEID>/site/archer.rc` file edit the section `[[PPTRANSFER_RESOURCE]]`: 
    1917 
    20  * Remove the line '''pre-script = "module load anaconda"'''. (Note this may appear in other `pptransfer` related sections) 
     18 * Replace the  line `host = dtn02.rdf.ac.uk` with `host = login.archer.ac.uk` 
    2119 
    22  * Replace the  line '''host = dtn02.rdf.ac.uk''' with '''host = jasmin-xfer2.ceda.ac.uk''' 
     20 * Add the line `inherit = HPC_SERIAL` directly above the line beginning `pre-script =` 
    2321 
    24 == Setup required on PUMA and JASMIN == 
     22 * Add these 2 lines:  
     23   {{{ 
     24   [[[job]]] 
     25       execution time limit = PT12H 
     26    }}} 
    2527 
    26 You now need to setup `ssh-agent` to be able to login to `jasmin-xfer2.ceda.ac.uk` and from there to `login.archer.ac.uk`. 
     28The section should now look something like the following (Remember it may not be identical due to suite differences) : 
     29{{{ 
     30    [[PPTRANSFER_RESOURCE]] 
     31        inherit = HPC_SERIAL 
     32        pre-script = """module load anaconda""" 
     33        [[[remote]]] 
     34            host = login.archer.ac.uk 
     35        [[[job]]] 
     36            execution time limit = PT12H 
     37        [[[environment]]] 
     38            UMDIR = ~um 
     39            PLATFORM = linux 
     40}}} 
    2741 
    28 '''PUMA:''' 
     42== Setup required on ARCHER == 
    2943 
    30 These instructions assume you already have an ssh-agent running on PUMA which you would already have needed in order to run a UM suite on ARCHER.  
     44You now need to setup `ssh-agent` on both ARCHER post-processing nodes (espp1 & espp2) to be able to login to `jasmin-xfer2.ceda.ac.uk`. 
    3145 
    32 * Log on to PUMA. 
     461. Login to ARCHER (Note: this needs to be from somewhere other than PUMA) 
    3347 
    34 * Make sure you have a copy of your JASMIN ssh-key in your `~/.ssh` directory.  This is the key you generated when you applied for access to JASMIN. The remainder of the instructions assume it is called `id_dsa_jasmin` please replace this with the name of your key as appropriate. 
     482. From ARCHER login node login to '''espp1''' (You will be prompted for your ARCHER password only) 
    3549 
    36 * Setup `~/.ssh/config` file to login to the jasmin-xfer node. For example:  
     503. Add the following lines to your `~/.bashrc` or `~/.profile` (May also be `~/.bash_profile`): 
     51{{{ 
     52# ssh-agent setup on pp nodes 
     53if [[ `hostname` = esPP00* ]] 
     54then 
     55  . ~/.ssh/ssh-setup 
     56fi 
     57}}} 
     58 
     593. Copy the `~/.ssh/ssh-setup` script. 
     60{{{ 
     61$ cp /work/y07/y07/umshared/um-training/ssh-setup ~/.ssh 
     62}}} 
     63 
     644. Copy the ssh-key you use to access JASMIN to `~/.ssh` directory (e.g. id_rsa_jasmin) 
     65 
     665. Add the following to your `~/.ssh/config` file (create one if it doesn't already exist): 
    3767{{{ 
    3868Host jasmin-xfer2.ceda.ac.uk 
    39 Hostname jasmin-xfer2.ceda.ac.uk 
    40 User <jasmin-username> 
    41 IdentityFile ~/.ssh/id_dsa_jasmin 
     69User <jasmin_username> 
     70IdentityFile ~/.ssh/<jasmin_key> 
    4271ForwardAgent no 
    4372}}} 
    4473 
    45   Replace `<jasmin-username>` with your JASMIN username. 
     74 Where `<jasmin_username>` is your JASMIN username and `<jasmin_key>` is the name of you ssh-key. 
    4675 
    47 * Run `ssh-add ~/.ssh/id_dsa_jasmin` to attach the key to your existing ssh-agent. 
     76 '''Note:''' in order to use `jasmin-xfer2.ceda.ac.uk` you need to have requested access to the High Performance Data Transfer service via the JASMIN accounts portal. 
    4877 
    49 Check that you can now ssh to `jasmin-xfer2.ceda.ac.uk` without prompt for password/passphrase by running: 
     786. Logout of espp1 and then log back in again to start up your ssh-agent. 
    5079 
    51 `ssh jasmin-xfer2.ceda.ac.uk` 
     807. Run `ssh-add ~/.ssh/<jasmin_key>` where <jasmin_key> is the name of your JASMIN ssh-key E.g. id_rsa_jasmin. (This is the key you generated when you applied for access to JASMIN). Type in your passphrase when prompted to do so. 
    5281 
    53 '''JASMIN:''' 
     828. You should now be able to login to the required JASMIN transfer node (either `jasmin-xfer1.ceda.ac.uk` or the high performance node `jasmin-xfer2.ceda.ac.uk`) without being prompted for passphrase/password. 
    5483 
    55 1. Log on to JASMIN: `ssh jasmin-xfer2.ceda.ac.uk`  
     849. Start up ssh-agent on the other post-processing node. Login to '''espp2''' to start up ssh-agent. 
    5685 
    57 2. Setup PATH to Rose, Cylc and FCM:''' 
     8610. Run `ssh-add ~/.ssh/<jasmin_key>` where <jasmin_key> is the name of your JASMIN ssh-key 
    5887 
    59 * In your `~/.bash_profile` add the following: 
    60 {{{ 
    61 if [[ `hostname` = "jasmin-cylc.ceda.ac.uk" ]]; then 
    62   # Rose/cylc on jasmin-cylc 
    63   export PATH=/apps/contrib/metomi/bin:$PATH 
    64 else 
    65   # Rose/cylc on jasmin-xfer 
    66   export PATH=/home/users/rshatcher/software/bin:$PATH  
    67 fi 
    68 }}} 
    69  
    70 3. Setup ssh-agent to access ARCHER Login Nodes: 
    71  
    72 * Copy the private key you use on PUMA to access ARCHER over to JASMIN. This will be something like `~/.ssh/id_dsa`. 
    73 {{{ 
    74 scp <puma-username>@puma.nerc.ac.uk:~/.ssh/id_dsa ~/.ssh 
    75 }}} 
    76  
    77 * Copy the ssh-agent start up script: 
    78 {{{ 
    79 cp ~rshatcher/setup ~/.ssh 
    80 }}} 
    81  
    82 * In your `~/.bashrc` add the following to start up ssh-agent: 
    83 {{{ 
    84 if [[ `hostname` = "jasmin-xfer2.ceda.ac.uk" ]] ; then 
    85   # ssh-agent setup for transfer app 
    86   . ~/.ssh/setup 
    87 fi 
    88 }}} 
    89  
    90 * In your `~/.ssh/config` add the following lines: 
    91 {{{ 
    92 Host login.archer.ac.uk 
    93   Hostname login.archer.ac.uk 
    94   User <archer-username> 
    95 }}} 
    96  
    97   Replacing `<archer-username>` with your ARCHER username. 
    98  
    99 * Log out of `jasmin-xfer2.ceda.ac.uk` and back in again. 
    100  
    101 * Run `ssh-add`. You will be prompted for your passphrase. 
    102  
    103 Check that you can now ssh to ARCHER without prompt for password/passphrase by running:  
    104  
    105  `ssh login.archer.ac.uk` 
     8811. Again you should now be able to login to the JASMIN transfer node without being prompted for passphrase/password. 
    10689 
    10790== Updating a Running Suite ==