Opened 13 days ago

Closed 10 days ago

#2990 closed help (fixed)

Archiving in MASS

Reported by: amenon Owned by: um_support
Component: UM Model Keywords: moo put
Cc: Platform: Monsoon2
UM Version: 10.9

Description

Dear Willie,

I have an ensemble suite in MONSooN which failed to archive the output automatically to MASS. This was the error in the job.err file:

 mkset command-id=647463477 failed: (SSC_TASK_REJECTION) one or more tasks are rejected.
  moose:/devfc/u-bb030: (TSSC_PROJECT_DOES_NOT_EXIST) the requested project does not exist.
mkset: failed (2)

After discussing with Stu I decided to carry on with the suite and to do the archiving later manually. I could make the directory using:

moo mkset --single-copy -p project-cascade moose:/devfc/u-bb030 || true

Currently, I am trying to manually archive it to MASS. This is the moo put command I give to archive one ensemble:

moo put -f -vv -c=umpp /projects/cascade/arame/cylc-run/u-bb030/share/cycle/20160703T0000Z/INCOMPASS/km4p4/ra1t_inc4p4/um/em0/umnsa* moose:/devfc/u-bb030/em0/umnsa*.pp

But I am getting the following message:

put: option local-file-format failed: No known converter or specification file on your $PATH corresponding to: umpp


Please let me know what should I give in place of umpp or how I should change this moo put command.

The suite is /home/d04/arame/roses/u-bb030

Thanks,
Arathy


Change History (6)

comment:1 Changed 13 days ago by willie

Hi Arathy,

If you do

moo ls moose:/devfc/u-bb030

you can see that it is already created and is empty. It is owned by Stuart Webster.

As for archiving, I think you need to use um-convpp for the conversion - I don't know what umpp is. It can be found in $UMDIR/vn10.9/xc40/utilities/ for example.

The command should be something like,

moo put f -vv -c=$UMDIR/vn11.1/xc40/utilities/um-convpp \
     /projects/cascade/arame/cylc-run/u-bb030/share/cycle/20160703T0000Z/INCOMPASS/km4p4/ra1t_inc4p4/um/em0/umnsa* \
     moose:/devfc/u-bb030

(I am not sure about the wild card here, you may need to do it file by file). Notice that only the top level of the MOOSE archive is specified. MOOSE creates the appropriate data structure based on the type of data it is converting (fields file or LBC file etc). See moo help put.

Willie

comment:2 Changed 12 days ago by amenon

Thanks Willie. I am getting the following error when I am trying to run the moo put command

put command-id=772540970 failed: (SSC_TASK_REJECTION) one or more tasks are rejected.
  /projects/cascade/arame/cylc-run/u-bb030/share/cycle/20160703T0000Z/INCOMPASS/km4p4/ra1t_inc4p4/um/em0/umnsaa_pa000 -> moose:/devfc/u-bb030: (TSSC_INVALID_COLLECTION) invalid path to a data collection.
put: failed (2)

Is it because it is owned by Stu? Is there a way I can change the permission?

Thanks,
Arathy

comment:3 Changed 11 days ago by willie

Hi Arathy,

Sorry, I was guessing at what the commands meant. It is much better to read the MOOSE manual https://collab.metoffice.gov.uk/twiki/bin/viewfile/Static/MASS/monsoon_user_guide.html#data-class.

The -c specifies a format, not a conversion tool. It should be umpp as you had originally.

The command should be

moo put -c=umpp umnsa* moose:/devfc/u-abb030/field.pp

but you need to create the directory "field.pp" first. Only a few collection names, as they are called, are allowed - see https://collab.metoffice.gov.uk/twiki/bin/viewfile/Static/MASS/data_classes.html#devfc.

Try that.

Willie

comment:4 Changed 11 days ago by amenon

Hi Willie,

I was in touch with Stu. My archiving app was old, hence it didn't archive the outputs. I replaced the app/archive directory with the contents of https://code.metoffice.gov.uk/trac/roses-u/browser/b/a/6/4/6/trunk/app/archive.

Then Stu suggested me to do "rose suite-run — — hold" and archive each ensemble output one at a time. I did that and then I had the following error when I triggered the archive job for one cycle:

[FAIL] [arch:field.pp/]source=*a_p*: configuration value error: [Errno 2] No such file or directory: '/home/d04/arame/cylc-run/u-bb030/share/cycle/20160703T0000Z/INCOMPASS/km4p4/ra1t_inc4p4/um/em0/*a_p*'
[FAIL] ! moose:/devfc/u-bb030/field.pp/ [compress=None]
2019-08-15T13:48:12Z CRITICAL - failed/EXIT

That is because the output was in /projects/cascade/arame/cylc-run/u-bb030 and not /home/d04/arame/cylc-run/u-bb030

The only command I ran was rose suite-run — —hold and while triggering the archive job, it didn't even get submitted. But now when I checked my output directory in /projects/cascade/arame/cylc-run/u-bb030/ I can't see any of my output. Please help.

Thanks,
Arathy

comment:5 Changed 10 days ago by amenon

Hi Willie,

It seems the output has definitely disappeared. I will be re-running this suite then. So we can close this ticket now.

Thanks,
Arathy

comment:6 Changed 10 days ago by willie

  • Resolution set to fixed
  • Status changed from new to closed

Arathy,
Sorry to hear that. Make sure you set up the archiving and do a short run to check that it works ok.
Willie

Note: See TracTickets for help on using tickets.