MASS @ JASMIN - Problem using SELECT and wildcards

[dan]

Hi All,

I'm attempting to extract some PP data from MASS on JASMIN using the SELECT option.
Previously this has worked fine. But I have now encountered an error:

[dlrhodso@mass-cli1 data]$ moo select N216_query_mslp moose:/crum/u-aq281/apm.pp N216_ATM/mslp/u-aq281/
select command-id=731076270 failed: (SSC_TASK_REJECTION) one or more tasks are rejected.

moose:/crum/u-aq281/apm.pp → N216_ATM/mslp/u-aq281/: (TSSC_SPANS_TOO_MANY_RESOURCES) retrieval request exceeds the maximum number of tape resources.

Tape count: 63, limit: 50.

select: failed (2)

In an attempt to limit the requests and avoid this error, I have been trying to use wildcards, as documented in the wiki (

​http://gws-access.ceda.ac.uk/public/mohc_shared/moose-user-doc/external_user_guide.html)

moo select N216_query_mslp moose:/crum/u-aq281/apm.pp/aq281a.pm1968* N216_ATM/mslp/u-aq281/

But this gives me:

select command-id=731077227 failed: (SSC_TASK_REJECTION) one or more tasks are rejected.

moose:/crum/u-aq281/apm.pp/aq281a.pm1968* → N216_ATM/mslp/u-aq281/: (TSSC_INVALID_SELECTION_TARGET) metadata querying commands must specify appropriate files, collections or parents thereof, and must specify a suitable file type if it cannot be inferred.

select: failed (2)
[
if i specify PP using:

moo select -T=pp N216_query_mslp moose:/crum/u-aq281/apm.pp/aq281a.pm1968* N216_ATM/mslp/u-aq281/

I get the same error.
N216_query_mslp is simply:

begin

lbproc=128
stash=16222

end

Have you any idea what I might be doing wrong?

It seems that the wildcards are not working as expected. I thought that this might be because the wildcards are somehow expanded before they are passed to moo in the wrapper script, but I'm not sure that's the case now.

I can go and ask JASMIN, but I thought you might be better acquainted with MOO!

Cheers,

Dan

[grenville]

Dan

Are you sure there is a file with aq281a.pm1968 in its name? Does aq281a.pm* fail too?

(not that I'm MOO acquainted)

Grenville

[luke]

I've only ever used moo select on directories within MASS, not separate files.

This webpage might be informative as it seems to give quite detailed information: ​https://collab.metoffice.gov.uk/twiki/bin/viewfile/Static/MASS/monsoon_user_guide.html#select-performs-filtered-retrieval-from-a-data-set

I would suggest having a select file (e.g. foo.stash) that looks like:

begin
    lbproc=128
    stash=16222
    year=1968
end

and then using the following command to just extract the 1968 values:

moo select -v foo.stash moose:/crum/u-aq281/apm.pp ./N216_ATM/mslp/u-aq281/apm.pp

as you also need to extract to a directory.

You can also specify a range of years like so:

    year=[1968..1970]

etc.

Thanks,
Luke

[dan]

Replying to grenville:

Dan

Are you sure there is a file with aq281a.pm1968 in its name? Does aq281a.pm* fail too?

(not that I'm MOO acquainted)

Grenville

Yes - these files exist as:

moo ls moose:/crum/u-aq281/apm.pp/aq281a.pm1968*
moose:/crum/u-aq281/apm.pp/aq281a.pm1968apr.pp
moose:/crum/u-aq281/apm.pp/aq281a.pm1968aug.pp
moose:/crum/u-aq281/apm.pp/aq281a.pm1968dec.pp
moose:/crum/u-aq281/apm.pp/aq281a.pm1968feb.pp
moose:/crum/u-aq281/apm.pp/aq281a.pm1968jan.pp
moose:/crum/u-aq281/apm.pp/aq281a.pm1968jul.pp
moose:/crum/u-aq281/apm.pp/aq281a.pm1968jun.pp
moose:/crum/u-aq281/apm.pp/aq281a.pm1968mar.pp
moose:/crum/u-aq281/apm.pp/aq281a.pm1968may.pp
moose:/crum/u-aq281/apm.pp/aq281a.pm1968nov.pp
moose:/crum/u-aq281/apm.pp/aq281a.pm1968oct.pp
moose:/crum/u-aq281/apm.pp/aq281a.pm1968sep.pp

but
moo select N216_query_mslp moose:/crum/u-aq281/apm.pp/aq281a.pm1968* N216_ATM/mslp/u-aq281/

fails!
I must be doing something else wrong!

[dan]

Replying to luke:

I would suggest having a select file (e.g. foo.stash) that looks like:

begin
    lbproc=128
    stash=16222
    year=1968
end

and then using the following command to just extract the 1968 values:

moo select -v foo.stash moose:/crum/u-aq281/apm.pp ./N216_ATM/mslp/u-aq281/apm.pp

as you also need to extract to a directory.

You can also specify a range of years like so:

    year=[1968..1970]

etc.

Excellent - I hadn't thought of subsetting the files by dates via query file - this works!!

Thanks, Luke - saved me a frustrating morning!

Cheers,

Dan

[luke]

Great! I'm glad that works.

Thanks,
Luke