Opened 11 years ago

Closed 9 years ago

#306 closed help (fixed)

HadGEM mods and scripts on Hector

Reported by: mjm Owned by: jeff
Component: UM Model Keywords: HECToR
Cc: j.cole@… Platform:
UM Version: 6.1

Description

Hello

We (i.e. Edinburgh group)are starting to use Hector for some HadGEM runs.

There are some mods we will need to make to all our runs - for example to convert files to 32bit PP format, and sometimes to send results to our cluster in Edinburgh. There are also a few scripts that we use on Eddie, our local machine running HadCM3 4.5, taht we would like to put on Hector.

May I ask:

  1. how should we share these files? (Others in n02-clim are welcome to read them should they care!) Is this a question for Hector help?
  1. is there a recommended way to send files more-or-less as they are created to another machine? (eddie.ecdf.ed.ac.uk) Something like scp from the archiving script would suffice, I expect, if that is allowed from a worker node that runs the UM top-level scripts…) - but I'd not want files with passwords to be held anywhere.

Regards
Mike Mineter

Change History (10)

comment:1 Changed 11 years ago by lois

  • Cc j.cole\'reading.ac.uk added

1) Most file sharing is done via the sub-group Mike, so you cud make your files accessible to n02-clim. Otherwise if you want people outside the group to share we could put them in the UM collective userid hum, where all n02 users should be able to access them. Let us know if you want hum to host your scripts.

2) I will leave this part of your query to Jeff Cole, who will be back from leave tomorrow. Other groups have used the UM archiving scripts to send data outside HECToR so this has been done before. Jeff will be able to give you more details.

LOis

comment:2 Changed 11 years ago by lois

  • Cc j.cole@… added; j.cole\'reading.ac.uk removed

comment:3 Changed 11 years ago by mjm

part 1

I want to be able to see Alessio's files,and then to have some scripts we can share with other colleagues. At present we are in different subgroups (or at least we have directories in different subgroup filespaces - Alessio Boszzo (~abozzo) in /work/n02/n02/abozzo and me in /work/n02/n02-clim/mjm . An email from Lois thought we should be able to share files through being in n02-clim. Maybe I need to refer this to Lois to check which subgroups' filespace we should be using? (I'm also in z03, I think for historical reasons…)

If I've missed documentnation on HECToR about subgroups, just tell me and I'll look again… I've got nervous after yesterdays error with home directories!

Part 2 - archiving and on occasions automated transfer of files to Eddie. This is important but not urgent - but it might be easy for you to guide me anyway??

I want to run with something like our archiving on Eddie - to create 32bit pp files in different subdirectories as our primary output on HECToR. On Eddie we use /exports/work/geos_sages/CESD/u5/PUM_Input/vn4.5/mods/scripts/general/um_archive45_eddie_7.mod to do this.

However the addition is that I'd like to automate the sending of files to Eddie, in the pp format. If you have experience we can build on for that, which Lois thought was the case, that would be a help.

comment:4 Changed 11 years ago by lois

I am curious Mike as to why you are using /work/n02/n02-clim/mjm. Most people use
/work/n02/n02/ and this ensures that all users of the NCAS allocation (n02) can share files. You could just move everything to a new /work/n02/n02 space.

Lois

comment:5 Changed 11 years ago by mjm

Hi Lois

I've no recollection of why I've a n02/n02-clim/mjm directory either.

I'll happily use whatever I can to simplify sharing of files and admin.

I can’t create /work/n02/n02/mjm so I assume it has to be done for me - presumably I or more likely NCAS ask Hector support?

Regards
Mike

comment:6 Changed 11 years ago by mjm

We have sorted out (thanks to Lois help) the diverse group memberships - we're all in n02.

May I condense this into two questions:

  1. we need to create 32 bit PP files automatically. As first step we are trying to use the simple post processing archiving scripts. Once this basic copying of files to another directory, and thinning of dump files etc is working, I expect that I can add the ff2pp as we have it on Eddie.

Latest attempt is in job xefse. We are using the mod um_archive61_hector.mu, but not
archfix_… .

Please can you check what needs changing in xefse (PUMA's UMUI) so that this first basic copying and thinning of files is accomplished?

  1. Less urgently, we do want to be able to automatically send files to Eddie (and in future perhasp BADC also). I spoke briefly to Jeff about htis, and understadn it has been done, so knowing what was necessary would be helpful.

comment:7 follow-up: Changed 11 years ago by jeff

  • Owner changed from um_support to jeff
  • Status changed from new to accepted

Hi Mike

In job xefse you need script mod $PUM_MODS61/script_archfix.mu. You will also need source mod $PUM_MODS61/xt4_archive.mf77. In umui window "Post Processing → Main switch + General Questions", The option "The system written in the Hadley Centre" needs to be selected.

Jeff.

comment:8 in reply to: ↑ 7 Changed 11 years ago by abozzo

Hi Jeff,

I've added the script and the source mods and checked the box in the "Post Processing" window, but now I get the following error message when the model run starts:

/work/n02/n02/abozzo/tmp/tmp.nid00007.26251/modscr_xefse/qsexecute[921]: /usr/sbin/mknod: not found [No such file or directory]
qsexecute: mknod fails to create pipe to archiving system

and the run crashes. I might have missed some env setting.

Thanks,
Regards
Alessio

Replying to jeff:

Hi Mike

In job xefse you need script mod $PUM_MODS61/script_archfix.mu. You will also need source mod $PUM_MODS61/xt4_archive.mf77. In umui window "Post Processing → Main switch + General Questions", The option "The system written in the Hadley Centre" needs to be selected.

Jeff.

comment:9 Changed 11 years ago by jeff

Hi

You need to have $PUM_MODS61/um_archive61_hector.mu turned on.

Jeff.

comment:10 Changed 9 years ago by ros

  • Resolution set to fixed
  • Status changed from accepted to closed
Note: See TracTickets for help on using tickets.