Opened 4 weeks ago

Last modified 3 weeks ago

#3254 new help

Slow writing to /gws/nopw/j04/jules

Reported by: dhertwig Owned by: pmcguire
Component: JULES Keywords:
Cc: Platform: JASMIN
UM Version:

Description

Hi,

I noticed that writing JULES output to the new JULES gws on JASMIN (/gws/nopw/j04/jules) is extremely slow (i.e. 1D runs that usually take < 2min now need > 1h). I ran a previously working suite (u-bp611) on both jasmin-cylc and jasmin-sci1. The problem was the same on both. I am not writing in parallel.

I was wondering if anyone experienced this too?

Cheers,
Denise

Change History (3)

comment:1 Changed 4 weeks ago by pmcguire

Hi Denise
Yes, the new jules GWS on JASMIN is possibly much slower. The no-parallel-write (nopw) GWS's don't handle many small files as well as the parallel-write GWS's.

If this is a problem, several alternatives include:
1) using the parallel-write scratch disk space for the output of your (non-parallel or parallel) JULES runs, and then transfering your files after the run to the nopw GWS.
2) using your home directory (there is maybe 100GB available in each user's home directory), and transfering the output from home to the final destination after the run is over.
3) using a parallel-write GWS.

Furthermore, if your runs are very large in output size, then maybe the jules GWS is not the most appropriate place to store them. The jules GWS is of limited size.

I looked into getting more storage space for land surface modelling (i.e., probably a nopw GWS) a couple of months ago, but only 2-3 people (including yourself) expressed interest in that. I will try again soon to find out if more people are interested.

Patrick

comment:2 Changed 3 weeks ago by dhertwig

Hi Patrick,

Thanks for your help! My JULES output is now very large; it's a 1D limited-time run.

The scratch disk you mentioned is just the usual /work/scratch/<username>?

Also thanks for looking into getting more storage for the land-surface modelling. Hopefully more people will provide feedback now.

Cheers,
Denise

comment:3 Changed 3 weeks ago by pmcguire

Hi Denise:
Yes, that's right, the parallel-write scratch space is at /work/scratch/<username>.
There's another no-parallel-write scratch space available as well, if you need extra temporary space for whatever reason (except for parallel-writing tasks).

Your JULES output might not be very large, but maybe there are a lot of small files. This might cause issues with the no-parallel-write storage space.

Or maybe there is some other issue that is causing your problems.
Patrick

Note: See TracTickets for help on using tickets.