Opened 9 months ago

Last modified 8 months ago

#3460 new help

Updating suite u-bt558 for SLURM libraries

Reported by: NoelClancy Owned by: jules_support
Component: JULES Keywords: JULES, SLURM, JASMIN
Cc: Platform: JASMIN
UM Version:

Description

The global gridded suite I'm trying to get working on JASMIN is u-bt558 which I ran successfully before the SLURM libraries were updated in 2020.

However, I want to update it for the vn5.4 version of JULES

JULES_FCM='/home/users/nmc/jules/vn5.4'
JULES_REVISION='vn5.4'

Attachments (3)

u-cb899-err2.txt (3.4 KB) - added by NoelClancy 9 months ago.
u-cb899-out2.txt (277.0 KB) - added by NoelClancy 9 months ago.
u-cc033_decade_wait_times.txt (4.7 KB) - added by NoelClancy 9 months ago.

Download all attachments as: .zip

Change History (48)

comment:1 Changed 9 months ago by NoelClancy

In the rose-suite.conf do I need to change

ACCOUNT_USR='climate'
to
!!ACCOUNT_USR='climate'

BASERUN_ID_STEM='JULES-GL7.0.vn5.3.CRUNCEPv7SLURM'
to
BASERUN_ID_STEM='JULES-ES.1p0.vn5.4.50.CRUJRA2.TRENDYv8.365'

FUNDING='hccp'
to
!!FUNDING='hccp'

HPC_QUEUE='normal'
to
!!HPC_QUEUE='normal'

JULES_FCM='fcm:jules.x_tr'
to
JULES_FCM='/home/users/nmc/jules/vn5.4'
(NOT SURE ABOUT THIS)

JULES_REVISION='vn5.4'
to
JULES_REVISION='13249'
(NOT SURE ABOUT THIS)

MODE_RUN='JASMIN'
to
MODE_RUN='jasmin-lotus'

MPI_NUM_TASKS=16
to
MPI_NUM_TASKS=10

OMP_NUM_THREADS=2
to
OMP_NUM_THREADS=1

SUBPROJECT='esms'
to
!!SUBPROJECT='esms'

WALLTIME_RUN='PT48H'
WALLTIME_SPINUP='PT48H'
to
WALLTIME_RUN='PT3H'
WALLTIME_SPINUP='PT3H'
(I recall making these changes were necessary in early 2020)

comment:3 Changed 9 months ago by pmcguire

  • Component changed from JASMIN to JULES
  • Keywords JULES, SLURM, JASMIN added
  • Owner changed from um_support to jules_support
  • Platform set to JASMIN

comment:4 Changed 9 months ago by pmcguire

Hi Noel:
Have you looked at the other GL7 + SLURM tickets that are on the NCAS CMS Helpdesk?
Patrick

comment:5 Changed 9 months ago by NoelClancy

I was looking at #3376 last night, but maybe there are better tickets, thanks for the direction. I am reading through them now.

comment:6 Changed 9 months ago by pmcguire

Hi Noel:
Yes #3376 is a good one. That should help.
Patrick

comment:7 Changed 9 months ago by NoelClancy

The following link was very useful
https://code.metoffice.gov.uk/trac/roses-u/changeset?reponame=&new=171494%40b%2Fx%2F7%2F2%2F3%2Ftrunk&old=171386%40b%2Fx%2F7%2F2%2F2%2Ftrunk
but this was done to change from u-bx722 to u-bx723.

However, I'm editing u-bt558 which is an earlier version of GL7 than u-bx722. I'm a little unsure if I need to make more changes such as ….

MODE_RUN='JASMIN'
to
MODE_RUN='jasmin-lotus'

I assume that I do need these changes also

comment:8 Changed 9 months ago by pmcguire

Hi Noel:
You should be able to use just the changes from u-bx722 to u-bx723, and implement those changes in your u-bt558 suite, in order to run with SLURM on JASMIN.
Patrick

comment:9 Changed 9 months ago by NoelClancy

ok thanks,

I have tried running with just the changes from u-bx722 the following .err message

Environment variables set for netCDF Fortran bindings in

/apps/libs/netCDF/intel14/fortran/4.2/

You will also need to link your code to a compatible netCDF C library in

/apps/libs/netCDF/intel14/4.3.2/

[FAIL] mpif90 -oo/logging_mod.o -c -DSCMA -DBL_DIAG_HACK -DINTEL_FORTRAN -I./include -I/home/users/siwilson/netcdf.openmpi/include -heap-arrays -fp-model precise -traceback /home/users/nmc/cylc-run/u-bt558/share/fcm_make/preprocess/src/jules/src/util/logging_mod.F90 # rc=1
[FAIL] ifort: command line warning #10212: -fp-model precise evaluates in source precision with Fortran.
[FAIL] /home/users/nmc/cylc-run/u-bt558/share/fcm_make/preprocess/src/jules/src/util/logging_mod.F90(21): error #7012: The module file cannot be read. Its format requires a more recent F90 compiler. [MPI]
[FAIL] USE mpi
[FAIL] ——
[FAIL] /home/users/nmc/cylc-run/u-bt558/share/fcm_make/preprocess/src/jules/src/util/logging_mod.F90(23): error #6580: Name in only-list does not exist. [OUTPUT_UNIT]
[FAIL] USE iso_fortran_env, ONLY: output_unit, error_unit
[FAIL] —————————————-

[FAIL] /home/users/nmc/cylc-run/u-bt558/share/fcm_make/preprocess/src/jules/src/util/logging_mod.F90(23): error #6580: Name in only-list does not exist. [ERROR_UNIT]
[FAIL] USE iso_fortran_env, ONLY: output_unit, error_unit
[FAIL] ————————————————————
[FAIL] /home/users/nmc/cylc-run/u-bt558/share/fcm_make/preprocess/src/jules/src/util/logging_mod.F90(129): error #6404: This name does not have a type, and must have an explicit type. [MPI_COMM_WORLD]
[FAIL] CALL mpi_comm_size(mpi_comm_world, ntasks, error)
[FAIL] ——————————-

[FAIL] /home/users/nmc/cylc-run/u-bt558/share/fcm_make/preprocess/src/jules/src/util/logging_mod.F90(146): error #6404: This name does not have a type, and must have an explicit type. [OUTPUT_UNIT]
[FAIL] WRITE(output_unit, "(A)") TRIM(full_message)
[FAIL] —————
[FAIL] /home/users/nmc/cylc-run/u-bt558/share/fcm_make/preprocess/src/jules/src/util/logging_mod.F90(149): error #6404: This name does not have a type, and must have an explicit type. [ERROR_UNIT]
[FAIL] WRITE(error_unit, "(A)") TRIM(full_message)
[FAIL] —————

[FAIL] compilation aborted for /home/users/nmc/cylc-run/u-bt558/share/fcm_make/preprocess/src/jules/src/util/logging_mod.F90 (code 1)
[FAIL] compile 0.4 ! logging_mod.o ← jules/src/util/logging_mod.F90
[FAIL] compile —— ! driver_ncdf_mod.o ← jules/src/io/file_handling/core/drivers/ncdf/driver_ncdf_mod.F90
[FAIL] compile —— ! dictionary_mod.o ← jules/src/util/dictionary_mod.F90
[FAIL] compile —— ! data_cube_mod.o ← jules/src/util/cube/data_cube_mod.F90
[FAIL] compile —— ! datetime_mod.o ← jules/src/util/datetime_mod.F90
[FAIL] ! data_cube_mod.mod : depends on failed target: data_cube_mod.o
[FAIL] ! data_cube_mod.o : depends on failed target: logging_mod.mod
[FAIL] ! datetime_mod.mod : depends on failed target: datetime_mod.o
[FAIL] ! datetime_mod.o : depends on failed target: logging_mod.mod
[FAIL] ! dictionary_mod.mod : depends on failed target: dictionary_mod.o
[FAIL] ! dictionary_mod.o : depends on failed target: logging_mod.mod
[FAIL] ! driver_ncdf_mod.mod : depends on failed target: driver_ncdf_mod.o
[FAIL] ! driver_ncdf_mod.o : depends on failed target: logging_mod.mod
[FAIL] ! logging_mod.mod : depends on failed target: logging_mod.o
[FAIL] ! logging_mod.o : update task failed

[FAIL] fcm make -f /work/scratch-pw/nmc/cylc-run/u-bt558/work/17000101T0000Z/fcm_make/fcm-make.cfg -C /home/users/nmc/cylc-run/u-bt558/share/fcm_make -j 4 # return-code=255
2021-02-03T11:46:18Z CRITICAL - failed/EXIT

comment:10 Changed 9 months ago by pmcguire

Hi Noel:
Have you tried to run u-bx723? That's a global gridded GL7 suite for SLURM. Maybe if that works, you can get the u-bt558 suite to work.
Patrick

comment:11 Changed 9 months ago by NoelClancy

Thank you, I'm running a copy of u-bx723. It passes fcm_make and RECON. SPINUP running now.

comment:12 Changed 9 months ago by NoelClancy

I've tried running copies of u-bx273 (u-cb899 and u-cc033) but without succuess.
u-cc033 runs through fcm_make, RECON and SPINUP but fails during the MAIN_RUN.

The error messages are as follows:

.err
cpu-bind=MASK - host091, task 0 0 [30230]: mask 0x104 set
[host095.jc.rl.ac.uk:11172] OPAL ERROR: Error in file pmix2x.c at line 326
[host095.jc.rl.ac.uk:11172] OPAL ERROR: Error in file pmix2x.c at line 326
[host095.jc.rl.ac.uk:11172] OPAL ERROR: Error in file pmix2x.c at line 326
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
jules.exe 00000000007D4AE3 Unknown Unknown Unknown
libpthread-2.17.s 00007F29DA27C630 Unknown Unknown Unknown
mca_pmix_pmix2x.s 00007F295137F3A7 pmix2x_value_unlo Unknown Unknown
mca_pmix_pmix2x.s 00007F295137F00F pmix2x_event_hdlr Unknown Unknown
mca_pmix_pmix2x.s 00007F29513990F8 pmix_invoke_local Unknown Unknown
mca_pmix_pmix2x.s 00007F295139E0B7 Unknown Unknown Unknown
mca_pmix_pmix2x.s 00007F295139D8FA Unknown Unknown Unknown
mca_pmix_pmix2x.s 00007F295140B296 pmix_ptl_base_pro Unknown Unknown
libopen-pal.so.40 00007F29D481FF92 opal_libevent2022 Unknown Unknown
mca_pmix_pmix2x.s 00007F29513DD642 Unknown Unknown Unknown
libpthread-2.17.s 00007F29DA274EA5 Unknown Unknown Unknown
libc-2.17.so 00007F29D9D998DD clone Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
jules.exe 00000000007D4B14 Unknown Unknown Unknown
libpthread-2.17.s 00007FA0BFB55630 Unknown Unknown Unknown
libc-2.17.so 00007FA0BF63985D nanosleep Unknown Unknown
libc-2.17.so 00007FA0BF66A134 usleep Unknown Unknown
libmpi.so.40.10.1 00007FA0C00B8299 ompi_mpi_init Unknown Unknown
libmpi.so.40.10.1 00007FA0C00E5DFE MPI_Init Unknown Unknown
libmpi_mpifh.so.4 00007FA0C03CFD17 PMPI_Init_f08 Unknown Unknown
jules.exe 000000000040CBDD MAIN 73 jules.F90
jules.exe 000000000040CB92 Unknown Unknown Unknown
libc-2.17.so 00007FA0BF596555
libc_start_main Unknown Unknown
jules.exe 000000000040CAA9 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
jules.exe 00000000007D4B14 Unknown Unknown Unknown
libpthread-2.17.s 00007FE86DD43630 Unknown Unknown Unknown
libc-2.17.so 00007FE86D82785D nanosleep Unknown Unknown
libc-2.17.so 00007FE86D858134 usleep Unknown Unknown
libmpi.so.40.10.1 00007FE86E2A6299 ompi_mpi_init Unknown Unknown
libmpi.so.40.10.1 00007FE86E2D3DFE MPI_Init Unknown Unknown
libmpi_mpifh.so.4 00007FE86E5BDD17 PMPI_Init_f08 Unknown Unknown
jules.exe 000000000040CBDD MAIN 73 jules.F90
jules.exe 000000000040CB92 Unknown Unknown Unknown
libc-2.17.so 00007FE86D784555
libc_start_main Unknown Unknown
jules.exe 000000000040CAA9 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
jules.exe 00000000007D4B14 Unknown Unknown Unknown
libpthread-2.17.s 00007FCDB6BA2630 Unknown Unknown Unknown
libc-2.17.so 00007FCDB668685D nanosleep Unknown Unknown
libc-2.17.so 00007FCDB66B7134 usleep Unknown Unknown
libmpi.so.40.10.1 00007FCDB7105299 ompi_mpi_init Unknown Unknown
libmpi.so.40.10.1 00007FCDB7132DFE MPI_Init Unknown Unknown
libmpi_mpifh.so.4 00007FCDB741CD17 PMPI_Init_f08 Unknown Unknown
jules.exe 000000000040CBDD MAIN 73 jules.F90
jules.exe 000000000040CB92 Unknown Unknown Unknown
libc-2.17.so 00007FCDB65E3555
libc_start_main Unknown Unknown
jules.exe 000000000040CAA9 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
jules.exe 00000000007D4B14 Unknown Unknown Unknown
libpthread-2.17.s 00007F27EF14E630 Unknown Unknown Unknown
libc-2.17.so 00007F27EEC3285D nanosleep Unknown Unknown
libc-2.17.so 00007F27EEC63134 usleep Unknown Unknown
libmpi.so.40.10.1 00007F27EF6B1299 ompi_mpi_init Unknown Unknown
libmpi.so.40.10.1 00007F27EF6DEDFE MPI_Init Unknown Unknown
libmpi_mpifh.so.4 00007F27EF9C8D17 PMPI_Init_f08 Unknown Unknown
jules.exe 000000000040CBDD MAIN 73 jules.F90
jules.exe 000000000040CB92 Unknown Unknown Unknown
libc-2.17.so 00007F27EEB8F555
libc_start_main Unknown Unknown
jules.exe 000000000040CAA9 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
jules.exe 00000000007D4B14 Unknown Unknown Unknown
libpthread-2.17.s 00007FC6E13D8630 Unknown Unknown Unknown
libc-2.17.so 00007FC6E0EBC85D nanosleep Unknown Unknown
libc-2.17.so 00007FC6E0EED134 usleep Unknown Unknown
libmpi.so.40.10.1 00007FC6E193B299 ompi_mpi_init Unknown Unknown
libmpi.so.40.10.1 00007FC6E1968DFE MPI_Init Unknown Unknown
libmpi_mpifh.so.4 00007FC6E1C52D17 PMPI_Init_f08 Unknown Unknown
jules.exe 000000000040CBDD MAIN 73 jules.F90
jules.exe 000000000040CB92 Unknown Unknown Unknown
libc-2.17.so 00007FC6E0E19555
libc_start_main Unknown Unknown
jules.exe 000000000040CAA9 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
jules.exe 00000000007D4B14 Unknown Unknown Unknown
libpthread-2.17.s 00007F8FA66AB630 Unknown Unknown Unknown
libc-2.17.so 00007F8FA618F85D nanosleep Unknown Unknown
libc-2.17.so 00007F8FA61C0134 usleep Unknown Unknown
libmpi.so.40.10.1 00007F8FA6C0E299 ompi_mpi_init Unknown Unknown
libmpi.so.40.10.1 00007F8FA6C3BDFE MPI_Init Unknown Unknown
libmpi_mpifh.so.4 00007F8FA6F25D17 PMPI_Init_f08 Unknown Unknown
jules.exe 000000000040CBDD MAIN 73 jules.F90
jules.exe 000000000040CB92 Unknown Unknown Unknown
libc-2.17.so 00007F8FA60EC555
libc_start_main Unknown Unknown
jules.exe 000000000040CAA9 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
jules.exe 00000000007D4B14 Unknown Unknown Unknown
libpthread-2.17.s 00007EFD65C5D630 Unknown Unknown Unknown
libc-2.17.so 00007EFD6574185D nanosleep Unknown Unknown
libc-2.17.so 00007EFD65772134 usleep Unknown Unknown
libmpi.so.40.10.1 00007EFD661C0299 ompi_mpi_init Unknown Unknown
libmpi.so.40.10.1 00007EFD661EDDFE MPI_Init Unknown Unknown
libmpi_mpifh.so.4 00007EFD664D7D17 PMPI_Init_f08 Unknown Unknown
jules.exe 000000000040CBDD MAIN 73 jules.F90
jules.exe 000000000040CB92 Unknown Unknown Unknown
libc-2.17.so 00007EFD6569E555
libc_start_main Unknown Unknown
jules.exe 000000000040CAA9 Unknown Unknown Unknown


mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

Process name: [[30469,1],9]
Exit code: 174


[FAIL] rose-jules-run <<'STDIN'
[FAIL]
[FAIL] 'STDIN' # return-code=174
2021-02-08T08:45:27Z CRITICAL - failed/EXIT

.out
Suite : u-cc033
Task Job : 19600101T0000Z/S2/01 (try 1)
User@Host: nmc@…

Currently Loaded Modulefiles:

1) intel/cce/19.0.0 4) contrib/gnu/binutils/2.31
2) intel/fce/19.0.0 5) contrib/gnu/gcc/7.3.0
3) intel/19.0.0 6) eb/OpenMPI/intel/3.1.1

LD_LIBRARY_PATH=/apps/eb/software/OpenMPI/3.1.1-iccifort-2018.3.222-GCC-7.3.0-2.30/lib:/apps/contrib/gnu/gcc/7.3.0/lib64:/apps/contrib/gnu/gcc/deps:/apps/intel/2019//itac/2019.0.018/intel64/slib:/apps/intel/2019/compilers_and_libraries_2019.0.117/linux/compiler/lib/intel64_lin:/apps/intel/2019/compilers_and_libraries_2019.0.117/linux/ipp/lib/intel64:/apps/intel/2019/compilers_and_libraries_2019.0.117/linux/compiler/lib/intel64_lin:/apps/intel/2019/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64_lin:/apps/intel/2019/compilers_and_libraries_2019.0.117/linux/tbb/lib/intel64/gcc4.7:/apps/intel/2019/compilers_and_libraries_2019.0.117/linux/tbb/lib/intel64/gcc4.7:/apps/intel/2019/debugger_2019/libipt/intel64/lib:/apps/intel/2019/compilers_and_libraries_2019.0.117/linux/daal/lib/intel64_lin:/apps/intel/2019/compilers_and_libraries_2019.0.117/linux/daal/../tbb/lib/intel64_lin/gcc4.4
LD_LIBRARY_PATH=/apps/eb/software/OpenMPI/3.1.1-iccifort-2018.3.222-GCC-7.3.0-2.30/lib:/apps/contrib/gnu/gcc/7.3.0/lib64:/apps/contrib/gnu/gcc/deps:/apps/intel/2019//itac/2019.0.018/intel64/slib:/apps/intel/2019/compilers_and_libraries_2019.0.117/linux/compiler/lib/intel64_lin:/apps/intel/2019/compilers_and_libraries_2019.0.117/linux/ipp/lib/intel64:/apps/intel/2019/compilers_and_libraries_2019.0.117/linux/compiler/lib/intel64_lin:/apps/intel/2019/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64_lin:/apps/intel/2019/compilers_and_libraries_2019.0.117/linux/tbb/lib/intel64/gcc4.7:/apps/intel/2019/compilers_and_libraries_2019.0.117/linux/tbb/lib/intel64/gcc4.7:/apps/intel/2019/debugger_2019/libipt/intel64/lib:/apps/intel/2019/compilers_and_libraries_2019.0.117/linux/daal/lib/intel64_lin:/apps/intel/2019/compilers_and_libraries_2019.0.117/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/home/users/siwilson/netcdf_par/3.1.1/intel.19.0.0/lib
2021-02-08T08:45:15Z INFO - started
[INFO] Running JULES in parallel MPI mode
[INFO] exec /apps/eb/software/OpenMPI/3.1.1-iccifort-2018.3.222-GCC-7.3.0-2.30/bin/mpirun /home/users/nmc/cylc-run/u-cc033/share/fcm_make/build/bin/jules.exe


Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.


comment:13 Changed 9 months ago by pmcguire

Hi Noel:
That's weird. If the spinup was for a non-zero number of months or years, then I am surprised that JULES ran ok during the spinup and then crashed during the main run. How many years or cycles was the spinup for?
Patrick

comment:14 Changed 9 months ago by NoelClancy

(base) [nmc@cylc1 u-cc033]$ grep -r spinup

.svn/pristine/1c/1c71df6eeeba2f3831b388235752ab09e1c6efbf.svn-base:file='$ANCIL_DIREC/JULES-GL7.0.vn5.2.CRUNCEPv7.spinup_10.dump.18800101.0.nc'
.svn/pristine/d3/d36ed8a7a1ff1a24e6527861e3c296263abe3a33.svn-base:source=namelist:jules_time namelist:jules_spinup
.svn/pristine/d3/d36ed8a7a1ff1a24e6527861e3c296263abe3a33.svn-base:output_spinup=.false.
.svn/pristine/d3/d36ed8a7a1ff1a24e6527861e3c296263abe3a33.svn-base:output_spinup=.false.
.svn/pristine/d3/d36ed8a7a1ff1a24e6527861e3c296263abe3a33.svn-base:output_spinup=.false.
.svn/pristine/d3/d36ed8a7a1ff1a24e6527861e3c296263abe3a33.svn-base:[namelist:jules_spinup]
.svn/pristine/d3/d36ed8a7a1ff1a24e6527861e3c296263abe3a33.svn-base:max_spinup_cycles=0
.svn/pristine/d3/d36ed8a7a1ff1a24e6527861e3c296263abe3a33.svn-base:!!spinup_end='1999-01-01 00:00:00'
.svn/pristine/d3/d36ed8a7a1ff1a24e6527861e3c296263abe3a33.svn-base:!!spinup_start='1989-01-01 00:00:00'
.svn/pristine/d3/d36ed8a7a1ff1a24e6527861e3c296263abe3a33.svn-base:!!terminate_on_spinup_fail=.false.
.svn/pristine/88/88523c3392fa55f325f0279c871fd821576d5dcc.svn-base:description = Maximum walltime after which the spinup task will be terminated - Cray only
.svn/pristine/88/88523c3392fa55f325f0279c871fd821576d5dcc.svn-base:title = Number of spinup cycles
.svn/pristine/88/88523c3392fa55f325f0279c871fd821576d5dcc.svn-base:description = Number of spinup cycles
.svn/pristine/88/88523c3392fa55f325f0279c871fd821576d5dcc.svn-base:title = Length of spinup cycles
.svn/pristine/88/88523c3392fa55f325f0279c871fd821576d5dcc.svn-base:description = Length of spinup cycles relative to main run start date
.svn/pristine/74/746e9bf90610ff31919d11fbe0fbcec5ee37c63c.svn-base:title=Multi-platform JULES-GL7 @ vn5.2 : TRENDY spinup u-aa388/trunk@1169: JULES-C @ vn4.3; JASMIN SLURM batch processing
.svn/pristine/a0/a034a738c7aeeb21bd8a4ce851d37b0ce2be9d0b.svn-base: {{ ('fcm_make ⇒' + ('fcm_make2 ⇒ ' if MODE_RUN == 'meto-xc40' else ) if BUILD else ) + ('RECON ⇒ spinup_01' if LSPINUP else 'RECON ⇒ S2' ) }}
.svn/pristine/a0/a034a738c7aeeb21bd8a4ce851d37b0ce2be9d0b.svn-base: spinup_{{ '%02d' % i }} ⇒ spinup_{{ '%02d' % (i+1) }}
.svn/pristine/a0/a034a738c7aeeb21bd8a4ce851d37b0ce2be9d0b.svn-base: {{ ( 'spinup_%02d ⇒ S2' ) % SPINCYCLES }}
.svn/pristine/a0/a034a738c7aeeb21bd8a4ce851d37b0ce2be9d0b.svn-base: SPINDUMP = $( if $CYLC_TASK_CYCLE_POINT == $CYLC_SUITE_INITIAL_CYCLE_POINT ?; then if {{ LSPINUP|lower() }} ;then echo {{ 'spinup_%02d' % SPINCYCLES }} ; else echo 'RECON' ; fi ; else echo $ROSE_TASK_NAME; fi )
.svn/pristine/a0/a034a738c7aeeb21bd8a4ce851d37b0ce2be9d0b.svn-base: ROSE_APP_OPT_CONF_KEYS = spinup
.svn/pristine/a0/a034a738c7aeeb21bd8a4ce851d37b0ce2be9d0b.svn-base: spinup_{{'%02d' % (i+1) }}?
Binary file .svn/wc.db matches
app/jules/opt/rose-app-recon.conf:file='$ANCIL_DIREC/JULES-GL7.0.vn5.2.CRUNCEPv7.spinup_10.dump.18800101.0.nc'
app/jules/rose-app.conf:source=namelist:jules_time namelist:jules_spinup
app/jules/rose-app.conf:output_spinup=.false.
app/jules/rose-app.conf:output_spinup=.false.
app/jules/rose-app.conf:output_spinup=.false.
app/jules/rose-app.conf:[namelist:jules_spinup]
app/jules/rose-app.conf:max_spinup_cycles=0
app/jules/rose-app.conf:!!spinup_end='1999-01-01 00:00:00'
app/jules/rose-app.conf:!!spinup_start='1989-01-01 00:00:00'
app/jules/rose-app.conf:!!terminate_on_spinup_fail=.false.
meta/rose-meta.conf:description = Maximum walltime after which the spinup task will be terminated - Cray only
meta/rose-meta.conf:title = Number of spinup cycles
meta/rose-meta.conf:description = Number of spinup cycles
meta/rose-meta.conf:title = Length of spinup cycles
meta/rose-meta.conf:description = Length of spinup cycles relative to main run start date
rose-suite.info:title=Multi-platform JULES-GL7 @ vn5.2 : TRENDY spinup u-aa388/trunk@1169: JULES-C @ vn4.3; JASMIN SLURM batch processing
suite.rc: {{ ('fcm_make ⇒' + ('fcm_make2 ⇒ ' if MODE_RUN == 'meto-xc40' else ) if BUILD else ) + ('RECON ⇒ spinup_01' if LSPINUP else 'RECON ⇒ S2' ) }}
suite.rc: spinup_{{ '%02d' % i }} ⇒ spinup_{{ '%02d' % (i+1) }}
suite.rc: {{ ( 'spinup_%02d ⇒ S2' ) % SPINCYCLES }}
suite.rc: SPINDUMP = $( if $CYLC_TASK_CYCLE_POINT == $CYLC_SUITE_INITIAL_CYCLE_POINT ?; then if {{ LSPINUP|lower() }} ;then echo {{ 'spinup_%02d' % SPINCYCLES }} ; else echo 'RECON' ; fi ; else echo $ROSE_TASK_NAME; fi )
suite.rc: ROSE_APP_OPT_CONF_KEYS = spinup
suite.rc: spinup_{{'%02d' % (i+1) }}?
(base) [nmc@cylc1 u-cc033]$

comment:15 Changed 9 months ago by NoelClancy

vi rose-suite.conf

SPINCYCLES=1
SPINLEN='P20Y'

comment:16 Changed 9 months ago by NoelClancy

This suite is a copy of u-bx723 with no further changes made.

However, when checking things such as group_workspaces, I get the following. But maybe this old group_workspaces pathway doesn't matter?

(base) [nmc@cylc1 u-cc033]$ grep -r workspaces
.svn/pristine/d3/d36ed8a7a1ff1a24e6527861e3c296263abe3a33.svn-base:meta=/group_workspaces/jasmin2/jules/admin/JULES/vn5.3/trunk/rose-meta/jules-standalone/vn5.3
app/jules/rose-app.conf:meta=/group_workspaces/jasmin2/jules/admin/JULES/vn5.3/trunk/rose-meta/jules-standalone/vn5.3
(base) [nmc@cylc1 u-cc033]$

comment:17 Changed 9 months ago by pmcguire

Hi Noel:
Yes, it is probably good to fix the jules GWS paths that are in that suite. Can you do that for your copy?

Can you open the dump file at the end of spinup or at the beginning of the main run with xconv?
Does it look ok?
Patrick

Last edited 9 months ago by pmcguire (previous) (diff)

comment:18 Changed 9 months ago by NoelClancy

I already did that in u-cb899

cd app/jules/rose-suite.conf

#meta=/group_workspaces/jasmin2/jules/admin/JULES/vn5.3/trunk/rose-meta/jules-standalone/vn5.3
meta=/gws/nopw/j04/jules/admin/JULES/vn5.3/trunk/rose-meta/jules-standalone/vn5.3

But that suite also failed. u-cb899 was also a copy of u-bx723

comment:19 Changed 9 months ago by NoelClancy

(base) [nmc@cylc1 u-cc033]$ xconv JULES-GL7.0.vn5.3.CRUNCEPv7SLURM.spinup_01.dump.18800101.0.nc
-bash: xconv: command not found
(base) [nmc@cylc1 u-cc033]$ xconv JULES-GL7.0.vn5.3.CRUNCEPv7SLURM.S2.dump.18600101.0.nc
-bash: xconv: command not found
(base) [nmc@cylc1 u-cc033]$

comment:20 Changed 9 months ago by pmcguire

Hi Noel:
Do you get the same error messages when you run u-cb899 as you do for u-cc033? Does u-cb899 also fail after successfully completing the 20-year spinup?

In order to run xconv, you need to do a module load jasmin-sci first.

Also, I fixed the original u-bx723 suite so that it has the proper path to the meta data, which you spotted. Thanks!
Patrick

comment:21 Changed 9 months ago by pmcguire

Hi Noel:
Furthermore, I thought the dump files you were looking at would be in .pp format instead of .nc format. For .nc format data, it might be easier to use ncview than it is to use xconv. I don't think you need to do a module load jasmin-sci to use ncview. The ncview may not work for 1D JULES data, but maybe xconv can handle 1D JULES data. Looking at the dump files may not be the most fruitful thing to do. If you need to look at 1D land-only JULES data (which I think the dump files use), and if xconv can't handle 1D land-only JULES data, you might need to do other things to visualise the data. You can at least use ncinfo and/or ncdump to see the header information and some of the numerical values of the .nc files.
Patrick

comment:22 Changed 9 months ago by NoelClancy

(base) [nmc@cylc1 u-cc033]$ ncdump -h JULES-GL7.0.vn5.3.CRUNCEPv7SLURM.S2.dump.18600101.0.nc
netcdf JULES-GL7.0.vn5.3.CRUNCEPv7SLURM.S2.dump.18600101.0 {
dimensions:

land = 7771 ;
tile = 9 ;
sclayer = 1 ;
scpool = 1 ;
soil = 4 ;
snow = 3 ;
type = 9 ;
scalar = 1 ;

variables:

float canopy(tile, land) ;
float cs(scpool, sclayer, land) ;
float gs(land) ;
float snow_tile(tile, land) ;
float t_soil(soil, land) ;
float tstar_tile(tile, land) ;
float sthuf(soil, land) ;
float sthzw(land) ;
float zw(land) ;
float rgrain(tile, land) ;
float rho_snow(tile, land) ;
float snow_depth(tile, land) ;
float snow_grnd(tile, land) ;
float nsnow(tile, land) ;
float snow_ds(snow, tile, land) ;
float snow_ice(snow, tile, land) ;
float snow_liq(snow, tile, land) ;
float tsnow(snow, tile, land) ;
float rgrainl(snow, tile, land) ;
float frac(type, land) ;
float b(soil, land) ;
float sathh(soil, land) ;
float satcon(soil, land) ;
float sm_sat(soil, land) ;
float sm_crit(soil, land) ;
float sm_wilt(soil, land) ;
float hcap(soil, land) ;
float hcon(soil, land) ;
float albsoil(land) ;
float fexp(land) ;
float ti_mean(land) ;
float ti_sig(land) ;
float frac_agr(land) ;
float co2_mmr(scalar) ;
float latitude(land) ;
float longitude(land) ;

}
(base) [nmc@cylc1 u-cc033]$

comment:23 Changed 9 months ago by NoelClancy

(base) [nmc@cylc1 u-cc033]$ ncinfo JULES-GL7.0.vn5.3.CRUNCEPv7SLURM.S2.dump.18600101.0.nc
-bash: ncinfo: command not found

comment:24 Changed 9 months ago by NoelClancy

yes, it seems that u-cb899 the one where I corrected the path to the meta data alos seems to complete the spinup and progress on to the main run. However, for some reason it failed earlier in the main run and only 40 files were produced, whereas 350 files were produced in u-cc033

(base) [nmc@cylc1 u-cb899]$ ls | wc -l
40
(base) [nmc@cylc1 u-cb899]$ cd ..
(base) [nmc@cylc1 nmc]$ cd u-cc033
(base) [nmc@cylc1 u-cc033]$ ls | wc -l
350
(base) [nmc@cylc1 u-cc033]$

comment:25 Changed 9 months ago by pmcguire

Hi Noel:
It looks like your u-cb899 run finished the years 1860-1870 (eleven years) of the main S2 run, and your u-cc033 run finished 1860-1970 (111 years) of the main S2 run. This kind of means that it's mostly working, if it finished multiple years of the main run.

You can't use ncview with these files, since they are 1D (and not 2D), as you can see in the ncdump above, where everything depends on the land dimension (which is only 1 dimension) instead of two dimensions (lat & lon grid, or x & y grid).

You can also get similar information by using ncinfo, but to use ncinfo on cylc1.jasmin, you need to run module load jaspybefore running ncinfo. It's best to check that loading either the jaspy or the jasmin-sci modules first before reporting that a command is missing. You can also module unload the modules after you're done with them, in case you want to avoid changing your environment too much.

These files can be read by the ILAMB system, which knows about 1D land arrays.

I don't see any directory in your ~nmc\cylc-run that corresponds to your u-cb899 run. Did you delete that? I wanted to look at your log files from that run.

Your files for ~nmc/cylc-run/u-cc033/log/job/19600101T0000Z/S2/02/job.err AND ~nmc/cylc-run/u-cc033/log/job/19600101T0000Z/S2/02/job.out look fine to me. No major errors. Your job is still running. It looks like the 1970 cycle was submitted less than 2 hours ago (see ~nmc/cylc-run/u-cc033/log/job/19700101T0000Z/S2/01/job.status and ~nmc/cylc-run/u-cc033/log/job/19700101T0000Z/S2/01/job-activity.log).

Patrick

comment:26 Changed 9 months ago by NoelClancy

I don't see any directory in your ~nmc\cylc-run that corresponds to your u-cb899 run. Did you delete that? I wanted to look at your log files from that run.

I tried to re-run that suiye but in order to do so I had to do a suite-clean but it still wouldn't run so I deleted the cylc-run files and re-ran the suite. However, the suite is currently submitted to the MAIN_RUN but not moving beyound that point at the moment.

However, I copied the error messages for my own records.

Changed 9 months ago by NoelClancy

Changed 9 months ago by NoelClancy

comment:27 Changed 9 months ago by NoelClancy

your u-cc033 run finished 1860-1970 (111 years) of the main S2 run. This kind of means that it's mostly working, if it finished multiple years of the main run.

It has now finished 1860-1980 (121 years) of the main S2 run. But very slow ?? Is this lenght of time unusual or expected?

comment:28 Changed 9 months ago by NoelClancy

I belive I can kill the job that is currently submitted and then re-trigger the job from the GUI. However, I'm a little cautious of that because it is already at 1980 and I fear I might risk corrupting the suite and having to re-start it again from scratch.

comment:29 Changed 9 months ago by pmcguire

Hi Noel:
Can you check the SLURM queues to see how many jobs there are in the queue that you are using?
Patrick

comment:30 Changed 9 months ago by NoelClancy

whaI have three suites running currently, all in submitted phase at present. What commands are best to see how many queues are being used?

comment:31 Changed 9 months ago by pmcguire

Hi Noel:
Here are some useful SLURM commands:

squeue -u nmc

sprio -n -p par-multi -u nmc

sprio -n -p par-multi

There are other commands too, which are useful, but you can start with those three.
Patrick

comment:32 Changed 9 months ago by NoelClancy

(base) [nmc@cylc1 ~]$ squeue -u nmc

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)

39611951 par-multi u-cc033. nmc PD 0:00 1 (Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions)
39641122 par-multi u-cb899. nmc PD 0:00 1 (None)
39765742 par-multi u-cb828. nmc PD 0:00 1 (None)

(base) [nmc@cylc1 ~]$ sprio -n -p par-multi -u nmc

JOBID PARTITION USER PRIORITY AGE ASSOC FAIRSHARE TRES

39611951 par-multi nmc 0.00000067 0.2691055 0.0000000 0.0000300 cpu=0.00
39641122 par-multi nmc 0.00000061 0.2406961 0.0000000 0.0000300 cpu=0.00
39765742 par-multi nmc 0.00000005 0.0187351 0.0000000 0.0000300 cpu=0.00

(base) [nmc@cylc1 ~]$ sprio -n -p par-multi

comment:33 Changed 9 months ago by NoelClancy

(base) [nmc@cylc1 ~]$ sprio -n -p par-multi

JOBID PARTITION PRIORITY AGE ASSOC FAIRSHARE TRES

39585123 par-multi 0.00000070 0.2806597 0.0000000 0.0000000 cpu=0.00
39596939 par-multi 0.00000069 0.2762979 0.0000000 0.0000000 cpu=0.00
39596941 par-multi 0.00000069 0.2762946 0.0000000 0.0000000 cpu=0.00
39597936 par-multi 0.00000069 0.2758978 0.0000000 0.0000000 cpu=0.00
39597939 par-multi 0.00000069 0.2758978 0.0000000 0.0000000 cpu=0.00
39597945 par-multi 0.00000069 0.2758912 0.0000000 0.0000000 cpu=0.00
39598623 par-multi 0.00000069 0.2754894 0.0000000 0.0000000 cpu=0.00
39598625 par-multi 0.00000069 0.2754894 0.0000000 0.0000000 cpu=0.00
39601556 par-multi 0.00000069 0.2742973 0.0000000 0.0000000 cpu=0.00
39602379 par-multi 0.00000069 0.2739319 0.0000000 0.0000000 cpu=0.00
39604369 par-multi 0.00000068 0.2731019 0.0000000 0.0000000 cpu=0.00
39606302 par-multi 0.00000068 0.2723462 0.0000000 0.0000000 cpu=0.00
39606311 par-multi 0.00000068 0.2723446 0.0000000 0.0000000 cpu=0.00
39606317 par-multi 0.00000068 0.2723413 0.0000000 0.0000000 cpu=0.00
39607859 par-multi 0.00000068 0.2715427 0.0000000 0.0000000 cpu=0.00
39607860 par-multi 0.00000068 0.2715427 0.0000000 0.0000000 cpu=0.00
39607865 par-multi 0.00000068 0.2715377 0.0000000 0.0000000 cpu=0.00
39609573 par-multi 0.00000068 0.2707110 0.0000000 0.0000000 cpu=0.00
39611951 par-multi 0.00000067 0.2696015 0.0000000 0.0000300 cpu=0.00
39611994 par-multi 0.00000068 0.2695602 0.0000000 0.0000000 cpu=0.00
39612619 par-multi 0.00000067 0.2691485 0.0000000 0.0000000 cpu=0.00
39612622 par-multi 0.00000067 0.2691485 0.0000000 0.0000000 cpu=0.00
39613627 par-multi 0.00000067 0.2687417 0.0000000 0.0000000 cpu=0.00
39613630 par-multi 0.00000067 0.2687417 0.0000000 0.0000000 cpu=0.00
39613638 par-multi 0.00000067 0.2687302 0.0000000 0.0000000 cpu=0.00
39614408 par-multi 0.00000067 0.2683416 0.0000000 0.0000000 cpu=0.00
39614410 par-multi 0.00000067 0.2683416 0.0000000 0.0000000 cpu=0.00
39615070 par-multi 0.00000067 0.2679795 0.0000000 0.0000000 cpu=0.00
39617359 par-multi 0.00000067 0.2667460 0.0000000 0.0000000 cpu=0.00
39617875 par-multi 0.00000067 0.2663757 0.0000000 0.0000000 cpu=0.00
39618391 par-multi 0.00000067 0.2659590 0.0000000 0.0000000 cpu=0.00
39618392 par-multi 0.00000067 0.2659590 0.0000000 0.0000000 cpu=0.00
39618728 par-multi 0.00000067 0.2655952 0.0000000 0.0000000 cpu=0.00
39618743 par-multi 0.00000067 0.2655853 0.0000000 0.0000000 cpu=0.00
39619179 par-multi 0.00000066 0.2651587 0.0000000 0.0000000 cpu=0.00
39619180 par-multi 0.00000066 0.2651587 0.0000000 0.0000000 cpu=0.00
39620041 par-multi 0.00000066 0.2647603 0.0000000 0.0000000 cpu=0.00
39620515 par-multi 0.00000066 0.2644031 0.0000000 0.0000000 cpu=0.00
39621685 par-multi 0.00000066 0.2639881 0.0000000 0.0000000 cpu=0.00
39622214 par-multi 0.00000066 0.2635896 0.0000000 0.0000000 cpu=0.00
39622558 par-multi 0.00000066 0.2633333 0.0000000 0.0000000 cpu=0.00
39622820 par-multi 0.00000066 0.2631663 0.0000000 0.0000000 cpu=0.00
39623358 par-multi 0.00000066 0.2628075 0.0000000 0.0000000 cpu=0.00
39623986 par-multi 0.00000066 0.2623958 0.0000000 0.0000000 cpu=0.00
39624873 par-multi 0.00000066 0.2616303 0.0000000 0.0000000 cpu=0.00
39624874 par-multi 0.00000066 0.2616303 0.0000000 0.0000000 cpu=0.00
39624876 par-multi 0.00000066 0.2616270 0.0000000 0.0000000 cpu=0.00
39624927 par-multi 0.00000066 0.2616104 0.0000000 0.0000000 cpu=0.00
39625730 par-multi 0.00000066 0.2612120 0.0000000 0.0000000 cpu=0.00
39625733 par-multi 0.00000066 0.2612103 0.0000000 0.0000000 cpu=0.00
39627215 par-multi 0.00000065 0.2604150 0.0000000 0.0000000 cpu=0.00
39628281 par-multi 0.00000065 0.2600182 0.0000000 0.0000000 cpu=0.00
39628287 par-multi 0.00000065 0.2600099 0.0000000 0.0000000 cpu=0.00
39628300 par-multi 0.00000065 0.2599983 0.0000000 0.0000000 cpu=0.00
39628835 par-multi 0.00000065 0.2596181 0.0000000 0.0000000 cpu=0.00
39628845 par-multi 0.00000065 0.2596081 0.0000000 0.0000000 cpu=0.00
39629034 par-multi 0.00000065 0.2592113 0.0000000 0.0000000 cpu=0.00
39630758 par-multi 0.00000065 0.2580192 0.0000000 0.0000000 cpu=0.00
39631195 par-multi 0.00000065 0.2576174 0.0000000 0.0000000 cpu=0.00
39631589 par-multi 0.00000065 0.2572437 0.0000000 0.0000000 cpu=0.00
39632095 par-multi 0.00000065 0.2567593 0.0000000 0.0000000 cpu=0.00
39633437 par-multi 0.00000064 0.2540840 0.0000000 0.0000000 cpu=0.00
39633438 par-multi 0.00000064 0.2540807 0.0000000 0.0000000 cpu=0.00
39633701 par-multi 0.00000064 0.2536806 0.0000000 0.0000000 cpu=0.00
39633913 par-multi 0.00000064 0.2532143 0.0000000 0.0000000 cpu=0.00
39633914 par-multi 0.00000064 0.2532110 0.0000000 0.0000000 cpu=0.00
39633954 par-multi 0.00000064 0.2528720 0.0000000 0.0000000 cpu=0.00
39633955 par-multi 0.00000064 0.2528704 0.0000000 0.0000000 cpu=0.00
39633956 par-multi 0.00000064 0.2528704 0.0000000 0.0000000 cpu=0.00
39634137 par-multi 0.00000064 0.2524884 0.0000000 0.0000000 cpu=0.00
39634330 par-multi 0.00000063 0.2520833 0.0000000 0.0000000 cpu=0.00
39634566 par-multi 0.00000063 0.2516749 0.0000000 0.0000000 cpu=0.00
39634773 par-multi 0.00000063 0.2512814 0.0000000 0.0000000 cpu=0.00
39634774 par-multi 0.00000063 0.2512765 0.0000000 0.0000000 cpu=0.00
39635691 par-multi 0.00000063 0.2494907 0.0000000 0.0000000 cpu=0.00
39635693 par-multi 0.00000063 0.2494858 0.0000000 0.0000000 cpu=0.00
39635695 par-multi 0.00000063 0.2494841 0.0000000 0.0000000 cpu=0.00
39635696 par-multi 0.00000063 0.2494841 0.0000000 0.0000000 cpu=0.00
39635697 par-multi 0.00000063 0.2494841 0.0000000 0.0000000 cpu=0.00
39635833 par-multi 0.00000063 0.2492890 0.0000000 0.0000000 cpu=0.00
39636295 par-multi 0.00000063 0.2485069 0.0000000 0.0000000 cpu=0.00
39636732 par-multi 0.00000062 0.2477364 0.0000000 0.0000000 cpu=0.00
39636985 par-multi 0.00000062 0.2472917 0.0000000 0.0000000 cpu=0.00
39637317 par-multi 0.00000062 0.2465228 0.0000000 0.0000000 cpu=0.00
39637636 par-multi 0.00000062 0.2456928 0.0000000 0.0000000 cpu=0.00
39637737 par-multi 0.00000062 0.2453406 0.0000000 0.0000000 cpu=0.00
39637904 par-multi 0.00000062 0.2449157 0.0000000 0.0000000 cpu=0.00
39637906 par-multi 0.00000062 0.2449107 0.0000000 0.0000000 cpu=0.00
39638135 par-multi 0.00000062 0.2445073 0.0000000 0.0000000 cpu=0.00
39638397 par-multi 0.00000062 0.2441171 0.0000000 0.0000000 cpu=0.00
39638399 par-multi 0.00000062 0.2441171 0.0000000 0.0000000 cpu=0.00
39638844 par-multi 0.00000061 0.2437483 0.0000000 0.0000000 cpu=0.00
39639361 par-multi 0.00000061 0.2433085 0.0000000 0.0000000 cpu=0.00
39640496 par-multi 0.00000061 0.2421181 0.0000000 0.0000000 cpu=0.00
39640787 par-multi 0.00000061 0.2417741 0.0000000 0.0000000 cpu=0.00
39640788 par-multi 0.00000061 0.2417741 0.0000000 0.0000000 cpu=0.00
39640789 par-multi 0.00000061 0.2417708 0.0000000 0.0000000 cpu=0.00
39640791 par-multi 0.00000061 0.2417692 0.0000000 0.0000000 cpu=0.00
39641051 par-multi 0.00000061 0.2413409 0.0000000 0.0000000 cpu=0.00
39641052 par-multi 0.00000061 0.2413409 0.0000000 0.0000000 cpu=0.00
39641122 par-multi 0.00000061 0.2411921 0.0000000 0.0000300 cpu=0.00
39641525 par-multi 0.00000061 0.2405787 0.0000000 0.0000000 cpu=0.00
39641833 par-multi 0.00000061 0.2401802 0.0000000 0.0000000 cpu=0.00
39641834 par-multi 0.00000061 0.2401786 0.0000000 0.0000000 cpu=0.00
39641836 par-multi 0.00000061 0.2401786 0.0000000 0.0000000 cpu=0.00
39641837 par-multi 0.00000061 0.2401786 0.0000000 0.0000000 cpu=0.00
39641838 par-multi 0.00000061 0.2401786 0.0000000 0.0000000 cpu=0.00
39642424 par-multi 0.00000060 0.2391386 0.0000000 0.0000000 cpu=0.00
39642599 par-multi 0.00000060 0.2387533 0.0000000 0.0000000 cpu=0.00
39642624 par-multi 0.00000060 0.2387500 0.0000000 0.0000000 cpu=0.00
39642920 par-multi 0.00000060 0.2383829 0.0000000 0.0000000 cpu=0.00
39643056 par-multi 0.00000060 0.2380241 0.0000000 0.0000000 cpu=0.00
39643057 par-multi 0.00000060 0.2380109 0.0000000 0.0000000 cpu=0.00
39643197 par-multi 0.00000060 0.2376587 0.0000000 0.0000000 cpu=0.00
39643475 par-multi 0.00000060 0.2370453 0.0000000 0.0000000 cpu=0.00
39643477 par-multi 0.00000060 0.2370420 0.0000000 0.0000000 cpu=0.00
39643612 par-multi 0.00000060 0.2367444 0.0000000 0.0000000 cpu=0.00
39643621 par-multi 0.00000060 0.2367361 0.0000000 0.0000000 cpu=0.00
39643819 par-multi 0.00000060 0.2364501 0.0000000 0.0000000 cpu=0.00
39643825 par-multi 0.00000060 0.2364468 0.0000000 0.0000000 cpu=0.00
39644028 par-multi 0.00000060 0.2360995 0.0000000 0.0000000 cpu=0.00
39644231 par-multi 0.00000060 0.2357920 0.0000000 0.0000000 cpu=0.00
39644235 par-multi 0.00000060 0.2357887 0.0000000 0.0000000 cpu=0.00
39644239 par-multi 0.00000060 0.2357821 0.0000000 0.0000000 cpu=0.00
39644241 par-multi 0.00000060 0.2357804 0.0000000 0.0000000 cpu=0.00
39644464 par-multi 0.00000059 0.2351521 0.0000000 0.0000000 cpu=0.00
39644466 par-multi 0.00000059 0.2351505 0.0000000 0.0000000 cpu=0.00
39644654 par-multi 0.00000059 0.2347834 0.0000000 0.0000000 cpu=0.00
39644821 par-multi 0.00000059 0.2344428 0.0000000 0.0000000 cpu=0.00
39645101 par-multi 0.00000059 0.2338112 0.0000000 0.0000000 cpu=0.00
39645103 par-multi 0.00000059 0.2338112 0.0000000 0.0000000 cpu=0.00
39645394 par-multi 0.00000059 0.2335103 0.0000000 0.0000000 cpu=0.00
39645950 par-multi 0.00000059 0.2326372 0.0000000 0.0000000 cpu=0.00
39646011 par-multi 0.00000059 0.2325876 0.0000000 0.0000000 cpu=0.00
39646138 par-multi 0.00000059 0.2323380 0.0000000 0.0000000 cpu=0.00
39646167 par-multi 0.00000059 0.2323181 0.0000000 0.0000000 cpu=0.00
39646207 par-multi 0.00000059 0.2322735 0.0000000 0.0000000 cpu=0.00
39646291 par-multi 0.00000059 0.2320155 0.0000000 0.0000000 cpu=0.00
39646390 par-multi 0.00000064 0.2318502 0.0000000 0.0000000 cpu=0.00
39646526 par-multi 0.00000059 0.2316915 0.0000000 0.0000000 cpu=0.00
39646952 par-multi 0.00000059 0.2308879 0.0000000 0.0000000 cpu=0.00
39647295 par-multi 0.00000058 0.2300810 0.0000000 0.0000000 cpu=0.00
39647296 par-multi 0.00000058 0.2300794 0.0000000 0.0000000 cpu=0.00
39647443 par-multi 0.00000058 0.2298380 0.0000000 0.0000000 cpu=0.00
39647592 par-multi 0.00000058 0.2295122 0.0000000 0.0000000 cpu=0.00
39648217 par-multi 0.00000058 0.2284491 0.0000000 0.0000000 cpu=0.00
39648228 par-multi 0.00000058 0.2284392 0.0000000 0.0000000 cpu=0.00
39648605 par-multi 0.00000058 0.2278952 0.0000000 0.0000000 cpu=0.00
39648608 par-multi 0.00000058 0.2278886 0.0000000 0.0000000 cpu=0.00
39648815 par-multi 0.00000058 0.2276224 0.0000000 0.0000000 cpu=0.00
39649701 par-multi 0.00000057 0.2265195 0.0000000 0.0000000 cpu=0.00
39649963 par-multi 0.00000057 0.2260069 0.0000000 0.0000000 cpu=0.00
39650285 par-multi 0.00000057 0.2255060 0.0000000 0.0000000 cpu=0.00
39650387 par-multi 0.00000057 0.2252596 0.0000000 0.0000000 cpu=0.00
39650470 par-multi 0.00000057 0.2249917 0.0000000 0.0000000 cpu=0.00
39650792 par-multi 0.00000057 0.2242427 0.0000000 0.0000000 cpu=0.00
39650793 par-multi 0.00000057 0.2242427 0.0000000 0.0000000 cpu=0.00
39650798 par-multi 0.00000057 0.2242229 0.0000000 0.0000000 cpu=0.00
39652386 par-multi 0.00000056 0.2210698 0.0000000 0.0000000 cpu=0.00
39652659 par-multi 0.00000056 0.2205655 0.0000000 0.0000000 cpu=0.00
39653506 par-multi 0.00000056 0.2195569 0.0000000 0.0000000 cpu=0.00
39654165 par-multi 0.00000056 0.2188013 0.0000000 0.0000000 cpu=0.00
39654917 par-multi 0.00000055 0.2176372 0.0000000 0.0000000 cpu=0.00
39655148 par-multi 0.00000055 0.2171329 0.0000000 0.0000000 cpu=0.00
39655447 par-multi 0.00000055 0.2166716 0.0000000 0.0000000 cpu=0.00
39656742 par-multi 0.00000055 0.2146296 0.0000000 0.0000000 cpu=0.00
39658045 par-multi 0.00000054 0.2115427 0.0000000 0.0000000 cpu=0.00
39662148 par-multi 0.00000052 0.2029018 0.0000000 0.0000000 cpu=0.00
39663711 par-multi 0.00000051 0.2003786 0.0000000 0.0000000 cpu=0.00
39664422 par-multi 0.00000051 0.1991353 0.0000000 0.0000000 cpu=0.00
39664453 par-multi 0.00000051 0.1990278 0.0000000 0.0000000 cpu=0.00
39668697 par-multi 0.00000049 0.1916319 0.0000000 0.0000000 cpu=0.00
39674195 par-multi 0.00000048 0.1848115 0.0000000 0.0000000 cpu=0.00
39674410 par-multi 0.00000048 0.1841815 0.0000000 0.0000000 cpu=0.00
39674516 par-multi 0.00000048 0.1839319 0.0000000 0.0000000 cpu=0.00
39674855 par-multi 0.00000047 0.1830853 0.0000000 0.0000000 cpu=0.00
39675079 par-multi 0.00000047 0.1829018 0.0000000 0.0000000 cpu=0.00
39680317 par-multi 0.00000052 0.1774157 0.0000000 0.0000000 cpu=0.00
39680319 par-multi 0.00000052 0.1774140 0.0000000 0.0000000 cpu=0.00
39680322 par-multi 0.00000052 0.1774140 0.0000000 0.0000000 cpu=0.00
39680324 par-multi 0.00000052 0.1774140 0.0000000 0.0000000 cpu=0.00
39680328 par-multi 0.00000052 0.1774140 0.0000000 0.0000000 cpu=0.00
39680329 par-multi 0.00000052 0.1774058 0.0000000 0.0000000 cpu=0.00
39680333 par-multi 0.00000052 0.1773991 0.0000000 0.0000000 cpu=0.00
39680338 par-multi 0.00000052 0.1773892 0.0000000 0.0000000 cpu=0.00
39680343 par-multi 0.00000052 0.1773727 0.0000000 0.0000000 cpu=0.00
39680349 par-multi 0.00000052 0.1773644 0.0000000 0.0000000 cpu=0.00
39680355 par-multi 0.00000052 0.1773545 0.0000000 0.0000000 cpu=0.00
39680359 par-multi 0.00000052 0.1773545 0.0000000 0.0000000 cpu=0.00
39680361 par-multi 0.00000052 0.1773330 0.0000000 0.0000000 cpu=0.00
39680367 par-multi 0.00000052 0.1773247 0.0000000 0.0000000 cpu=0.00
39680371 par-multi 0.00000052 0.1773198 0.0000000 0.0000000 cpu=0.00
39680374 par-multi 0.00000052 0.1773198 0.0000000 0.0000000 cpu=0.00
39680378 par-multi 0.00000052 0.1773049 0.0000000 0.0000000 cpu=0.00
39680381 par-multi 0.00000052 0.1773016 0.0000000 0.0000000 cpu=0.00
39680386 par-multi 0.00000052 0.1773016 0.0000000 0.0000000 cpu=0.00
39680387 par-multi 0.00000052 0.1773016 0.0000000 0.0000000 cpu=0.00
39680390 par-multi 0.00000052 0.1773016 0.0000000 0.0000000 cpu=0.00
39680393 par-multi 0.00000052 0.1772999 0.0000000 0.0000000 cpu=0.00
39680396 par-multi 0.00000052 0.1772999 0.0000000 0.0000000 cpu=0.00
39680398 par-multi 0.00000052 0.1772983 0.0000000 0.0000000 cpu=0.00
39680401 par-multi 0.00000052 0.1772983 0.0000000 0.0000000 cpu=0.00
39680402 par-multi 0.00000052 0.1772983 0.0000000 0.0000000 cpu=0.00
39680405 par-multi 0.00000052 0.1772983 0.0000000 0.0000000 cpu=0.00
39680407 par-multi 0.00000052 0.1772983 0.0000000 0.0000000 cpu=0.00
39680408 par-multi 0.00000052 0.1772983 0.0000000 0.0000000 cpu=0.00
39680421 par-multi 0.00000052 0.1772966 0.0000000 0.0000000 cpu=0.00
39680425 par-multi 0.00000052 0.1772966 0.0000000 0.0000000 cpu=0.00
39680427 par-multi 0.00000052 0.1772966 0.0000000 0.0000000 cpu=0.00
39680428 par-multi 0.00000052 0.1772966 0.0000000 0.0000000 cpu=0.00
39680430 par-multi 0.00000052 0.1772950 0.0000000 0.0000000 cpu=0.00
39680432 par-multi 0.00000052 0.1772950 0.0000000 0.0000000 cpu=0.00
39680433 par-multi 0.00000052 0.1772950 0.0000000 0.0000000 cpu=0.00
39680435 par-multi 0.00000052 0.1772950 0.0000000 0.0000000 cpu=0.00
39680436 par-multi 0.00000052 0.1772950 0.0000000 0.0000000 cpu=0.00
39680439 par-multi 0.00000052 0.1772950 0.0000000 0.0000000 cpu=0.00
39680441 par-multi 0.00000052 0.1772950 0.0000000 0.0000000 cpu=0.00
39680444 par-multi 0.00000052 0.1772950 0.0000000 0.0000000 cpu=0.00
39680450 par-multi 0.00000052 0.1772933 0.0000000 0.0000000 cpu=0.00
39680455 par-multi 0.00000052 0.1772917 0.0000000 0.0000000 cpu=0.00
39680457 par-multi 0.00000052 0.1772900 0.0000000 0.0000000 cpu=0.00
39680459 par-multi 0.00000052 0.1772884 0.0000000 0.0000000 cpu=0.00
39680462 par-multi 0.00000052 0.1772867 0.0000000 0.0000000 cpu=0.00
39680463 par-multi 0.00000052 0.1772867 0.0000000 0.0000000 cpu=0.00
39680465 par-multi 0.00000052 0.1772867 0.0000000 0.0000000 cpu=0.00
39680468 par-multi 0.00000052 0.1772867 0.0000000 0.0000000 cpu=0.00
39680471 par-multi 0.00000052 0.1772867 0.0000000 0.0000000 cpu=0.00
39680831 par-multi 0.00000046 0.1769924 0.0000000 0.0000000 cpu=0.00
39681903 par-multi 0.00000046 0.1761921 0.0000000 0.0000000 cpu=0.00
39686384 par-multi 0.00000062 0.1743022 0.0000000 0.0000000 cpu=0.01
39690515 par-multi 0.00000045 0.1727050 0.0000000 0.0000000 cpu=0.00
39690519 par-multi 0.00000045 0.1727050 0.0000000 0.0000000 cpu=0.00
39690520 par-multi 0.00000045 0.1727050 0.0000000 0.0000000 cpu=0.00
39690525 par-multi 0.00000045 0.1727034 0.0000000 0.0000000 cpu=0.00
39690533 par-multi 0.00000045 0.1727017 0.0000000 0.0000000 cpu=0.00
39690548 par-multi 0.00000045 0.1726935 0.0000000 0.0000000 cpu=0.00
39690549 par-multi 0.00000045 0.1726935 0.0000000 0.0000000 cpu=0.00
39690554 par-multi 0.00000045 0.1726935 0.0000000 0.0000000 cpu=0.00
39690556 par-multi 0.00000045 0.1726935 0.0000000 0.0000000 cpu=0.00
39690557 par-multi 0.00000045 0.1726918 0.0000000 0.0000000 cpu=0.00
39690562 par-multi 0.00000045 0.1726802 0.0000000 0.0000000 cpu=0.00
39690563 par-multi 0.00000045 0.1726802 0.0000000 0.0000000 cpu=0.00
39690564 par-multi 0.00000045 0.1726786 0.0000000 0.0000000 cpu=0.00
39690577 par-multi 0.00000045 0.1726769 0.0000000 0.0000000 cpu=0.00
39691899 par-multi 0.00000057 0.1721329 0.0000000 0.0000000 cpu=0.01
39692287 par-multi 0.00000057 0.1720321 0.0000000 0.0000000 cpu=0.01
39692295 par-multi 0.00000057 0.1720304 0.0000000 0.0000000 cpu=0.01
39692300 par-multi 0.00000057 0.1720304 0.0000000 0.0000000 cpu=0.01
39692302 par-multi 0.00000057 0.1720304 0.0000000 0.0000000 cpu=0.01
39692306 par-multi 0.00000057 0.1720304 0.0000000 0.0000000 cpu=0.01
39692311 par-multi 0.00000057 0.1720304 0.0000000 0.0000000 cpu=0.01
39692316 par-multi 0.00000057 0.1720304 0.0000000 0.0000000 cpu=0.01
39692319 par-multi 0.00000057 0.1720304 0.0000000 0.0000000 cpu=0.01
39697486 par-multi 0.00000039 0.1694296 0.0000000 0.0000000 cpu=0.00
39715416 par-multi 0.00000053 0.1608168 0.0000000 0.0003846 cpu=0.00
39716408 par-multi 0.00000042 0.1603191 0.0000000 0.0000000 cpu=0.00
39717539 par-multi 0.00000042 0.1599702 0.0000000 0.0000000 cpu=0.00
39721035 par-multi 0.00000042 0.1578770 0.0000000 0.0000000 cpu=0.00
39730787 par-multi 0.00000040 0.1524223 0.0000000 0.0000000 cpu=0.00
39731615 par-multi 0.00000051 0.1517675 0.0000000 0.0003846 cpu=0.00
39733891 par-multi 0.00000040 0.1503654 0.0000000 0.0000000 cpu=0.00
39733893 par-multi 0.00000040 0.1503571 0.0000000 0.0000000 cpu=0.00
39735100 par-multi 0.00000040 0.1496329 0.0000000 0.0000000 cpu=0.00
39735265 par-multi 0.00000040 0.1495635 0.0000000 0.0000000 cpu=0.00
39735304 par-multi 0.00000040 0.1495139 0.0000000 0.0000000 cpu=0.00
39738615 par-multi 0.00000039 0.1475744 0.0000000 0.0000000 cpu=0.00
39740290 par-multi 0.00000039 0.1465807 0.0000000 0.0000000 cpu=0.00
39741199 par-multi 0.00000039 0.1460367 0.0000000 0.0000000 cpu=0.00
39743079 par-multi 0.00000038 0.1443882 0.0000000 0.0000000 cpu=0.00
39743253 par-multi 0.00000038 0.1437021 0.0000000 0.0000000 cpu=0.00
39743321 par-multi 0.00000038 0.1434954 0.0000000 0.0000000 cpu=0.00
39743322 par-multi 0.00000038 0.1434954 0.0000000 0.0000000 cpu=0.00
39743329 par-multi 0.00000038 0.1434689 0.0000000 0.0000000 cpu=0.00
39743333 par-multi 0.00000038 0.1434673 0.0000000 0.0000000 cpu=0.00
39743335 par-multi 0.00000038 0.1434673 0.0000000 0.0000000 cpu=0.00
39743338 par-multi 0.00000038 0.1434673 0.0000000 0.0000000 cpu=0.00
39743340 par-multi 0.00000038 0.1434673 0.0000000 0.0000000 cpu=0.00
39743341 par-multi 0.00000038 0.1434673 0.0000000 0.0000000 cpu=0.00
39743343 par-multi 0.00000038 0.1434673 0.0000000 0.0000000 cpu=0.00
39743450 par-multi 0.00000038 0.1430241 0.0000000 0.0000000 cpu=0.00
39743812 par-multi 0.00000038 0.1417560 0.0000000 0.0000000 cpu=0.00
39743816 par-multi 0.00000038 0.1417394 0.0000000 0.0000000 cpu=0.00
39744763 par-multi 0.00000037 0.1403621 0.0000000 0.0000000 cpu=0.00
39744885 par-multi 0.00000037 0.1399322 0.0000000 0.0000000 cpu=0.00
39744917 par-multi 0.00000037 0.1398016 0.0000000 0.0000000 cpu=0.00
39745293 par-multi 0.00000037 0.1390774 0.0000000 0.0000000 cpu=0.00
39745888 par-multi 0.00000037 0.1379679 0.0000000 0.0000000 cpu=0.00
39746110 par-multi 0.00000037 0.1376587 0.0000000 0.0000000 cpu=0.00
39746117 par-multi 0.00000037 0.1376339 0.0000000 0.0000000 cpu=0.00
39746128 par-multi 0.00000037 0.1376224 0.0000000 0.0000000 cpu=0.00
39746320 par-multi 0.00000037 0.1369478 0.0000000 0.0000000 cpu=0.00
39746562 par-multi 0.00000037 0.1367576 0.0000000 0.0000000 cpu=0.00
39747776 par-multi 0.00000036 0.1359888 0.0000000 0.0000000 cpu=0.00
39747822 par-multi 0.00000036 0.1359474 0.0000000 0.0000000 cpu=0.00
39747959 par-multi 0.00000036 0.1356696 0.0000000 0.0000000 cpu=0.00
39747965 par-multi 0.00000036 0.1356217 0.0000000 0.0000000 cpu=0.00
39748049 par-multi 0.00000036 0.1353406 0.0000000 0.0000000 cpu=0.00
39748056 par-multi 0.00000036 0.1353042 0.0000000 0.0000000 cpu=0.00
39748066 par-multi 0.00000036 0.1352778 0.0000000 0.0000000 cpu=0.00
39748387 par-multi 0.00000036 0.1349091 0.0000000 0.0000000 cpu=0.00
39749180 par-multi 0.00000036 0.1333862 0.0000000 0.0000000 cpu=0.00
39749194 par-multi 0.00000036 0.1333813 0.0000000 0.0000000 cpu=0.00
39749626 par-multi 0.00000036 0.1329563 0.0000000 0.0000000 cpu=0.00
39749631 par-multi 0.00000036 0.1328968 0.0000000 0.0000000 cpu=0.00
39750031 par-multi 0.00000047 0.1306068 0.0000000 0.0000000 cpu=0.01
39750033 par-multi 0.00000047 0.1306019 0.0000000 0.0000000 cpu=0.01
39750037 par-multi 0.00000047 0.1305969 0.0000000 0.0000000 cpu=0.01
39750042 par-multi 0.00000047 0.1305853 0.0000000 0.0000000 cpu=0.01
39750044 par-multi 0.00000047 0.1305853 0.0000000 0.0000000 cpu=0.01
39750050 par-multi 0.00000047 0.1305688 0.0000000 0.0000000 cpu=0.01
39750052 par-multi 0.00000047 0.1305556 0.0000000 0.0000000 cpu=0.01
39750053 par-multi 0.00000047 0.1305556 0.0000000 0.0000000 cpu=0.01
39750381 par-multi 0.00000035 0.1291104 0.0000000 0.0000000 cpu=0.00
39750383 par-multi 0.00000035 0.1291088 0.0000000 0.0000000 cpu=0.00
39750386 par-multi 0.00000035 0.1291071 0.0000000 0.0000000 cpu=0.00
39750387 par-multi 0.00000035 0.1291038 0.0000000 0.0000000 cpu=0.00
39750396 par-multi 0.00000035 0.1290939 0.0000000 0.0000000 cpu=0.00
39750408 par-multi 0.00000035 0.1290063 0.0000000 0.0000000 cpu=0.00
39751930 par-multi 0.00000033 0.1211343 0.0000000 0.0000000 cpu=0.00
39751931 par-multi 0.00000033 0.1211343 0.0000000 0.0000000 cpu=0.00
39751941 par-multi 0.00000033 0.1211260 0.0000000 0.0000000 cpu=0.00
39751948 par-multi 0.00000033 0.1211227 0.0000000 0.0000000 cpu=0.00
39751951 par-multi 0.00000033 0.1211144 0.0000000 0.0000000 cpu=0.00
39751958 par-multi 0.00000033 0.1210714 0.0000000 0.0000000 cpu=0.00
39751963 par-multi 0.00000033 0.1210532 0.0000000 0.0000000 cpu=0.00
39751966 par-multi 0.00000033 0.1210086 0.0000000 0.0000000 cpu=0.00
39751970 par-multi 0.00000033 0.1210053 0.0000000 0.0000000 cpu=0.00
39751973 par-multi 0.00000033 0.1210020 0.0000000 0.0000000 cpu=0.00
39751974 par-multi 0.00000033 0.1210020 0.0000000 0.0000000 cpu=0.00
39751976 par-multi 0.00000033 0.1210003 0.0000000 0.0000000 cpu=0.00
39751978 par-multi 0.00000033 0.1210003 0.0000000 0.0000000 cpu=0.00
39751980 par-multi 0.00000033 0.1210003 0.0000000 0.0000000 cpu=0.00
39751983 par-multi 0.00000033 0.1210003 0.0000000 0.0000000 cpu=0.00
39751985 par-multi 0.00000033 0.1209871 0.0000000 0.0000000 cpu=0.00
39751988 par-multi 0.00000033 0.1209854 0.0000000 0.0000000 cpu=0.00
39756751 par-multi 0.00000027 0.0748313 0.0000000 0.0000000 cpu=0.00
39759103 par-multi 0.00000017 0.0551835 0.0000000 0.0001921 cpu=0.00
39761460 par-multi 0.00000014 0.0410384 0.0000000 0.0000000 cpu=0.00
39761846 par-multi 0.00000020 0.0392642 0.0000000 0.0000000 cpu=0.00
39761849 par-multi 0.00000020 0.0392626 0.0000000 0.0000000 cpu=0.00
39761851 par-multi 0.00000020 0.0392626 0.0000000 0.0000000 cpu=0.00
39761854 par-multi 0.00000020 0.0392278 0.0000000 0.0000000 cpu=0.00
39761857 par-multi 0.00000020 0.0392278 0.0000000 0.0000000 cpu=0.00
39761859 par-multi 0.00000020 0.0392179 0.0000000 0.0000000 cpu=0.00
39761861 par-multi 0.00000020 0.0391981 0.0000000 0.0000000 cpu=0.00
39761864 par-multi 0.00000020 0.0391931 0.0000000 0.0000000 cpu=0.00
39761867 par-multi 0.00000020 0.0391319 0.0000000 0.0000000 cpu=0.00
39761899 par-multi 0.00000020 0.0391104 0.0000000 0.0000000 cpu=0.00
39761909 par-multi 0.00000019 0.0388938 0.0000000 0.0000000 cpu=0.00
39761911 par-multi 0.00000019 0.0388905 0.0000000 0.0000000 cpu=0.00
39761915 par-multi 0.00000019 0.0388294 0.0000000 0.0000000 cpu=0.00
39761919 par-multi 0.00000019 0.0386607 0.0000000 0.0000000 cpu=0.00
39761931 par-multi 0.00000019 0.0385136 0.0000000 0.0000000 cpu=0.00
39761936 par-multi 0.00000019 0.0385119 0.0000000 0.0000000 cpu=0.00
39761939 par-multi 0.00000019 0.0384954 0.0000000 0.0000000 cpu=0.00
39761950 par-multi 0.00000019 0.0384507 0.0000000 0.0000000 cpu=0.00
39761952 par-multi 0.00000019 0.0384408 0.0000000 0.0000000 cpu=0.00
39761955 par-multi 0.00000019 0.0384243 0.0000000 0.0000000 cpu=0.00
39761957 par-multi 0.00000019 0.0384110 0.0000000 0.0000000 cpu=0.00
39761959 par-multi 0.00000019 0.0384011 0.0000000 0.0000000 cpu=0.00
39761961 par-multi 0.00000019 0.0384011 0.0000000 0.0000000 cpu=0.00
39761963 par-multi 0.00000019 0.0384011 0.0000000 0.0000000 cpu=0.00
39761965 par-multi 0.00000019 0.0383978 0.0000000 0.0000000 cpu=0.00
39761967 par-multi 0.00000019 0.0383862 0.0000000 0.0000000 cpu=0.00
39761970 par-multi 0.00000019 0.0383846 0.0000000 0.0000000 cpu=0.00
39761972 par-multi 0.00000019 0.0383681 0.0000000 0.0000000 cpu=0.00
39761974 par-multi 0.00000019 0.0383548 0.0000000 0.0000000 cpu=0.00
39761976 par-multi 0.00000019 0.0383548 0.0000000 0.0000000 cpu=0.00
39761978 par-multi 0.00000019 0.0383383 0.0000000 0.0000000 cpu=0.00
39761980 par-multi 0.00000019 0.0383251 0.0000000 0.0000000 cpu=0.00
39761984 par-multi 0.00000019 0.0382556 0.0000000 0.0000000 cpu=0.00
39762877 par-multi 0.00000023 0.0334325 0.0000000 0.0003846 cpu=0.00
39765742 par-multi 0.00000005 0.0192312 0.0000000 0.0000300 cpu=0.00
39767593 par-multi 0.00000013 0.0130919 0.0000000 0.0000000 cpu=0.00
39767597 par-multi 0.00000013 0.0130754 0.0000000 0.0000000 cpu=0.00
39767609 par-multi 0.00000013 0.0130704 0.0000000 0.0000000 cpu=0.00
39767611 par-multi 0.00000013 0.0130655 0.0000000 0.0000000 cpu=0.00
39767619 par-multi 0.00000013 0.0130638 0.0000000 0.0000000 cpu=0.00
39767620 par-multi 0.00000013 0.0130357 0.0000000 0.0000000 cpu=0.00
39767624 par-multi 0.00000013 0.0130060 0.0000000 0.0000000 cpu=0.00
39767626 par-multi 0.00000013 0.0129663 0.0000000 0.0000000 cpu=0.00
39767628 par-multi 0.00000013 0.0129597 0.0000000 0.0000000 cpu=0.00
39767634 par-multi 0.00000013 0.0129001 0.0000000 0.0000000 cpu=0.00
39767637 par-multi 0.00000013 0.0128819 0.0000000 0.0000000 cpu=0.00
39767643 par-multi 0.00000013 0.0128356 0.0000000 0.0000000 cpu=0.00
39767648 par-multi 0.00000013 0.0127298 0.0000000 0.0000000 cpu=0.00
39767650 par-multi 0.00000013 0.0126984 0.0000000 0.0000000 cpu=0.00
39767658 par-multi 0.00000013 0.0126620 0.0000000 0.0000000 cpu=0.00
39767662 par-multi 0.00000013 0.0126339 0.0000000 0.0000000 cpu=0.00
39767664 par-multi 0.00000013 0.0125661 0.0000000 0.0000000 cpu=0.00
39767674 par-multi 0.00000013 0.0124901 0.0000000 0.0000000 cpu=0.00
39767678 par-multi 0.00000013 0.0124719 0.0000000 0.0000000 cpu=0.00
39767680 par-multi 0.00000013 0.0124537 0.0000000 0.0000000 cpu=0.00
39767709 par-multi 0.00000013 0.0124008 0.0000000 0.0000000 cpu=0.00
39767723 par-multi 0.00000013 0.0123760 0.0000000 0.0000000 cpu=0.00
39767727 par-multi 0.00000013 0.0123760 0.0000000 0.0000000 cpu=0.00

comment:34 Changed 9 months ago by pmcguire

Hi Noel:
I am glad that the SLURM commands work for you.
Can you interpret the output of your SLURM commands to answer your questions?
How many jobs are ahead of you in the par-multi queue?
Is your job running yet?

Can you also look at your log files for each decade of your run to see what the queueing time was for each decade so far, and what the running time for each decade was?
Patrick

comment:35 Changed 9 months ago by NoelClancy

I have three jobs in the par-multi queue

(base) [nmc@cylc1 ~]$ sacct

JobID JobName? Partition Account AllocCPUS State ExitCode?


39611951 u-cc033.S+ par-multi jasmin 10 PENDING 0:0
39641122 u-cb899.R+ par-multi jasmin 10 PENDING 0:0
39765742 u-cb828.m+ par-multi jasmin 1 PENDING 0:0

No jobs running, all currently submitted

comment:36 Changed 9 months ago by NoelClancy

How can you tell how jobs are ahead of you in the par-multi queue?

Priority = 3932 ???

(base) [nmc@cylc1 ~]$ scontrol show job 39641122
JobId?=39641122 JobName?=u-cb899.RECON.18600101T0000Z

UserId?=nmc(7051669) GroupId?=users(26030) MCS_label=N/A
Priority=3932 Nice=0 Account=jasmin QOS=normal
JobState?=PENDING Reason=None Dependency=(null)
Requeue=0 Restarts=0 BatchFlag?=1 Reboot=0 ExitCode?=0:0
RunTime?=00:00:00 TimeLimit?=00:10:00 TimeMin?=N/A
SubmitTime?=2021-02-08T23:33:04 EligibleTime?=2021-02-08T23:33:04
AccrueTime?=2021-02-08T23:33:04
StartTime?=Unknown EndTime?=Unknown Deadline=N/A
SuspendTime?=None SecsPreSuspend?=0 LastSchedEval?=2021-02-09T13:55:27
Partition=par-multi AllocNode:Sid=cylc1:32656
ReqNodeList?=(null) ExcNodeList?=(null)
NodeList?=(null)
NumNodes?=1 NumCPUs=10 NumTasks?=10 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
TRES=cpu=10,mem=80000M,node=1,billing=10
Socks/Node?=* NtasksPerN:B:S:C=0:0:*:* CoreSpec?=*
MinCPUsNode=1 MinMemoryCPU=8000M MinTmpDiskNode?=0
Features=(null) DelayBoot?=00:00:00
OverSubscribe?=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/home/users/nmc/cylc-run/u-cb899/log/job/18600101T0000Z/RECON/01/job
WorkDir?=/
StdErr?=/home/users/nmc/cylc-run/u-cb899/log/job/18600101T0000Z/RECON/01/job.err
StdIn?=/dev/null
StdOut?=/home/users/nmc/cylc-run/u-cb899/log/job/18600101T0000Z/RECON/01/job.out
Power=

comment:37 Changed 9 months ago by NoelClancy

Can you also look at your log files for each decade of your run to see what the queueing time was for each decade so far, and what the running time for each decade was?

It took decade starting 1950 just over an hour

(base) [nmc@cylc1 01]$ cd /home/users/nmc/cylc-run/u-cc033/log/job
(base) [nmc@cylc1 job]$ ls
18600101T0000Z 18800101T0000Z 19000101T0000Z 19200101T0000Z 19400101T0000Z 19600101T0000Z 19800101T0000Z
18700101T0000Z 18900101T0000Z 19100101T0000Z 19300101T0000Z 19500101T0000Z 19700101T0000Z
(base) [nmc@cylc1 job]$ cd 19500101T0000Z
(base) [nmc@cylc1 19500101T0000Z]$ ls
S2
(base) [nmc@cylc1 19500101T0000Z]$ cd S2/
(base) [nmc@cylc1 S2]$ ls
01 NN
(base) [nmc@cylc1 S2]$ cd 01
(base) [nmc@cylc1 01]$ ls
job job-activity.log job.err job.out job.status
(base) [nmc@cylc1 01]$ vi job.status

CYLC_BATCH_SYS_NAME=slurm
CYLC_BATCH_SYS_JOB_ID=39470893
CYLC_BATCH_SYS_JOB_SUBMIT_TIME=2021-02-08T07:37:16Z
CYLC_JOB_PID=23889
CYLC_JOB_INIT_TIME=2021-02-08T07:39:13Z
CYLC_JOB_EXIT=SUCCEEDED
CYLC_JOB_EXIT_TIME=2021-02-08T08:43:15Z
~

comment:38 Changed 9 months ago by pmcguire

Hi Noel:
How long are the waiting times and the running times of the other decades? I am wondering about the variability of these times.
You were commenting that it was taking a long time.

You can use sprio -n -p par-multi to list all the jobs in the queue. Your jobs are amongst them, I presume. Maybe you can look at the documentation for sprio to tell where you are in the queue from the output of sprio?
Patrick

comment:39 Changed 9 months ago by NoelClancy

Thanks very much. That command is useful.

job 39611951 failed to submit at about 11:30 this morning
job 39641122 is 21st in the queue.
job 39765742 is 291st

(base) [nmc@cylc1 ~]$ sprio -n -p par-multi

JOBID PARTITION PRIORITY AGE ASSOC FAIRSHARE TRES

39636295 par-multi 0.00000097 0.3948181 0.0000000 0.0000000 cpu=0.00
39636732 par-multi 0.00000096 0.3940476 0.0000000 0.0000000 cpu=0.00
39636985 par-multi 0.00000096 0.3936028 0.0000000 0.0000000 cpu=0.00
39637317 par-multi 0.00000096 0.3928340 0.0000000 0.0000000 cpu=0.00
39637636 par-multi 0.00000096 0.3920040 0.0000000 0.0000000 cpu=0.00
39637737 par-multi 0.00000096 0.3916518 0.0000000 0.0000000 cpu=0.00
39637904 par-multi 0.00000096 0.3912269 0.0000000 0.0000000 cpu=0.00
39637906 par-multi 0.00000096 0.3912219 0.0000000 0.0000000 cpu=0.00
39638135 par-multi 0.00000096 0.3908185 0.0000000 0.0000000 cpu=0.00
39638397 par-multi 0.00000096 0.3904282 0.0000000 0.0000000 cpu=0.00
39638399 par-multi 0.00000096 0.3904282 0.0000000 0.0000000 cpu=0.00
39638844 par-multi 0.00000096 0.3900595 0.0000000 0.0000000 cpu=0.00
39639361 par-multi 0.00000095 0.3896197 0.0000000 0.0000000 cpu=0.00
39640496 par-multi 0.00000095 0.3884292 0.0000000 0.0000000 cpu=0.00
39640787 par-multi 0.00000095 0.3880853 0.0000000 0.0000000 cpu=0.00
39640788 par-multi 0.00000095 0.3880853 0.0000000 0.0000000 cpu=0.00
39640789 par-multi 0.00000095 0.3880820 0.0000000 0.0000000 cpu=0.00
39640791 par-multi 0.00000095 0.3880804 0.0000000 0.0000000 cpu=0.00
39641051 par-multi 0.00000095 0.3876521 0.0000000 0.0000000 cpu=0.00
39641052 par-multi 0.00000095 0.3876521 0.0000000 0.0000000 cpu=0.00
39641122 par-multi 0.00000095 0.3875033 0.0000000 0.0000304 cpu=0.00

comment:40 Changed 9 months ago by NoelClancy

(base) [nmc@cylc1 ~]$ squeue —user=nmc

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)

39641122 par-multi u-cb899. nmc PD 0:00 1 (Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions)
39765742 par-multi u-cb828. nmc PD 0:00 1 (Priority)
39817131 par-multi u-cc033. nmc PD 0:00 1 (None)

I'm not sure exactly what this means for job 39641122 which is currently 21st in the queue.

comment:41 Changed 9 months ago by pmcguire

Hi Noel
I think "Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions" means you have to wait your turn in the queue.
Patrick

comment:42 Changed 9 months ago by pmcguire

Hi Noel:
There was an email from JASMIN about SLURM. The email pointed to this blog page where updates will be posted:
https://www.ceda.ac.uk/blog/problem-with-lotus-batch-scheduler-slurm/
Patrick

comment:43 Changed 9 months ago by NoelClancy

How long are the waiting times and the running times of the other decades? I am wondering about the variability of these times.
You were commenting that it was taking a long time.

About one hour per decade

Changed 9 months ago by NoelClancy

comment:44 Changed 8 months ago by pmcguire

Hi Noel:
Wow! That waiting/running times file is interesting! Thanks!
I wouldn't be too surprised by the occasional 5 to 72 hour queueing times, if most of the queueing times are a few minutes.
Patrick

comment:45 Changed 8 months ago by pmcguire

Hi again Noel:
Since the suite is currently resubmitting every ten years and it is taking 1 hour for each 10 years,
and every time you do this it means waiting in the queue,
maybe you can try resubmitting it for (say) every 50 years (with a corresponding increase of requested wallclock time).
You'll have to make sure that the last 14 or 24 years (or whatever the remainder is) get run properly.

If that works, and if all the output files look good, then maybe it's worth trying to extend the resubmission to 100 years or more.
Patrick

Note: See TracTickets for help on using tickets.