Opened 3 months ago

Last modified 3 months ago

#3140 accepted help

err.out message

Reported by: NoelClancy Owned by: pmcguire
Component: JULES Keywords: rose/Cylc, JULES
Cc: Platform: Monsoon2
UM Version:

Description

Hi Patrick,

Do you know why u-bm268 fails when the 4-dimensional output variable 'gc' is added? When I add it to output profile 2 to generate hourly data output files it fails.

I'm not sure if I understand the err.out message

Noel

Suite : u-bm268
Task Job : 1/fcm_make/01 (try 1)
User@Host: nclancy@shared100

Currently Loaded Modulefiles:

1) modules/3.2.10.3
2) nodestat/2.2-1.0502.60539.1.31.ari
3) sdb/1.1-1.0502.63652.4.25.ari
4) alps/5.2.5-2.0502.9955.44.1.ari
5) lustre-cray_ari_s/2.5_3.0.101_0.46.1_1.0502.8871.43.1-1.0502.21728.74.6
6) udreg/2.3.2-1.0502.10518.2.17.ari
7) ugni/6.0-1.0502.10863.8.29.ari
8) gni-headers/4.0-1.0502.10859.9.25.ari
9) dmapp/7.0.1-1.0502.11080.8.76.ari

10) xpmem/0.1-2.0502.64982.7.26.ari
11) hss-llm/7.2.0
12) Base-opts/1.0.2-1.0502.60680.2.4.ari
13) craype-network-aries
14) cce/8.3.4
15) craype/2.2.1
16) craype-haswell
17) cray-mpich/7.0.4
18) pbs/13.0.412.192482
19) moose-client-wrapper
20) python/v2.7.9
21) metoffice/tempdir
22) metoffice/userenv
23) subversion-1.8/1.8.19
24) cray-libsci/13.0.1
25) pmi/5.0.5-1.0000.10300.134.8.ari
26) rca/1.0.0-2.0502.60530.1.62.ari
27) atp/1.7.5
28) PrgEnv?-cray/5.2.40
29) cray-netcdf-hdf5parallel/4.3.2
30) cray-snplauncher/7.0.4

2020-01-17T00:18:19Z INFO - started
[INFO] create: /home/d04/nclancy/cylc-run/u-bm268/share/data
[INFO] create: /home/d04/nclancy/cylc-run/u-bm268/share/cycle/1
[INFO] export ROSE_CYCLING_MODE=integer
[INFO] export ROSE_DATA=/home/d04/nclancy/cylc-run/u-bm268/share/data
[INFO] export ROSE_DATAC=/home/d04/nclancy/cylc-run/u-bm268/share/cycle/1
[INFO] export ROSE_SUITE_DIR=/home/d04/nclancy/cylc-run/u-bm268
[INFO] export ROSE_SUITE_DIR_REL=cylc-run/u-bm268
[INFO] export ROSE_SUITE_NAME=u-bm268
[INFO] export ROSE_TASK_CYCLE_TIME=1
[INFO] export ROSE_TASK_ID=fcm_make.1
[INFO] export ROSE_TASK_IS_COLD_START=true
[INFO] export ROSE_TASK_LOG_DIR=/home/d04/nclancy/cylc-run/u-bm268/log/job/1/fcm_make/01
[INFO] export ROSE_TASK_LOG_ROOT=/home/d04/nclancy/cylc-run/u-bm268/log/job/1/fcm_make/01/job
[INFO] export ROSE_TASK_NAME=fcm_make
[INFO] export ROSE_TASK_PREFIX=fcm
[INFO] export ROSE_TASK_SUFFIX=make
[INFO] export PATH=/common/um1/bin:/projects/um1/bin:/common/fcm/rose-2019.01.2/bin:/home/d04/nclancy/cylc-run/u-bm268/bin:/home/d04/nclancy/cylc-run/u-bm268/bin:/opt/cray/snplauncher/7.0.4/bin:/opt/cray/netcdf/4.3.2/bin:/opt/cray/atp/1.7.5/bin:/opt/cray/rca/1.0.0-2.0502.60530.1.62.ari/bin:/opt/cray/pmi/5.0.5-1.0000.10300.134.8.ari/bin:/opt/cray/cce/8.3.4/cray-binutils/x86_64-unknown-linux-gnu/bin:/opt/cray/cce/8.3.4/craylibs/x86-64/bin:/opt/cray/cce/8.3.4/cftn/bin:/opt/cray/cce/8.3.4/CC/bin:/opt/ukmo/subversion/1.8.19/bin:/opt/python/gnu/2.7.9/bin/:/opt/ukmo/mass/moose-monsoon-client-latest/bin:/opt/pbs/13.0.412.192482/bin:/opt/cray/mpt/7.0.4/gni/bin:/opt/cray/craype/2.2.1/bin:/opt/cray/llm/default/bin:/opt/cray/llm/default/etc:/opt/cray/xpmem/0.1-2.0502.64982.7.26.ari/bin:/opt/cray/ugni/6.0-1.0502.10863.8.29.ari/bin:/opt/cray/udreg/2.3.2-1.0502.10518.2.17.ari/bin:/opt/cray/lustre-cray_ari_s/2.5_3.0.101_0.46.1_1.0502.8871.43.1-1.0502.21728.74.6/sbin:/opt/cray/lustre-cray_ari_s/2.5_3.0.101_0.46.1_1.0502.8871.43.1-1.0502.21728.74.6/bin:/opt/cray/alps/5.2.5-2.0502.9955.44.1.ari/sbin:/opt/cray/alps/5.2.5-2.0502.9955.44.1.ari/bin:/opt/cray/sdb/1.1-1.0502.63652.4.25.ari/bin:/opt/cray/nodestat/2.2-1.0502.60539.1.31.ari/bin:/opt/modules/3.2.10.3/bin:/home/d04/nclancy/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:/usr/lib/qt3/bin:/opt/ukmo/supported/bin/:/opt/cray/bin:/opt/ukmo/supported/bin/:/opt/ukmo/supported/bin/:/opt/ukmo/supported/bin/:/opt/ukmo/supported/bin/
[INFO] Configuration: /home/d04/nclancy/cylc-run/u-bm268/app/fcm_make/
[INFO] file: rose-app.conf
[INFO] source: /home/d04/nclancy/cylc-run/u-bm268/app/fcm_make/file/fcm-make.cfg
[INFO] install: fcm-make.cfg
[INFO] source: /home/d04/nclancy/cylc-run/u-bm268/app/fcm_make/file/fcm-make.cfg
[init] make # 2020-01-17T00:18:23Z
[info] FCM 2019.05.0 (/common/fcm/fcm-2019.05.0)
[init] make config-parse # 2020-01-17T00:18:23Z
[info] config-file=/working/d04/nclancy/cylc-run/u-bm268/work/1/fcm_make/fcm-make.cfg
[info] config-file= - file:///home/d04/fcm/srv/svn/jules.xm/main/trunk/etc/fcm-make/make.cfg@14197
[info] config-file= - - file:///home/d04/fcm/srv/svn/jules.xm/main/trunk/etc/fcm-make/platform/meto-xc40-cce.cfg@14197
[info] config-file= - - - file:///home/d04/fcm/srv/svn/jules.xm/main/trunk/etc/fcm-make/platform/custom.cfg@14197
[info] config-file= - - - - file:///home/d04/fcm/srv/svn/jules.xm/main/trunk/etc/fcm-make/remote/local.cfg@14197
[info] config-file= - - - - file:///home/d04/fcm/srv/svn/jules.xm/main/trunk/etc/fcm-make/compiler/cray.cfg@14197
[info] config-file= - - - - file:///home/d04/fcm/srv/svn/jules.xm/main/trunk/etc/fcm-make/build/normal.cfg@14197
[info] config-file= - - - - file:///home/d04/fcm/srv/svn/jules.xm/main/trunk/etc/fcm-make/omp/omp.cfg@14197
[info] config-file= - - - - file:///home/d04/fcm/srv/svn/jules.xm/main/trunk/etc/fcm-make/ncdf/netcdf.cfg@14197
[info] config-file= - - - - file:///home/d04/fcm/srv/svn/jules.xm/main/trunk/etc/fcm-make/mpi/mpi.cfg@14197
[done] make config-parse # 1.5s
[init] make dest-init # 2020-01-17T00:18:25Z
[info] dest=nclancy@shared100:/home/d04/nclancy/cylc-run/u-bm268/share/fcm_make
[info] mode=new
[done] make dest-init # 0.0s
[init] make extract # 2020-01-17T00:18:25Z
[info] location jules: 0: file:///home/d04/fcm/srv/svn/jules.xm/main/trunk@14197 (14197)
[info] dest: 553 [A added]
[info] source: 553 [U from base]
[done] make extract # 34.4s
[init] make preprocess # 2020-01-17T00:18:59Z
[info] sources: total=553, analysed=549, elapsed-time=1.0s, total-time=1.1s
[info] target-tree-analysis: elapsed-time=0.1s
[info] install targets: modified=154, unchanged=0, failed=0, total-time=1.7s
[info] process targets: modified=395, unchanged=0, failed=0, total-time=5.3s
[info] TOTAL targets: modified=549, unchanged=0, failed=0, elapsed-time=1.8s
[done] make preprocess # 3.0s
[init] make build # 2020-01-17T00:19:02Z
[info] sources: total=553, analysed=553, elapsed-time=2.1s, total-time=8.7s
[info] target-tree-analysis: elapsed-time=0.6s
[info] compile targets: modified=381, unchanged=0, failed=0, total-time=467.0s
[info] compile+ targets: modified=342, unchanged=0, failed=0, total-time=0.5s
[info] install targets: modified=1, unchanged=0, failed=0, total-time=0.0s
[info] link targets: modified=1, unchanged=0, failed=0, total-time=24.6s
[info] TOTAL targets: modified=725, unchanged=0, failed=0, elapsed-time=278.5s
[done] make build # 280.6s
[done] make # 319.4s
2020-01-17T00:23:44Z INFO - succeeded
============================= PBS epilogue =============================

file %r not found…
End of Job Report
Run at 2020-01-17 00:23:44 for job 1520266.xcs00
Submitted : 2020-01-17 00:16:58
Queued : 2020-01-17 00:16:58
Started : 2020-01-17 00:18:17
Completed : 2020-01-17 00:23:44
Queued Time : 0:01:19 (79 seconds)
Elapsed Time : 0:05:27 (327 seconds, 27% of limit)
Walltime Limit : 0:20:00 (1200 seconds)
Node Time Limit : 0:03:45 (225 seconds)
Node Time : 0:01:01 (61 seconds, 27% of limit)
Job Name : fcm_make.1.u-bm268
Queue : shared
Owner : nclancy
Group : mo_users
Project : ukca-read
Subproject :
Funding :
Trustzone : collaboration
STDOUT : /home/d04/nclancy/cylc-run/u-bm268/log/job/1/fcm_make/01/job.out
STDERR : /home/d04/nclancy/cylc-run/u-bm268/log/job/1/fcm_make/01/job.err
Job Directory : /scratch/jtmp/pbs.1520267.xcs00.x8z
Job Arch :
CPU Core Type : broadwell
Total Nodes : 1
Total Tasks : 6
Parent Node : shared100
Parent Node Memory :
Parent Node CPU Time :
Compute Nodes :
Electrical Groups :
Run Version : 1

CPU Wallclock Wallclock RSS Memory Memory Memory

Node ID Count Used Requested Used Used Requested CPU Time
========== ========== ========== ========== ========== ========== ========== ==========
633 12 328 27% 0.0 1.4G 76.8% 407.0
========== ========== ========== ========== ========== ========== ========== ==========

For more information see documentation

Change History (1)

comment:1 Changed 3 months ago by pmcguire

  • Cc mcguirepatr@… removed
  • Keywords rose/Cylc, JULES added
  • Platform set to Monsoon2
  • Status changed from new to accepted

Hi Noel
Are you sure that it is failing in the fcm_make task? The log above is for the fcm_make task, and I don't readily see any 'error' or 'fail' messages. Maybe it is for another task in your suite? Can you take a look at the log files for the other tasks?

If you submit Helpdesk queries to the 'JULES' component or the 'Land Surface Modelling' component, you needn't attach my email address to the CC: list, as such queries will be first directed to me regardless.
Patrick

Note: See TracTickets for help on using tickets.