Opened 5 months ago

Closed 5 weeks ago

#3232 closed help (answered)

Global-JULES on JASMIN

Reported by: NoelClancy Owned by: pmcguire
Component: JULES Keywords: group workspace, GWS
Cc: Platform: JASMIN
UM Version:

Description

Hi Patrick,

I ran suite u-bs482 last week but as far as I recall it passed the fcm_make and failed during the main_run because of a missing input file.

That input file is there now and when I run it the suite fails at the fcm_make stage but I don't understand why. I copied the error messages below. Do you know what might be wrong?

/home/users/nmc/cylc-run/u-bs482/log/job/17000101T0000Z/fcm_make/01/job.err

Environment variables set for netCDF Fortran bindings in

/apps/libs/netCDF/intel14/fortran/4.2/

You will also need to link your code to a compatible netCDF C library in

/apps/libs/netCDF/intel14/4.3.2/

[FAIL] mpif90 -obin/jules.exe o/jules.o -L/tmp/be6syL_Zo7 -ljules -L/apps/libs/netCDF/intel14/4.3.2/lib -L/apps/libs/netCDF/intel14/fortran/4.2/lib -L/group_workspaces/jasmin2/jules/admin/curl/curl-lotus-parallel-intel/lib/ -L/apps/libs/PHDF5/intel14/1.8.12/lib -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lcurl -heap-arrays -fp-model precise -traceback # rc=1
[FAIL] ifort: command line warning #10212: -fp-model precise evaluates in source precision with Fortran.
[FAIL] ld: cannot find -lcurl
[FAIL] link 1.4 ! jules.exe ← jules/src/control/standalone/jules.F90
[FAIL] ! jules.exe : update task failed

[FAIL] fcm make -f /work/scratch/nmc/cylc-run/u-bs482/work/17000101T0000Z/fcm_make/fcm-make.cfg -C /home/users/nmc/cylc-run/u-bs482/share/fcm_make -j 4 # return-code=255
2020-03-24T09:26:36Z CRITICAL - failed/EXIT

Suite : u-bs482
Task Job : 17000101T0000Z/fcm_make/01 (try 1)
User@Host: nmc@…

/home/users/nmc/cylc-run/u-bs482/log/job/17000101T0000Z/fcm_make/01/job.out

Currently Loaded Modulefiles:

1) lsfmodules/10.1 7) libpnetcdf/intel/14.0/1.5.0
2) lotus-mpi/8.2 8) netcdf/intel/14.0/4.3.2
3) intel/cce/14.0.1.106 9) netcdff/intel/14.0/4.2
4) intel/fce/14.0.1.106 10) intel/fce/14.0.2.144
5) intel/14.0 11) intel/cce/14.0.2.144
6) libPHDF5/intel/14.0/1.8.12 12) parallel-netcdf/intel/20141122

LD_LIBRARY_PATH=/apps/intel/2013_sp1.2.144/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/apps/intel/2013_sp1.2.144/composer_xe_2013_sp1.2.144/ipp/lib/intel64:/apps/libs/netCDF/intel14/fortran/4.2/lib:/apps/libs/netCDF/intel14/4.3.2/lib:/apps/intel/2013_sp1.1.106/composer_xe_2013_sp1.1.106/ipp/lib/intel64:/apps/intel/2013_sp1.1.106/composer_xe_2013_sp1.1.106/compiler/lib/intel64:/opt/platform_mpi/lib/linux_amd64:/apps/lsf/10.1/linux2.6-glibc2.3-x86_64/lib
LD_LIBRARY_PATH=/apps/intel/2013_sp1.2.144/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/apps/intel/2013_sp1.2.144/composer_xe_2013_sp1.2.144/ipp/lib/intel64:/apps/libs/netCDF/intel14/fortran/4.2/lib:/apps/libs/netCDF/intel14/4.3.2/lib:/apps/intel/2013_sp1.1.106/composer_xe_2013_sp1.1.106/ipp/lib/intel64:/apps/intel/2013_sp1.1.106/composer_xe_2013_sp1.1.106/compiler/lib/intel64:/opt/platform_mpi/lib/linux_amd64:/apps/lsf/10.1/linux2.6-glibc2.3-x86_64/lib:/apps/libs/PHDF5/intel14/1.8.12/lib
2020-03-24T09:21:57Z INFO - started
[init] make # 2020-03-24T09:22:00Z
[info] FCM 2017.10.0 (/apps/contrib/metomi/fcm-2017.10.0)
[init] make config-parse # 2020-03-24T09:22:00Z
[info] config-file=/work/scratch/nmc/cylc-run/u-bs482/work/17000101T0000Z/fcm_make/fcm-make.cfg
[info] config-file= - https://code.metoffice.gov.uk/svn/jules/main/trunk/etc/fcm-make/make.cfg@14197
"job.out" 54L, 4377C

Change History (17)

comment:1 Changed 5 months ago by NoelClancy

Hi Patrick

[FAIL] ld: cannot find -lcurl

The former path
/group_workspaces/jasmin2/jules/admin/curl

has now been changed to
/group_workspaces/jasmin2/jules_OLD/admin/curl

I think this is the probem

comment:2 Changed 5 months ago by pmcguire

Hi Noel
The location of the JULES group workspace was changed last week from
/group_workspaces/jasmin2/jules
to
/gws/nopw/j04/jules

Do you belong to the jules-users email list group? If so, you should have received an email about that on March 16th.

This suite that you're using uses JULES5.4. In JULES5.4 in the MOSRS, the
file etc/fcm-make/platform/jasmin-lotus-intel.cfg points to the old GWS, and needs to be updated.
You can create a branch of JULES5.4, and make this change for yourself, and then update that GWS path in the etc/fcm-make/platform/jasmin-lotus-intel.cfg file.

You also need to change the GWS in these files:
~nmc/roses/u-bs482/benchmark/include/jasmin/variables.rc
~nmc/roses/u-bs482/rose-stem/include/jasmin/runtime.rc
~nmc/roses/u-bs482/suite.rc

Patrick

comment:3 Changed 5 months ago by pmcguire

  • Status changed from new to accepted

comment:4 Changed 5 months ago by pmcguire

  • Keywords group workspace, GWS added
  • Platform set to JASMIN

comment:5 Changed 5 months ago by pmcguire

Hi Noel
When you modify your new branch of JULES5.4, you'll also need to update the location of the branch in your rose-suite.conf file.
Currently, the relevant variables are set as:

JULES_FCM='fcm:jules.x_tr'
JULES_REVISION='vn5.4'

Patrick

Last edited 5 months ago by pmcguire (previous) (diff)

comment:6 Changed 5 months ago by NoelClancy

Hi Patrick,

There may also be something wrong with your pathway

etc/fcm-make/platform/jasmin-lotus-intel.cfg

I can cd /etc
but there is no fcm-make sub directory there

Could you check this please ?

Also, how exactly do I create my own branch of JULES5.4?

comment:7 Changed 5 months ago by pmcguire

Hi Noel:
To get access to that pathway or path, you can either create a branch of JULES 5.4 yourself or check out a trunk copy of JULES 5.4.

For example, to check out a trunk copy of JULES 5.4, you can:

cd ~
mkdir jules 
cd jules
fcm co fcm:jules.x_tr@vn5.4 vn5.4
cd vn5.4

Then you should be able to:

cd etc/fcm-make/platform
more jasmin-lotus-intel.cfg

etc.

Patrick

comment:8 Changed 5 months ago by NoelClancy

Do you mean change the following
JULES_FCM='fcm:jules.x_tr'
JULES_REVISION='vn5.4'

to
JULES_FCM='/home/users/nmc/jules/vn5.4
JULES_REVISION='vn5.4'

comment:9 Changed 5 months ago by NoelClancy

Do you mean change the following
JULES_FCM='fcm:jules.x_tr'
JULES_REVISION='vn5.4'

to
JULES_FCM='/home/users/nmc/jules/vn5.4'
JULES_REVISION='vn5.4'

comment:10 Changed 5 months ago by NoelClancy

[FAIL] {% set JULES_FCM='/home/users/nmc/jules/vn5.4'

ok, that was wrong, how to you edit the following two lines?

JULES_FCM='fcm:jules.x_tr'
JULES_REVISION='vn5.4'

comment:11 Changed 5 months ago by pmcguire

Hi Noel:
I think you will also need to change another file.

If you do:

cd ~/roses/u-bs482/
grep -ir JULES_REVISION

then you should see these lines (plus some svn lines):

app/fcm_make/file/fcm-make.cfg:include = $JULES_FCM/etc/fcm-make/make.cfg@$JULES_REVISION

meta/rose-meta.conf:trigger = jinja2:suite.rc=JULES_REVISION: true;

meta/rose-meta.conf:[jinja2:suite.rc=JULES_REVISION]

suite.rc: JULES_REVISION = {{ JULES_REVISION if BUILD else 'No build required' }}

suite.rc: JULES_SOURCE=$JULES_FCM@$JULES_REVISION

rose-suite.conf:JULES_REVISION='vn5.4'

Since you're not using the MOSRS trunk version anymore, then you'll need to modify:

app/fcm_make/file/fcm-make.cfg

so that it has this line in it (as well as the other lines):

include = $JULES_FCM/etc/fcm-make/make.cfg

instead of

include = $JULES_FCM/etc/fcm-make/make.cfg@$JULES_REVISION

Your values of:

JULES_FCM='/home/users/nmc/jules/vn5.4'
JULES_REVISION='vn5.4'

should be fine.

Let me know if this works.
Patrick

comment:12 Changed 5 months ago by NoelClancy

Patrick,

Done what you asked but still the same error

[FAIL] cylc validate -o /tmp/tmpbcto3n —strict u-bs482 # return-code=1, stderr=
[FAIL] Jinja2Error:
[FAIL] File "<unknown>", line 8, in template
[FAIL] TemplateSyntaxError?: unexpected char u'#' at 297
[FAIL] Context lines:
[FAIL] {% set CYLC_VERSION="7.8.1" %}
[FAIL] {% set FUNDING='hccp' %}
[FAIL] {% set HPC_QUEUE='normal' %}
[FAIL] {% set JULES_FCM='/home/users/nmc/jules/vn5.4' #This is for the vn5.4 branch checked out on '/home/users/nmc/jules/vn5.4' %} ←- Jinja2Error

comment:13 Changed 5 months ago by NoelClancy

Don't understand this

[FAIL] File "<unknown>", line 8, in template
[FAIL] TemplateSyntaxError??: unexpected char u'#' at 297

comment:14 Changed 5 months ago by pmcguire

Hi Noel:
Can you try to get rid of the inline comment in your rose-suite.conf file, by changing:

JULES_FCM='/home/users/nmc/jules/vn5.4' #This is for the vn5.4 branch checked out on '/home/users/nmc/jules/vn5.4'

to

JULES_FCM='/home/users/nmc/jules/vn5.4' 

Does that work?
Patrick

comment:15 Changed 5 months ago by NoelClancy

Patrick,

One final thing from yesterday, I made the following change in the suite.rc

#JULES_SOURCE=$JULES_FCM@$JULES_REVISION
JULES_SOURCE=$JULES_FCM

Was that correct?

fcm_make?

{%- if MODE_RUN == 'meto-xc40' %}

inherit = None, FCM_MAKE_XC40

{%- else %}

inherit = None, FCM_MAKE_JASMIN

{%- endif %}

script = """

rose task-run —quiet —path=share/fcm_make/build/bin
"""

[environment?]

JULES_BUILD=normal
#JULES_SOURCE=$JULES_FCM@$JULES_REVISION
JULES_SOURCE=$JULES_FCM

comment:16 Changed 5 months ago by pmcguire

Hi Noel:
Is it working now?

Regarding your most recent comment/question, if you do this:

grep -r JULES_SOURCE ~nmc/roses/u-bs482 

then you will see:

/home/users/nmc/roses/u-bs482/suite.rc: #JULES_SOURCE=$JULES_FCM@$JULES_REVISION
/home/users/nmc/roses/u-bs482/suite.rc: JULES_SOURCE=$JULES_FCM

This means that the variable JULES_SOURCE is defined by the suite, but you will note that it isn't used anywhere else in the suite.

The result of this is that it shouldn't matter what you set the JULES_SOURCE variable as, and you can leave it at either the old setting or with your new setting.
Patrick

comment:17 Changed 5 weeks ago by pmcguire

  • Resolution set to answered
  • Status changed from accepted to closed
Note: See TracTickets for help on using tickets.