Opened 4 years ago

Closed 4 years ago

#1628 closed help (fixed)

Compilation with NetCDF on Cray

Reported by: PUMA_GarryHayman Owned by: ros
Component: UM Model Keywords:
Cc: Platform: MONSooN
UM Version: <select version>

Description

I have successfully upgraded one of my jobs (xlrlc) to compile and run on the CRAY. This is an atmosphere-only job using the UKCA Std Tropospheric chemistry scheme with nudging to ECMWF re-analyses (all nudging-related options have been switched off).

I am now trying to switch on the various nudging-related options (handedits, branches, overrides). This job xlrld fails on compilation. It would fail in any case as the netCDF handedit and overrides are for the IBM. Please advise on error and how to compile on the CRAY using netCDF.

See also attached correspondence with Mohit Dalvi.

Attachments (1)

correspondence_mohit.txt (6.2 KB) - added by ros 4 years ago.
Email thread

Download all attachments as: .zip

Change History (10)

Changed 4 years ago by ros

Email thread

comment:1 Changed 4 years ago by ros

Hi Garry,

I've transferred your ticket to the CMS helpdesk as you'd put it in the UM Trac by mistake.

You don't need to worry about loading the netcdf library as this is already done in the UMUI scripts. (See umui_jobs/xlrld/SUBMIT)

I have created new versions of your 2 override files:

~umui/overrides/monsoon/netcdf_xc40.ovr to replace 2c_netcdf_7.3-7.4.ovr
and
~ros/umui_jobs/overrides/monsoon/file_ovr_ukca_ncdf_666 to replace file_ovr_ukca_ncdf_663

I'll now look into the "banner_output" problem.

Cheers,
Ros.

comment:2 Changed 4 years ago by ros

Hi Garry,

The banner_output problem is due to including hg6.6.3 branches, it is getting confused. Using branches from a previous revision can work, but can also lead to oddities. You will need to upgrade your branches. Unfortunately, when you created the hg6.6.3_nudging_change and hg6.6.3_aircraft_satellite_emulator branches you copied them from other branches (I assume using fcm/svn copy), which means the standard fcm merge won't work as it can't sort out the inheritance.

You will need to do the following for each branch (which I have just tested):
1) Create a new branch at hg6.6.6 using the /usr/local/bin/create_HG2_branch script on PUMA.

2) FCM checkout the new branch

3) cd to the working copy created by step (2)

4) Merge in the changes from your branch and the branch that it is copied from.
For the "nudging_change" run:

svn merge --revision 5321:18880 svn://puma/UM_svn/UM/branches/dev/PUMA_GarryHayman/hg6.6.3_nudging_change

For the "aircraft_satellite_emulator" run:

svn merge --revision 5321:19745 svn://puma/UM_svn/UM/branches/dev/PUMA_GarryHayman/hg6.6.3_aircraft_satellite_emulator

5) Run "fcm commit" to commit the merge to the repository.

Add these new branches to your job and the compile should continue, hopefully successfully.

If you do need to make a copy of a branch in the future, it is best to create a new branch and then merge in the branch (fcm merge <branch-name>) you wish to make a copy of. That way FCM can use & track changes more easily.

Cheers,
Ros.

comment:3 Changed 4 years ago by ros

Ros

Many thanks for this.

I was hoping to avoid the creation of new branches! OK, I have created a new branch for the aircraft/satellite emulator and followed your steps below (the nudging option is more involved). I have also used the new override files and have modified the job xlrld accordingly. The compilation has proceeded much further but still fails (see /home/ghayma/output/xlrld000.xlrld.d15232.t094209.comp.leave) - I think that this is because the new routines use free format (?). Is this compiler option set? It probably also explains why I get a lot of warnings.

On another matter, I found the reason why I was getting a timed out response (the job would be copied to but not submitted to the CRAY). The CRAY cluster option was not set (User Information and Target Machine → Target Machine). I noticed that it must have been switched on in xlpoi since I copied it. It was switched off in xlqja (the AO example), when I looked a day or two ago.

Regards

Garry

comment:4 Changed 4 years ago by ros

Hi Garry,

The Cray compiler is less forgiving than the IBM one. You need to be careful when you split strings over multiple lines.

In src/atmosphere/ukca/nadir_init.F90 lines 300/301

WRITE(*,'(A8,I4,A1,I2.2,A1,I2.2,1X,I2.2,A1,I2.2,2X, &
          A10,F14.6,2x,I2.2)') &

needs to be

WRITE(*,'(A8,I4,A1,I2.2,A1,I2.2,1X,I2.2,A1,I2.2,2X,'// &
         'A10,F14.6,2x,I2.2)') &

Cheers,
Ros.

comment:5 Changed 4 years ago by ros

Hi Garry,

Realised I didn't answer you other comment. The reason the CRAY cluster option got unset last week was due to the problems with qsub on the CRAY and having to switch to submit to xcml01 rather than xcml00.

Regards,
Ros.

comment:6 Changed 4 years ago by ros

Ros

OK, I will have a quick check through the code to correct these. The compiler, I guess, will pick up any that I miss.

Another question, I have found compiler directives for the IBM for example in the nudging routines that I am adding to my branch hg6.6.6_nudging_stdtrop (which I have commented out here):

!#if defined(IBM)
             '/projects/ukca-admin/analyses/era-in/'
!#else
!             '/work/n02/n02/ukca/nudging/era-in/'
!#endif

This was used to set the directory for MONSOON or HECTOR/ARCHER. There are other examples:

PUMA_GarryHayman@puma:~/hg6.6.6_nudging_stdtrop> grep -inr 'if defined(IBM' *
src/include/other/c_fort2c.h:76:#if defined(IBM)
src/control/misc/um_fort_flush.F90:43:#elif defined(IBM)
src/control/top_level/hdppxrf.F90:322:#if defined(IBM)
src/control/top_level/adv_uto.F90:247:#if defined(IBM)
src/control/top_level/rdbasis.F90:248:#if defined(IBM)
src/control/top_level/limiter.F90:68:#if defined(IBM)
src/utility/qxreconf/rcf_calc_2d_cca_mod.F90:64:#if defined(IBM) || defined(LINUX_PORTLAND_COMPILER)
src/atmosphere/nudging/nudging_control.F90:42:!#if defined(IBM)
src/atmosphere/UKCA/ukca_strat_update.F90:1167:#if defined(IBM)
src/atmosphere/UKCA/ukca_strat_update.F90:1203:#if defined(IBM)
src/atmosphere/land_surface/frunoff.F90:93:#if defined(IBM)
src/atmosphere/dynamics_advection/calc_non_int_sl_theta.F90:405:#if defined(IBM)
src/atmosphere/dynamics_advection/mono_enforce.F90:92:#if defined(IBM)
src/atmosphere/dynamics_advection/interpolation.F90:438:#if defined(IBM)
src/atmosphere/dynamics_advection/bi_linear_h.F90:227:#if defined(IBM)
src/atmosphere/tracer_advection/interpolation_multi.F90:392:#if defined(IBM)
src/atmosphere/tracer_advection/interpolation_qcon_multi.F90:393:#if defined(IBM)

Should these use the directive for the CRAY?

Thanks

Garry

comment:7 Changed 4 years ago by ros

Hi Garry,

No, unfortunately there is no specific directive for the CRAY.

I think you have 2 options here:

1) Create a machine override file to add a new XCM def:

%machdefs64 %machdefs64 XCM=xcm

and then change all the relevant IBMs to XCM in your code or add an additional || defined(XCM).
Luke has had to do this for UKCA at 7.3 and has an override file (~ukca/comp_overides/xcm_mach.ovr) which you could probably use.

or

2) Comment out all the #if defined(IBM) as per your example, leaving just the MONSooN path.

Cheers,
Ros.

comment:8 Changed 4 years ago by ros

Ros

I will make the necessary changes and use the machine override that you suggest.

Actually most of the examples of IBM compiler directives are in the 'original' code before I made the changes (the file starred in the list below is one that I added).

PUMA_GarryHayman@puma:~> grep -inr '#if defined(IBM' hg6.6.6_nudging_stdtrop/src
hg6.6.6_nudging_stdtrop/src/include/other/c_fort2c.h:76:#if defined(IBM)
hg6.6.6_nudging_stdtrop/src/control/top_level/hdppxrf.F90:322:#if defined(IBM)
hg6.6.6_nudging_stdtrop/src/control/top_level/adv_uto.F90:247:#if defined(IBM)
hg6.6.6_nudging_stdtrop/src/control/top_level/rdbasis.F90:248:#if defined(IBM)
hg6.6.6_nudging_stdtrop/src/control/top_level/limiter.F90:68:#if defined(IBM)
hg6.6.6_nudging_stdtrop/src/utility/qxreconf/rcf_calc_2d_cca_mod.F90:64:#if defined(IBM) || defined(LINUX_PORTLAND_COMPILER)
*** hg6.6.6_nudging_stdtrop/src/atmosphere/nudging/nudging_control.F90:42:!#if defined(IBM)
hg6.6.6_nudging_stdtrop/src/atmosphere/UKCA/ukca_strat_update.F90:1167:#if defined(IBM)
hg6.6.6_nudging_stdtrop/src/atmosphere/UKCA/ukca_strat_update.F90:1203:#if defined(IBM)
hg6.6.6_nudging_stdtrop/src/atmosphere/land_surface/frunoff.F90:93:#if defined(IBM)
hg6.6.6_nudging_stdtrop/src/atmosphere/dynamics_advection/calc_non_int_sl_theta.F90:405:#if defined(IBM)
hg6.6.6_nudging_stdtrop/src/atmosphere/dynamics_advection/mono_enforce.F90:92:#if defined(IBM)
hg6.6.6_nudging_stdtrop/src/atmosphere/dynamics_advection/interpolation.F90:438:#if defined(IBM)
hg6.6.6_nudging_stdtrop/src/atmosphere/dynamics_advection/bi_linear_h.F90:227:#if defined(IBM)
hg6.6.6_nudging_stdtrop/src/atmosphere/tracer_advection/interpolation_multi.F90:392:#if defined(IBM)
hg6.6.6_nudging_stdtrop/src/atmosphere/tracer_advection/interpolation_qcon_multi.F90:393:#if defined(IBM)

I don't know how significant these are. I will have to check them in case …..

Regards

Garry

comment:9 Changed 4 years ago by ros

  • Resolution set to fixed
  • Status changed from new to closed

Continued in #1649

Note: See TracTickets for help on using tickets.