How can I use CrayPAT with UM vn8.6 on XCM

[markr]

Hello CMS,
I would like to use craypat to analyse the performance of the UM.
I have activated OpenMP within the UKCA component and would like a clearer report on where workload is.

Do you have any recommendation on what I can do within the UMUI to achieve that?

I will need to do a process similar to this:

(1) activate the perftools module during build and run stages
(2) export some environment variables also during build and run
(3) post build I will have to process the executable with the pat_build command and some options to create an instrumented exec with different name
(4) run the different exec and save the resulting *.xf *.apa files
(5) re-process the exec with pat_build and create a different one with name+apa
(6) submit a run for the new exec for tracing experiment.

Some questions include: is the exec built next to the objects as pat_build and pat_report need to see those?
As for drhook I believe the expt files (*.xf) will appear in the run directory.

[ros]

Email received from MO:

Hi Mark,
I have not attempted to run CRAYpat with the UMUI, however, it should be perfectly possible - assuming you're using a version which has been ported to the Cray (hg6.6.6 and vn8.2 - vn8.5 has been run unofficially but with handedits and a UMDIR override).

(1) You'll need to handedit a "module load perftools" into the top of FCM_BLD_COMMAND (I think, for build) and SCRIPT (for run).

(2) At later UMUI versions (including vn8.2) you will be able to export user defined environment variables to both the build and the run fro the UMUI. Earlier versions will require you hand edit them in to the build script (FCM_BLD_COMMAND).

At all versions user defined variables can be exported to the run via the UMUI panel, and will end up in SCRIPT.

(3) Set the UMUI to do a build and stop, then do your postprocessing. You can set the name of the exec to whatever you want in the build panel.

(4) Point the UMUI at the appropriate exec from the build panel, and select "run only"

(5) and (6) Repeat the above.

If you need me to be more specific let me know what version you're running with.

[ros]

Closing this ticket for now. Mark please reopen or start a new ticket if you need anymore help with this.

Cheers,
Ros.

[markr]

Hello CMS,
I am back on this. On Monsoon I did this: set the flags "-h keepfiles -hpl=xlsqg.pl" as directed by CMS notes.

$ pat_build -O apa xlsqg.exe
INFO: A maximum of 130 functions from group 'caf' will be traced.
INFO: A maximum of 154 functions from group 'mpi' will be traced.
INFO: A maximum of 25 functions from group 'realtime' will be traced.
INFO: A maximum of 63 functions from group 'syscall' will be traced.
INFO: A maximum of 100 functions from group 'upc' will be traced.
/opt/cray/cce/8.3.4/cray-binutils/x86_64-unknown-linux-gnu/bin/ld: cannot find -l__fcm__xlsqg
/opt/cray/cce/8.3.4/cray-binutils/x86_64-unknown-linux-gnu/bin/ld: cannot find -lalpslli
/opt/cray/cce/8.3.4/cray-binutils/x86_64-unknown-linux-gnu/bin/ld: cannot find -lalpsutil
FATAL: The executable '/opt/cray/cce/8.3.4/cray-binutils/x86_64-unknown-linux-gnu/bin/ld' returned error status 0x100.

I made some progress with the help of Cray:

$ pat_build -w -D link-instr=-L/opt/cray/alps/default/lib64 xlsqg.exe
/opt/cray/cce/8.3.4/cray-binutils/x86_64-unknown-linux-gnu/bin/ld: cannot find -l__fcm__xlsqg
FATAL: The executable '/opt/cray/cce/8.3.4/cray-binutils/x86_64-unknown-linux-gnu/bin/ld' returned error status 0x100.
[xcml00:mricha:bin]

When I look in the comp.leave log I see that

ar: creating /projects/ukca-leeds/mricha/xlsqg/umatmos/tmp/lib__fcm__xlsqg.a
ftn -o xlsqg.exe /projects/ukca-leeds/mricha/xlsqg/umatmos/obj/um_main.o /projects/ukca-leeds/mricha/xlsqg/umatmos/obj/blkdata.o -L/projects/ukca-leeds/mricha/xlsqg/umatmos/lib -L/projects/ukca-leeds/mricha/xlsqg/baserepos/JULES/lib -L/projects/ukca-leeds/mricha/xlsqg/baserepos/JULES/lib -L/projects/ukca-leeds/mricha/xlsqg/baserepos/UMATMOS/lib -l__fcm__xlsqg -h keepfiles -hpl=xlsqg.pl -L/projects/um1/gcom/gcom4.6/meto_cray_xc40_mpp/build/lib -lgcom  -LNetCDFmodule -lnetcdf  -L/projects/um1/grib_api/cce-8.3.4/1.13.0/lib -lgrib_api_f90 -lgrib_api -h omp

However, this tmp file is no longer there (bt the directory is) and the "in-source" UMATMOS/lib directory is not there either (in link line).

I believe all this is because the build system moves the files and at some point deletes them separately to the ftn so keepfiles is not obeyed. (I think).

Cheers,
Mark

[ros]

Hi Mark,

FCM deletes the interim files and archive automatically and under FCM1 there's not a simple switch to turn this off. I will need to hack a version of FCM for you.

Cheers,
Ros.

[markr]

Okay Ros,
thank you for the reply.
Is this specific to vn8.x ?
What level of FCM is associated with vn10.4 ?
Should I focus on those and "modifying rose suites" ?
I will need to get further guidance on that.

Cheers

Mark

[ros]

Mark started work on a later 10.x (Rose) version. If CrayPAT required for vn8.x then a modification to FCM will need to be done to force FCM to keep the interim files required by CrayPAT.