Opened 5 years ago

Closed 4 years ago

#1711 closed help (answered)

How can I use CrayPAT with UM vn8.6 on XCM

Reported by: markr Owned by: ros
Component: UM Model Keywords: craypat, perftools, pat_build
Cc: Platform: MONSooN
UM Version: 8.6


Hello CMS,
I would like to use craypat to analyse the performance of the UM.
I have activated OpenMP within the UKCA component and would like a clearer report on where workload is.

Do you have any recommendation on what I can do within the UMUI to achieve that?

I will need to do a process similar to this:

(1) activate the perftools module during build and run stages
(2) export some environment variables also during build and run
(3) post build I will have to process the executable with the pat_build command and some options to create an instrumented exec with different name
(4) run the different exec and save the resulting *.xf *.apa files
(5) re-process the exec with pat_build and create a different one with name+apa
(6) submit a run for the new exec for tracing experiment.

Some questions include: is the exec built next to the objects as pat_build and pat_report need to see those?
As for drhook I believe the expt files (*.xf) will appear in the run directory.

Change History (6)

comment:1 Changed 5 years ago by ros

Email received from MO:

Hi Mark,
I have not attempted to run CRAYpat with the UMUI, however, it should be perfectly possible - assuming you're using a version which has been ported to the Cray (hg6.6.6 and vn8.2 - vn8.5 has been run unofficially but with handedits and a UMDIR override).

(1) You'll need to handedit a "module load perftools" into the top of FCM_BLD_COMMAND (I think, for build) and SCRIPT (for run).

(2) At later UMUI versions (including vn8.2) you will be able to export user defined environment variables to both the build and the run fro the UMUI. Earlier versions will require you hand edit them in to the build script (FCM_BLD_COMMAND).

At all versions user defined variables can be exported to the run via the UMUI panel, and will end up in SCRIPT.

(3) Set the UMUI to do a build and stop, then do your postprocessing. You can set the name of the exec to whatever you want in the build panel.

(4) Point the UMUI at the appropriate exec from the build panel, and select "run only"

(5) and (6) Repeat the above.

If you need me to be more specific let me know what version you're running with.

comment:2 Changed 5 years ago by ros

  • Resolution set to answered
  • Status changed from new to closed

Closing this ticket for now. Mark please reopen or start a new ticket if you need anymore help with this.


comment:3 Changed 5 years ago by markr

  • Resolution answered deleted
  • Status changed from closed to reopened

Hello CMS,
I am back on this. On Monsoon I did this: set the flags "-h keepfiles" as directed by CMS notes.

$ pat_build -O apa xlsqg.exe
INFO: A maximum of 130 functions from group 'caf' will be traced.
INFO: A maximum of 154 functions from group 'mpi' will be traced.
INFO: A maximum of 25 functions from group 'realtime' will be traced.
INFO: A maximum of 63 functions from group 'syscall' will be traced.
INFO: A maximum of 100 functions from group 'upc' will be traced.
/opt/cray/cce/8.3.4/cray-binutils/x86_64-unknown-linux-gnu/bin/ld: cannot find -l__fcm__xlsqg
/opt/cray/cce/8.3.4/cray-binutils/x86_64-unknown-linux-gnu/bin/ld: cannot find -lalpslli
/opt/cray/cce/8.3.4/cray-binutils/x86_64-unknown-linux-gnu/bin/ld: cannot find -lalpsutil
FATAL: The executable '/opt/cray/cce/8.3.4/cray-binutils/x86_64-unknown-linux-gnu/bin/ld' returned error status 0x100.

I made some progress with the help of Cray:

$ pat_build -w -D link-instr=-L/opt/cray/alps/default/lib64 xlsqg.exe
/opt/cray/cce/8.3.4/cray-binutils/x86_64-unknown-linux-gnu/bin/ld: cannot find -l__fcm__xlsqg
FATAL: The executable '/opt/cray/cce/8.3.4/cray-binutils/x86_64-unknown-linux-gnu/bin/ld' returned error status 0x100.

When I look in the comp.leave log I see that

ar: creating /projects/ukca-leeds/mricha/xlsqg/umatmos/tmp/lib__fcm__xlsqg.a
ftn -o xlsqg.exe /projects/ukca-leeds/mricha/xlsqg/umatmos/obj/um_main.o /projects/ukca-leeds/mricha/xlsqg/umatmos/obj/blkdata.o -L/projects/ukca-leeds/mricha/xlsqg/umatmos/lib -L/projects/ukca-leeds/mricha/xlsqg/baserepos/JULES/lib -L/projects/ukca-leeds/mricha/xlsqg/baserepos/JULES/lib -L/projects/ukca-leeds/mricha/xlsqg/baserepos/UMATMOS/lib -l__fcm__xlsqg -h keepfiles -L/projects/um1/gcom/gcom4.6/meto_cray_xc40_mpp/build/lib -lgcom  -LNetCDFmodule -lnetcdf  -L/projects/um1/grib_api/cce-8.3.4/1.13.0/lib -lgrib_api_f90 -lgrib_api -h omp

However, this tmp file is no longer there (bt the directory is) and the "in-source" UMATMOS/lib directory is not there either (in link line).

I believe all this is because the build system moves the files and at some point deletes them separately to the ftn so keepfiles is not obeyed. (I think).


comment:4 Changed 4 years ago by ros

  • Owner changed from um_support to ros
  • Status changed from reopened to accepted

Hi Mark,

FCM deletes the interim files and archive automatically and under FCM1 there's not a simple switch to turn this off. I will need to hack a version of FCM for you.


comment:5 Changed 4 years ago by markr

Okay Ros,
thank you for the reply.
Is this specific to vn8.x ?
What level of FCM is associated with vn10.4 ?
Should I focus on those and "modifying rose suites" ?
I will need to get further guidance on that.



comment:6 Changed 4 years ago by ros

  • Resolution set to answered
  • Status changed from accepted to closed

Mark started work on a later 10.x (Rose) version. If CrayPAT required for vn8.x then a modification to FCM will need to be done to force FCM to keep the interim files required by CrayPAT.

Note: See TracTickets for help on using tickets.