Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#1873 closed help (fixed)

L_TRACER_MASS is unrecognized in namelist input

Reported by: ucfaako Owned by: ros
Component: UM Model Keywords: hand edits, namelists
Cc: Platform: ARCHER
UM Version: 6.6.3

Description

Hello,
I'm trying to get a job I converted from the MetOffice? to run on ARCHER (xmfch). The command L_TRACER_MASS= .TRUE in the hand edit /home/ucfaako/umui_jobs/aniym.monsoon/HAND_EDITS/tracer_mass.ed however seems to cause some trouble in the namelists (see /home/n02/n02/akncas/um/umui_out/xmfch000.xmfch.d1627.t173110.archive. This webpage recommends removing it from the namelists, is that still up-to-date?
Would setting is to L_TRACER_MASS= .FALSE just do the job or would that also disable L_CO2_EMITS?

Thank you very much for your help!

Best wishes,
Alex

Attachments (1)

Change History (35)

comment:1 Changed 4 years ago by ucfaako

According to ticket #633, it looks like a branch is missing, so following up on that I checked my branches (fcm info) and realised the fcm metadata (fcm URL, ownership etc.) haven't changed from the Met Office repository.
I converted them using create_HG2_branch and then following the FAQ entry from step 3 on. One issue might be that I had to comment out the 4 lines after #Commands to apply patches as the revision number didn't seem to be correct. Could that be the culprit?
Branches are in /home/ucfaako/branches.
Any help on this would be greatly appreciated!

Many thanks,
Alex

Last edited 4 years ago by ucfaako (previous) (diff)

comment:2 Changed 4 years ago by ros

Hi Alex,

The problem with your working copies in /home/ucfaako/branches are that you have checked out your new branch into the same directory as the patch which is causing a problem. You should keep the FCM patches and branches separate.

If you look in the Trac (e.g. https://puma.nerc.ac.uk/trac/UM/log/UM/branches/dev/ucfaako/hg6.6.3_scale_land_CO2_flux_by_land_frac?rev=21009) you will see that the patches have not been applied.

For step 4 you don't need to checkout the branches first, just apply them straight to the repository by supplying the URL.

Try running the following command (hopefully the patches will still work with the other stuff in there)

E.g. for hg6.6.3_LUE_to_atm

/home/ucfaako/branches/hg6.6.3_LUE_to_atm/fcm-import-patch svn://puma/UM_svn/UM/branches/dev/ucfaako/hg6.6.3_LUE_to_atm

If you get any errors at all please post them into this ticket.
Do similar for each patch.

Cheers,
Ros.

comment:3 Changed 4 years ago by ucfaako

Hi Ros,

Thanks for you quick reply! I ran the fcm-import-patch again from my home directory and it finished normally (fcm-import-patch: finished normally.). But when looking at fcm info it seems that the patches are still not applied.
With same directory you mean that e.g. for hg6.6.3_LUE_to_atm, both, patch and branch are checked out in /home/ucfaako/branches/hg6.6.3_LUE_to_atm, whereas the FCM branch should be checked out in /branches/ and the FCM patch in /branches/hg6.6.3_LUE_to_atm?
Would it make sense to delete the working copies (I still have backups), copy them back into the /branches/ folder and keep FCM patches and branches separate in the first place?

Cheers,
Alex

comment:4 Changed 4 years ago by ros

Hi Alex,

I can see the patch still didn't work as nothing was committed to the repository.

By "same directory", I do indeed mean that. I think that the best idea would be to start again. Delete what you have, put the patches in one place. Then run the fcm-import-patch command - no need to checkout the branch first.

You should see lines like:

Transmitting file data ......
Committed revision 21020.
At revision 21020.

If it doesn't work please paste all the output here.

Cheers,
Ros.

comment:5 Changed 4 years ago by ucfaako

Hi Ros,

I've tried several combinations of branch-working copy-patch locations (patch in working copy folder & patch in separate folder; I created the branch always in /branches/, never in a working copy folder) and there are two outcomes independent from the patch location:
When I leave the fcm-import-patch script as it is I usually get the following error (here for hg6.6.3_scale_land_CO2_flux_by_land_frac:

ucfaako@puma:/home/ucfaako/branches> /home/ucfaako/branches/hg6.6.3_scale_land_CO2_flux_by_land_frac_patch/fcm-import-patch svn://puma/UM_svn/UM/branches/dev/ucfaako/hg6.6.3_scale_land_CO2_flux_by_land_frac
apply-patch: Applying patch for changeset 30728.
patching file src/atmosphere/boundary_layer/bl_trmix_dd.F90
Hunk #1 FAILED at 1147.
1 out of 1 hunk FAILED -- saving rejects to file src/atmosphere/boundary_layer/bl_trmix_dd.F90.rej

After commenting out the 4 lines after #Commands to apply patches in the fcm-import-script as suggested here the script doesn't return an error anymore (fcm-import-patch: finished normally.), but I it also doesn't apply any patches. I assume that's because I commented the commands out? I just thought it might be worth a try, but didn't work.

Many thanks for your help!

Cheers,
Alex

comment:6 Changed 4 years ago by ros

  • Owner changed from um_support to ros
  • Status changed from new to accepted

Hi Alex,

Unfortunately, the patches have not been created properly. They only contain the final changeset rather than the whole branch.

I have regenerated the patch for hg6.6.3_scale_land_CO2_flux_by_land_frac to include the 3 changesets 30721, 30726 and 30728. You can find it on PUMA at ~ros/temp/hg6.6.3_scale_land_CO2_flux_by_land_frac_patch.tgz

If you let me know the URLs of the other 2 Met Office branches I'll get them for you.

Cheers,
Ros.

comment:7 Changed 4 years ago by ucfaako

Hi Ros,

Oh that explains a lot. Thank you ever so much for fixing them! I've just re-uploaded the remaining branches (hg6.6.3_LUE_to_atm and r24306_CONSERVE_ATMOS_CO2_MAY_2013), they are now in /home/ucfaako/branches/.

Cheers,
Alex

comment:8 Changed 4 years ago by ucfaako

Just tried the fcm-import-patch for scale_land_CO2_flux_by_land_frac, it seems to have worked (https://puma.nerc.ac.uk/trac/UM/browser/UM/branches/dev/ucfaako/hg6.6.3_scale_land_CO2_flux_by_land_frac?rev=21027)

Cheers,
Alex

ucfaako@puma:/home/ucfaako/branches> ./hg6.6.3_scale_land_CO2_flux_by_land_frac_patch/fcm-import-patch svn://puma/UM_svn/UM/branches/dev/ucfaako/hg6.6.3_scale_land_CO2_flux_by_land_frac
apply-patch: Applying patch for changeset 30721.
patching file src/atmosphere/boundary_layer/bl_trmix_dd.F90
apply-patch: finished normally.
Sending        src/atmosphere/boundary_layer/bl_trmix_dd.F90
Transmitting file data .
Committed revision 21027.
Updating '.':
At revision 21027.
apply-patch: Applying patch for changeset 30726.
patching file src/atmosphere/boundary_layer/bl_trmix_dd.F90
apply-patch: finished normally.
Sending        src/atmosphere/boundary_layer/bl_trmix_dd.F90
Transmitting file data .
Committed revision 21028.
Updating '.':
At revision 21028.
apply-patch: Applying patch for changeset 30728.
patching file src/atmosphere/boundary_layer/bl_trmix_dd.F90
apply-patch: finished normally.
Sending        src/atmosphere/boundary_layer/bl_trmix_dd.F90
Transmitting file data .
Committed revision 21029.
Updating '.':
At revision 21029.
fcm-import-patch: finished normally.

comment:9 Changed 4 years ago by ros

Hi Alex,

The patch for hg6.6.3_LUE_to_atm is at: /home/ros/temp/hg6.6.3_LUE_to_atm_patch.tgz

r24306_CONSERVE_ATMOS_CO2_MAY_2013 is a bit more complicated as it is actually a copy of another branch with changes on top. So once I've worked out the easiest way to get that one I'll send you instructions.

Cheers,
Ros.

comment:10 Changed 4 years ago by ros

Hi Alex,

I've put the branch into the repository for you - it's at fcm:um-br/dev/Share/hg6.6.3_CONSERVE_ATMOS_CO2_MAY_2013

If you would prefer to have it in your own area here's what I did (all patches are under /home/ros/temp/for_alex):

  • Create hg6.6.3_CONSERVE_ATMOS_CO2 branch using create_HG2_branch
  • Apply the HG6.6.3_CONSERVE_ATMOS_CO2 patch
    ./HG6.6.3_CONSERVE_ATMOS_CO2_patch/fcm\-import\-patch svn://puma/UM_svn/UM/branches/dev/Share/hg6.6.3_CONSERVE_ATMOS_CO2
  • Copy hg6.6.3_CONSERVE_ATMOS_CO2 branch to the new hg6.6.3_CONSERVE_ATMOS_CO2_MAY_2013
    fcm copy fcm:um-br/dev/Share/hg6.6.3_CONSERVE_ATMOS_CO2 fcm:um-br/dev/Share/hg6.6.3_CONSERVE_ATMOS_CO2_MAY_2013
  • Apply r24306_CONSERVE_ATMOS_CO2_MAY_2013 patch to the branch created in previous step
    ./r24306_CONSERVE_ATMOS_CO2_MAY_2013_patch/fcm\-import\-patch svn://puma/UM_svn/UM/branches/dev/Share/hg6.6.3_CONSERVE_ATMOS_CO2_MAY_2013

Cheers,
Ros.

comment:11 Changed 4 years ago by ucfaako

Hi Ros,

That is totally fine by me, but thank you very much for showing me how to get to the result and for all your help.
I've copied the patches from /home/ros/temp/for_alex to my own directory, in case you need the space.

Let's hope that will fix the L_TRACER_MASS issue.
Again, many thanks!

Cheers,
Alex

comment:12 Changed 4 years ago by ucfaako

Hi Ros,

The error message disappeared.
Of course only to be replaced with a new one: "unit 56 - No such file or directory" (see /home/n02/n02/akncas/um/umui_out/xmfch000.xmfch.d16138.t164002.archive)
It appears to be similar to ticket #778. I checked to file and directory for the volcanic forcing file and it seems to be fine.

Would you like me to open a new ticket for this?

Cheers,
Alex

comment:13 Changed 4 years ago by ros

Hi Alex,

There are annoyingly a few places in the UM where you need to specify absolute path names and not use environment variables. Volcanic forcing and solar forcing files are 2 of these. If you replace $DATADIR with /work/n02/n02/akncas in UMUI window Atmos → Ancil and input data files → Climatologies & potential climatologies → Natural climate forcing you should find it will then find those files.

Cheers,
Ros.

comment:14 Changed 4 years ago by ucfaako

Hi Ros,

Thanks, that was exactly the problem. I feel we're getting close to actually running the job. :)
There weren't any apparent issues when I last ran it but it aborted and I'm not sure what went wrong as I couldn't find an error message in the .leave/.archive file (/work/n02/n02/akncas/um/um_umui/xmfch000.xmfch.d16141.t190144.archive). Does the warning message about volcanic forcing / NI_rad_ct13c (L20174) have anything to do with it?

Many thanks for all your help!

Cheers,
Alex

comment:15 Changed 4 years ago by ucfaako

Update:
I realised I hadn't set up the archiving on Archer properly (http://cms.ncas.ac.uk/wiki/Archer/NercArchiving), so I set it up and re-run the simulation. It still failed, but I didn't expect it to be the cause of the problem.

The most up-to-date .leave/.archive file is: /work/n02/n02/akncas/um/um_umui/xmfch000.xmfch.d16144.t114559.archive

Cheers,
Alex

comment:16 Changed 4 years ago by grenville

Alex

In /work/n02/n02/akncas/xmfch/dataw/xmfch.fort6.pe12 (for example)

ISCCP - diagnostics not available externally

Either switch off the ISCCP diags in STASH
or contact the code author for details

*
UM ERROR (Model aborting) :
Routine generating error: ISCCP
Error code: 1
Error message:

Routine should not be called

Grenville

comment:17 Changed 4 years ago by ucfaako

Thanks Grenville, turned off ISCCP diags (ISCCP CLOUD WEIGHTS; ISCCP CLOUD 0.3 <= tau to ISCCP CLOUD 60 <= tau) in STASH and the error disappeared.

It now however complains that the treatment for canopy radiation is set to 0. I checked the UM and it's set to 2. See for example /work/n02/n02/akncas/xmfch/dataw/xmfch.fort6.pe35:

CAN_RAD_MOD ERROR:
 CAN_RAD_MOD =  0
 THIS VALUE IS NOT SUPPORTED - CHECK SETUP
 THE VARIABLE NITROGEN MOD IS ONLY DESIGNED FOR
 USE HERE WITH CAN_RAD_MOD=1,2
 
 THE MODEL RUN HAS BEEN STOPPED IN sf_stom
 *********************************************************************************
 UM ERROR (Model aborting) :
 Routine generating error: sf_stom.F90                                                                                                                                                                                                                                                     
 Error code:  2
 Error message: 
 CAN_RAD_MOD value disallowed in sf_stom

There is also this warning message in the .archive file that sounds worrying (/home/n02/n02/akncas/um/um_umui/xmfch000.xmfch.d16147.t141651.archive):

 ****************** WARNING *******************
 ** Conservation enforcement failed          **
 ** Run continuing using best estimate       **
 ****************** WARNING *******************
  non-conservation for field  24

Many thanks for all your help so far - sorry I didn't know that it would be so tricky to get a Met Office CMIP5 run to work on ARCHER and learning lots of new stuff here, so thanks for your support!

Cheers,
Alex

comment:18 Changed 4 years ago by ucfaako

Hi Ros and Grenville,

Sorry to bug you but has one of you had a chance to look at this? Would it make sense to open a new thread for this problem?

Many thanks,
Alex

comment:19 Changed 4 years ago by ros

Hi Alex,

Sorry for the delay. I'll try and take a look this afternoon.

Cheers,
Ros.

comment:20 Changed 4 years ago by ros

Hi Alex,

Do you have a copy of this job in the state it came from the Met Office? I see that xmfch was a copy of xmfcg and you've then made stash changes but xmfcg has since been deleted. If not can you confirm that it was /home/ucfaako/umui_jobs/aniym.monsoon/basis_aniym that you loaded in orginally for this job?

Cheers,
Ros.

comment:21 Changed 4 years ago by ucfaako

Hi Ros,

If I remember correctly I think I created this job from /home/ucfaako/umui_jobs/aniym.monsoon/umui_jobs/aniym/basis_aniym, which was a .gz file that I unpacked (hence the cp_basis_aniym.gz). I somehow missed that there was already a basis file in /home/ucfaako/umui_jobs/aniym.monsoon/ when I started working on this run.
Do you think it's worth recreating the job with the /home/ucfaako/umui_jobs/aniym.monsoon/basis_aniym?

Cheers,
Alex

comment:22 Changed 4 years ago by ucfaako

Hi Ros,

I re-created xmfch with /home/ucfaako/umui_jobs/aniym.monsoon/basis_aniym (called xmfck) but the error still persists (see leave file /home/n02/n02/akncas/um/umui_out/xmfck000.xmfck.d16158.t174033.archive). There's also been an error about FF2PP_HECTOR when I checked to setup of xmfck:

Errors will be output in this window
List Check Error in window subindep_PostProc_Gen 
Variable: FF2PP_HECTOR
 -> Model Selection
   -> Sub-Model Independent
     -> Post Processing
       -> Main Switch + General Questions
Verification is complete.  
If an error was detected then find the window, enter and close it. 
This will either generate a more informative error message or it will 
result in the setting of a previously unset hidden variable.

I've followed the Archer/NercArchiving guide.

Cheers,
Alex

comment:23 Changed 4 years ago by ros

Hi Alex,

This is looking like an overwriting problem which we will need to take and debug.

Can you please confirm that this exact configuration has been run successfully on the Met Office Cray? We would expect them to see a similar problem.

Regards,
Ros.

comment:24 Changed 4 years ago by ucfaako

Hi Ros,

Yes, /home/ucfaako/umui_jobs/aniym.monsoon/basis_aniym reflects the exact configuration as used by the Met Office for the simulations from the HadGEM2-ES CMIP5 effort. I just transferred it, including all the handedits, ancils etc. The only difference is the absence of hand edit ~hadsm/handedits/prg_12_1_0_10, which told the previous Met Office supercomputer which compiler to use and is apparently not needed on Archer.
The person who conducted the runs at the Met Office is Eddy Robertson. I can put you in touch with him if that helps?

Many thanks,
Alex

comment:25 Changed 4 years ago by ros

Hi Alex,

If you could give me Eddy's email that would be great.

The Met Office's previous computer was an IBM so that could behave differently.

Cheers,
Ros.

comment:26 Changed 4 years ago by ucfaako

Sent an email to both of you!

Cheers,
Alex

comment:27 Changed 4 years ago by grenville

Alex

You are using the wrong revision number for

fcm:um_br/dev/Share/hg6.6.3_CONSERVE_ATMOS_CO2_MAY_2013/src

it should be 21073.

You will have other problems with UKCA photolysis files because the paths are hard wired in the code(!!), for example in

ukca_phot2d.f90 file2='/home/cr/cce/hadfo/um/tropdata/photol/'

I guess you can fix these issues as they arise.

Grenville

comment:28 Changed 4 years ago by grenville

Alex

We may have the files here:

/work/n02/n02/hum/hg6.6.4/ctldata/UKCA/tropdata/photol

Grenville

comment:29 Changed 4 years ago by ucfaako

Hi Grenville,

Thanks, changed the revision number and started the run, lets see with what error it comes up next. I also checked the code, it seems that there are three options and the first two are commented out so it points to /work/n02/n02/hum/hg6.6.3/HG2ES_ancils/UKCA which seems to have all the files in there (hopefully).

All the best,
Alex

comment:30 Changed 4 years ago by grenville

Alex

I was mislead a bit by the error message - the problem is with PHOT2D_DIR which is set to

$UMDIR/hg6.6.3/HG2ES_ancils/UKCA

in the UKCA window.

the model can't understand $UMDIR - pl use the fill path

Grenville

comment:31 Changed 4 years ago by ucfaako

Hi Grenville,

Just checked, the "Directory pathname for the 2D photolysis rates" in UKCA was already set to the full path name, but I now also changed $HG2ES_ANICLS in the file & directory/environment variables to the full path.

Many thanks,
Alex

Changed 4 years ago by ucfaako

comment:32 Changed 4 years ago by ucfaako

Hi Ros and Grenville,

The run successfully starts now until I run out of disk quota (see attached .archive file). Is there any chance you could increase my quota on /work/ or is there a better way to store ancils and dumps for my run? The surface and aircraft emissions files are huge despite being already chunked into 20-year slices.

Thank you both ever so much for all your efforts!

Alex

comment:33 Changed 4 years ago by grenville

  • Resolution set to fixed
  • Status changed from accepted to closed

comment:34 Changed 4 years ago by grenville

Alex

Increased to 100GB - let us know if that's insufficeint

Note: See TracTickets for help on using tickets.