Segmentation fault when using the PC2 cloud scheme

[Emma_Turner]

Hello,

I am trying to run a previously working job but with the new PC2 cloud scheme implemented. I followed the instructions in the documentation that show how to do this and have included a difference report to show the changes made at the bottom of this ticket. The new job is xihte, old is xihte. It starts running but crashes and produces a blank first time step. The .leave file shows the error:

_pmiu_daemon(SIGCHLD): [NID 02386] [c5-1c1s6n0] [Tue Jul 23 16:25:45 2013] PE RANK 27 exit signal Segmentation fault

Trying to attach xihte000.xihte.d13204.t145458.leave to this ticket.

Any help would be appreciated.

Thanks
Emma

Difference in window atmos_Science_Section_LSRain

→ Model Selection

→ Atmosphere

→ Scientific Parameters and Sections

→ Section by section choices

→ Section 4: Large Scale Precipitation

Radio button: Choose version

Job xihtd: Entry is set to '<3C> Improved microphysics scheme'
Job xihte: Entry is set to '<3D> Advanced microphysics scheme (recommended), compatible with PC2'

Difference in window atmos_Science_Section_Conv

→ Model Selection

→ Atmosphere

→ Scientific Parameters and Sections

→ Section by section choices

→ Section 5: Convection

Entry box: Defining anvil factor

Job xihtd: Entry is set to '3.0000'
Job xihte: Entry is set to '0.0000'

Check box: Use a phase change temperature in the plume other than 0 deg C

Job xihtd: Entry is set to 'OFF'
Job xihte: Entry is set to 'ON'

Entry box: Defining tower factor

Job xihtd: Entry is set to '0.2500'
Job xihte: Entry is set to '0.0000'

Entry box: Estimate saturation specific humidity at this phase change T / kg kg-1 (qstice). See Help.

Job xihtd: Entry is inactive
Job xihte: Entry is set to '3.5000e-03'

Entry box: Specify phase change temperature / K (Tice)

Job xihtd: Entry is inactive
Job xihte: Entry is set to '263.1500'

Difference in window atmos_Science_Section_Cloud

→ Model Selection

→ Atmosphere

→ Scientific Parameters and Sections

→ Section by section choices

→ Section 9: Large Scale Cloud

Hidden variable: Method of cloud area parametrization

Job xihtd: Variable is set to '1'
Job xihte: Entry is inactive

Check box: Including cloud area parametrization (Gen2 Rad only)

Job xihtd: Entry is set to 'ON'
Job xihte: Entry is set to 'OFF'

Check box: Use prognostic cloud scheme PC2?

Job xihtd: Entry is set to 'OFF'
Job xihte: Entry is set to 'ON'

Entry box: Enter value of cloud erosion / s-1

Job xihtd: Entry is inactive
Job xihte: Entry is set to '-2.25e-05'

[willie]

Hi Emma,

UDMP29 states

You will also need to be sure that you have PC2 compatable
options selected for the other physics schemes and, because PC2 is still being developed,
uptodate modsets. You are therefore advised to run with PC2 only after
seeking guidance.

It might be helpful to discuss this with Damian Wilson at the Met Office (cc'd).

I hope that helps.

Regards,

Willie

[Emma_Turner]

Hi Willie,

I spoke with Cyril Morcrette of cloud parameterisation before running the scheme, he advised the changes I made. Happy to talk with Damian about the other physics schemes if there are other bits that need to be changed. Thanks Damian!

Emma

[willie]

Hi Emma,

We need to get some more debug info to find out what's going on:

• In Output options, switch on subroutine timer diagnostics
• In compiler options for the model, select debug
• add a user machine over ride: this is a file (on PUMA) containing one line

%fflags_opt_debug -g

• In Atmos > scientific sections > section by section > section 13, select DIAG_PRN and tick flush buffer if run fails.

Then delete any file /work/d43/d43-geos-climate/eturner/um/xihte/M/core and rebuild and run the job again.

Regards,

Willie

[Emma_Turner]

Hi Willie,

The job has finished running with the extra options.

Thaks
Emma

[willie]

Could you let me have read permission on the core file please.

Willie

[Emma_Turner]

That file has read permissions but I think from looking back at emails regarding giving project access to yourself and grenvill that you have to join the d43 project if you haven't, then you read and write in the directory.

Let me know if there are any further problems.

Thanks
Emma

[willie]

Hi Emma,

The segmentation fault occurs in Atm_step at the first time step. It is trying to allocate some memory in

! ----------------------------------------------------------------------
 ! Section 6.0 Calculate u, v, w, theta, q, qcl, qcf at new time level.
 !             Extrapolate u, v, w, to time level n+1.5 for use in
 !             advection step on next timestep.
 ! ----------------------------------------------------------------------

This is very early on, and the error has probably occurred before this point. This suggests that the set up is at fault. I am afraid I am unable to go further than this.

I hope that helps,

Regards,

Willie

[Emma_Turner]

Hi Willie,

I've been speaking with Cyril Morcrette and we think that we need to tell the UM that we are using extra prognostics for the PC2 scheme in the initial dump. Therefore I think I need to add an extra STASHmaster file. I'm just reading through how to create these.. Can I have write access to the /home/umui/hadgem2/userstash directory so I can put it in here?

Thanks
Emma

[Emma_Turner]

Hi again,

I had a quick look at the help desk tickets and wondered if such a user STASHmaster file exists for these extra PC2 prognostics? These would be the cloud variables qcl, qcf, CFL, CFF, and BCF. This ticket http://cms.ncas.ac.uk/ticket/352 seems relevant but it relates to version 7.1. I wonder if an equivalent exists for version 6.6.3? Creating it myself does not look straight forward!

Thanks
Emma

[willie]

Hi Emma,

You can create local user STASH master files under your own account on PUMA. You can get the basics by copying entries you need from $UMDIR/vn8.2/ctldata/STASHmaster/STASHmaster_A, say. It is possible that the 7.1 version could suffice.

regards,

Willie

[Emma_Turner]

Hi Willie,

I have followed the instructions to create and add a user STASHmaster file (USMF) containing new prognostics, a copy of STASHmaster_PC2 is at the bottom of this comments. I initialised all fields to zero by specifying the code '3' in the Model Selection - Atmosphere - STASH - Initialisation of Prognostics window and added the file at the local location in User STASHmaster files, Diags, Progs & Ancils at ~/STASHmaster_PC2. I think they added because they appear in the PRESM_A file after the run. The run still failed with the same segmentation fault, can you check I did this USMF step correctly?

Thanks
Emma

H1| SUBMODEL_NUMBER=1
H2| SUBMODEL_NAME=ATMOS
H3| UM_VERSION=6.6.3
#
#|Model |Sectn | Item |Name |
#|Space |Point | Time | Grid |LevelT|LevelF|LevelL|PseudT|PseudF|PseudL|LevCom?|
#| Option Codes | Version Mask | Halo |
#|DataT |DumpP | PC1 PC2 PC3 PC4 PC5 PC6 PC7 PC8 PC9 PCA |
#|Rotate| PPF | USER | LBVC | BLEV | TLEV |RBLEVV| CFLL | CFFF |
#
#===============================================================================
#
1| 1 | 0 | 12 |QCF AFTER TIMESTEP |
2| 2 | 0 | 1 | 1 | 2 | 1 | 3 | 0 | 0 | 0 | 0 |
3| 000000000000000000000000000000 | 00000000000000000001 | 2 |
4| 1 | 2 | -24 -24 -24 -24 -99 30 -99 -99 -99 -99 |
5| 0 | 78 | 0 | 65 | 0 | 0 | 0 | 0 | 118 |
#
1| 1 | 0 | 254 |QCL AFTER TIMESTEP |
2| 2 | 0 | 1 | 1 | 2 | 1 | 3 | 0 | 0 | 0 | 0 |
3| 000000000000000000000000000000 | 00000000000000000001 | 2 |
4| 1 | 2 | -24 -24 -24 -24 -99 30 -99 -99 -99 -99 |
5| 0 | 79 | 0 | 65 | 0 | 0 | 0 | 0 | 117 |
#
1| 1 | 0 | 265 |AREA CLOUD FRACTION IN EACH LAYER |
2| 2 | 0 | 1 | 1 | 2 | 1 | 3 | 0 | 0 | 0 | 0 |
3| 000000000000000000000000000000 | 00000000000000000001 | 3 |
4| 1 | 2 | -6 -6 -6 -6 -6 16 -99 -99 -99 -99 |
5| 0 | 1729 | 0 | 65 | 0 | 0 | 0 | 0 | 0 |
#
1| 1 | 0 | 266 |BULK CLOUD FRACTION IN EACH LAYER |
2| 2 | 0 | 1 | 1 | 2 | 1 | 3 | 0 | 0 | 0 | 0 |
3| 000000000000000000000000000000 | 00000000000000000001 | 2 |
4| 1 | 2 | -6 -6 -6 -6 -6 16 -99 -99 -99 -99 |
5| 0 | 220 | 0 | 65 | 0 | 0 | 0 | 0 | 79 |
#
1| 1 | 0 | 267 |LIQUID CLOUD FRACTION IN EACH LAYER |
2| 2 | 0 | 1 | 1 | 2 | 1 | 3 | 0 | 0 | 0 | 0 |
3| 000000000000000000000000000000 | 00000000000000000001 | 2 |
4| 1 | 2 | -6 -6 -6 -6 -6 16 -99 -99 -99 -99 |
5| 0 | 1425 | 0 | 65 | 0 | 0 | 0 | 0 | 0 |
#
1| 1 | 0 | 268 |FROZEN CLOUD FRACTION IN EACH LAYER |
2| 2 | 0 | 1 | 1 | 2 | 1 | 3 | 0 | 0 | 0 | 0 |
3| 000000000000000000000000000000 | 00000000000000000001 | 2 |
4| 1 | 2 | -6 -6 -6 -6 -6 16 -99 -99 -99 -99 |
5| 0 | 1426 | 0 | 65 | 0 | 0 | 0 | 0 | 0 |
#
#===============================================================================
#
# Section 90 Reserved for post-processed derived fields
# 1| 1 | 90 | 201 |Reserved for Thickness fields for FS |
#
#===============================================================================
#
1| -1 | -1 | -1 |END OF FILE MARK |
2| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
3| 000000000000000000000000000000 | 00000000000000000000 | 0 |
4| 0 | 0 | -99 -99 -99 -99 -30 -99 -99 -99 -99 -99 |
5| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
#

[willie]

Hi Emma,

It seems OK, but the items that you've introduces are already present in the 6.6.3 STASH master file. Perhaps the problem is elsewhere?

I hope that helps.

regards

Willie