negative mass in set_thermodynamic

[amenon]

Dear CMS,

I am getting the following error in one of my simulations:

???!!!???!!!???!!!???!!!???!!!       ERROR        ???!!!???!!!???!!!???!!!???!!!
?  Error code: 100
?  Error from routine: set_thermodynamic
?  Error message: A total of     4 points had negative mass in set_thermodynamic. This indicates the pressure fields are inconsistent between different levels and the model is about to fail.
?  Error from processor: 3615
?  Error number: 24
????????????????????????????????????????????????????????????????????????????????

I recognize that this is a known UM failure point as shown in ​https://code.metoffice.gov.uk/trac/um/wiki/KnownUMFailurePoints

I re-ran the LAM forecast a few times and they are persistently failing with this error. I also copied this suite fresh and tried to run it. But I am still getting this error. Could you please advice how to proceed with this now?

The suite id is u-bq536 in Archer.

Thanks,
Arathy

[dcase]

Would you mind giving me a path to the log file with the error? Wherever I look I see jobs succeeding.. Thanks

[amenon]

Thanks for looking into this, Dave. Here is the path:
/home/amenon/cylc-run/u-bq536/log/job/20180812T0000Z/INCOMPASS_km1p5_RA1T_qcons_um_fcst_000/05/job.err

[dcase]

Have you managed to run this with a shorter timestep? I can't see one which isn't 75s

[amenon]

Hi Dave,

As you suggested offline, I ran this simulation by reducing the time step to 25s and it succeeded one cycle without generating the above error. Now I have changed it back to 75s. I will let you know if the same error appears again.

Thanks,
Arathy

[dcase]

That's a nice result (I was hopeful rather than confident of that working).

If you are happy, then can we close the ticket? You can always re-open it if the error reappears and can't be solved like this.

[amenon]

Sure, we can close the ticket now. Thanks Dave.