Opened 2 months ago

Closed 8 days ago

#3330 closed help (fixed)

AutoAssess on JASMIN

Reported by: annette Owned by: pmcguire
Component: Data Keywords: AutoAssess, JASMIN
Cc: Platform: JASMIN
UM Version:

Description

Has anyone run AutoAssess on JASMIN?
Annette O.

Attachments (2)

0010_PMSL_djf_v_era-interim.png (446.1 KB) - added by pmcguire 2 months ago.
Screenshot 2020-07-20 at 13.27.40.png (888.7 KB) - added by pmcguire 2 months ago.

Download all attachments as: .zip

Change History (77)

comment:1 Changed 2 months ago by pmcguire

Yes, I have run AutoAssess on JASMIN.

My most recent setup, including preprocessing scripts for supermeans is in jasmin-sci1: ~pmcguire/autoassess3a.

Patrick

comment:2 Changed 2 months ago by pmcguire

  • Owner changed from um_support to pmcguire
  • Status changed from new to accepted

comment:3 Changed 2 months ago by pmcguire

Thanks Patrick
I will look at that. I might message you if I have questions.
Annette

comment:4 Changed 2 months ago by pmcguire

AutoAssess has the advantage over !ESMValTool that it already has a lot of pre-programmed recipes embedded in it, and the people who have used AutoAssess before are familiar with the page layout for these recipes.
Patrick

comment:5 Changed 2 months ago by pmcguire

It might be as simple as modifying the latest .scr AutoAssess script file, and inserting the settings for your experiment run and your control run. At least to start.
Patrick

comment:6 Changed 2 months ago by pmcguire

Ah thanks that sounds promising…
Annette

comment:7 Changed 2 months ago by pmcguire

You probably also need to run the Python supermeans preprocessing script before running the AutoAssess script.
(Unless you already have computed supermeans).
Patrick

comment:8 Changed 2 months ago by pmcguire

Does AutoAssess have documentation somewhere?
Annette

comment:9 Changed 2 months ago by pmcguire

We haven't computed the supermeans. And the data is quite large so I think it would take some time. I am not totally sure what PL wants to compare, so I might just have a look at one of your scripts to get an idea.
Annette

comment:10 Changed 2 months ago by pmcguire

I checked out the code and built the documentation so just looking at that.
Annette

comment:11 Changed 2 months ago by pmcguire

Have you seen this?:
https://code.metoffice.gov.uk/trac/cma/wiki/OldAutoAssess/JASMIN
I updated that previously, for the things I had to change to get it working.
Patrick

comment:12 Changed 2 months ago by pmcguire

Ah no I didn't see that thanks!
Annette

comment:13 Changed 2 months ago by pmcguire

Do have access to the esmeval GWS?
Patrick

comment:14 Changed 2 months ago by pmcguire

No I don't.
Annette

comment:15 Changed 2 months ago by pmcguire

Maybe you can apply for it.
The obs data is in there.
Patrick

comment:16 Changed 2 months ago by pmcguire

Yes I will request access.
Annette

comment:17 Changed 2 months ago by pmcguire

Here's another link I got from Robin Smith who was one of the people I got help from on this:
https://code.metoffice.gov.uk/trac/cma

There's also an esmeval virtual machine which maybe it's worthwhile to apply for access to.
But I think you can run the AutoAssess scripts that I pointed you to from jasmin-sci1, and you don't need access to the esmeval VM.

And you might not need to compile or change anything if you use the AutoAssess scripts/directory that I pointed you to. The only changes you might need to make is in the parent script that defines where your data is at.
Patrick

comment:18 Changed 2 months ago by pmcguire

OK yes I will try using your scripts first.
Annette

comment:19 Changed 2 months ago by pmcguire

Do you load something to set up your environment to get all the python packages? For the supermean script.
Annette

comment:20 Changed 2 months ago by pmcguire

let me check.
Patrick

comment:21 Changed 2 months ago by pmcguire

I just copied the lines that load the Python libraries from supermean4b.py to test_libs.py, and then did python2.7 < test_libs.py, and that works. I didn't start any anaconda environment or anything. I am not sure right now if I had installed any of those libraries in test_libs.py as site-packages or anything. Maybe you can do that same quick check python2.7 < test_libs.py? It doesn't test everything, but at least all the libraries loaded for me without complaining.
Patrick

comment:22 Changed 2 months ago by pmcguire

Btw I need to look at the jasmin scripts again later. But the reason I messaged you is that I was getting errors loading some of the modules.

I think I got further loading jaspy but there was still something else which I can't recall off the top of my head.
Annette

comment:23 Changed 2 months ago by pmcguire

I am running it on jasmin-sci1. The jaspy module is loaded already on jasmin-sci1, I think. I don't load it myself.
I am using python2.7 without site-packages and without starting an anaconda environment.

does the python2.7 < test_libs.py work?
Patrick

comment:24 Changed 2 months ago by pmcguire

Let me try again then I might have been on the wrong server. I will try your test.
Annette

comment:25 Changed 2 months ago by pmcguire

jasmin-sci1 has less memory than jasmin-sci3, so using jasmin-sci3 might help you?
Patrick

comment:26 Changed 2 months ago by pmcguire

If I do python2.7 < test_libs.py it works but when I try to run the supermean script it can't find the braceexpand module.
Annette

comment:27 Changed 2 months ago by pmcguire

ah yeah.

I remember that problem.

There's a braceexpand python script in the autoassess3a directory. Can you copy that over, if you haven't already?
Patrick

comment:28 Changed 2 months ago by pmcguire

Oh doh ! I didn't spot that thank you. […]
Annette

comment:29 Changed 2 months ago by pmcguire

that's ok.
I had to download that braceexpand script from somewhere back in February, in order to get the supermeans script to work.
Patrick

comment:30 Changed 2 months ago by pmcguire

Seems to be working now. I have just set it to do the first season and 1 stream to test before I do the whole lot.
Annette

comment:31 Changed 2 months ago by pmcguire

Excellent!
Patrick

comment:32 Changed 2 months ago by pmcguire

did the supermeans script work and produce reasonable output? Did you try the AutoAssess script yet?
Patrick

comment:33 Changed 2 months ago by pmcguire

I have only just got the last data transferred over from Archer. But I did test the supermean script to make a single season and it seemed to work OK.
Annette

comment:34 Changed 2 months ago by pmcguire

I am glad the supermean script worked, even if it's for such a big file. It's N2560 isn't it?
Patrick

comment:35 Changed 2 months ago by pmcguire

Supermeans are chugging away. I have split it up to do a year at a time and submitted to lotus. This is the N1280 model.
Annette

comment:36 Changed 2 months ago by pmcguire

I am making progress with AutoAssess…

What do you do with the plots once you have them? Do you put them up on the collab twiki?
Annette

comment:37 Changed 2 months ago by pmcguire

There's various things you can do with the plots.

One thing you could do is put them on the collab twiki, if you want, but I have never done that.

Another thing you can do is put them on the jasmin web page, by copying them from :
/home/users/pmcguire/valnote_v2b
to :
/gws/nopw/j04/ncas_climate_vol2/public/pmcguire

Patrick

comment:38 Changed 2 months ago by pmcguire

Yes I was just looking at putting them on a Jasmin webpage and I think that would be good to do.
Annette

comment:39 Changed 2 months ago by pmcguire

Yes we have a public directory under the hrcm gws.
Annette

comment:40 Changed 2 months ago by pmcguire

But the web pages on jasmin may or may not be slow.

Another thing you can do is download the whole subdirectory of valnote_v2b to your own machine, and it might be faster to view there in your browser.

And you can also make a tarball of the subdirectory of valnote_v2b and put that in a dropbox for people to download.
Patrick

comment:41 Changed 2 months ago by pmcguire

I should add that I have "parallelised" the supermean calculation by just submitting each season in a separate job.
Annette

comment:42 Changed 2 months ago by pmcguire

maybe making a tarball as well as the whole untarred directory for the Jasmin website option might be worthwhile, in case viewing the web page directly on JASMIN is too slow. Then people can just download the tarball from the JASMIN website and view on their own machine.

Can I try out your parallel code? I need to run AutoAssess now myself.
Patrick

comment:43 Changed 2 months ago by pmcguire

This script: /home/users/aosprey/aa_u-br938/supermean-time.py takes as a command-line argument one of the times to create (djf,mam,jja,son or ann).

[Hmm… I may have changed the script a little from your version. I think the only important thing is just that I made the supermean label work for other numbers of years.]

I made a batch script for each season as they take so long to run, e.g. submit-supermean-djf . This is using slurm so needs to be submitted from one of the new sci machines. You could do something cleverer I'm sure rather than manually creating each batch script.
Annette

Last edited 2 months ago by pmcguire (previous) (diff)

comment:44 Changed 2 months ago by pmcguire

thanks!

I'll have a look. I am sure that it works great.
Patrick

comment:45 Changed 2 months ago by pmcguire

It works well for what I am doing as the supermeans take so long to generate.
Annette

comment:46 Changed 2 months ago by pmcguire

yes. the supermeans take awhile for me too.
Patrick

comment:47 Changed 2 months ago by pmcguire

Do you run interactively?
Annette

comment:48 Changed 2 months ago by pmcguire

I recall that I did.
Patrick

comment:49 Changed 2 months ago by pmcguire

Did you run AutoAssess interactively too? I may need to run that in batch - it has just had a seg fault I wonder if it is memory.
Annette

comment:50 Changed 2 months ago by pmcguire

yes, I did the AutoAssess interactively, from what I remember. But it was only for 2-4 years.
Patrick

comment:51 Changed 2 months ago by pmcguire

Do you know why the PMSL plots are like this?
See attachment.
Annette

Last edited 2 months ago by pmcguire (previous) (diff)

Changed 2 months ago by pmcguire

comment:52 Changed 2 months ago by pmcguire

yes. I had the same problem for that plot. I will look into it now.
Patrick

comment:53 Changed 2 months ago by pmcguire

I changed the plots shown for this variable, and it makes more sense.
See attachment.
Patrick

Changed 2 months ago by pmcguire

comment:54 Changed 2 months ago by pmcguire

the lower right hand plot is the same difference variable that you and I were plotting before that was completely purple before, but now it is very close to, but not exactly, zero.

That is better.

the changed plots in the upper right and lower left are unsubtracted original variables.

and they are slightly different than the upper left.

That is good.

The upper left is the experiment.

The lower left is the control.

The upper right is the observational data.

comment:55 Changed 2 months ago by pmcguire

In this file, you can edit the plots for the PMSL and other variables:

aa_trunk_a1.0_pcm/lib/valnote/control/plots_file.dat
#ORIGINAL PLOTS: These are the original plots, which had fully-purple ones for the 2 plots on the right (b, d)
start:PMSL djf (v era-interim)
  a:exper_djfm_pmsl:pmsl_abs
  b:exper_djfm_pmsl - control_djfm_pmsl:pmsl_diff
  c:control_djfm_pmsl - erai_djfm_pmsl:pmsl_diff
  d:exper_djfm_pmsl - erai_djfm_pmsl:pmsl_diff
  significance:(exper_djfv_pmsl*0.5+control_djfv_pmsl*0.5)/2.0
  rms:bcd
end:PMSL djf (v era-interim)
#NEW PLOTS: These plots all looked ok.
#start:PMSL djf (v era-interim)
#  a:exper_djfm_pmsl:pmsl_abs
#  b:control_djfm_pmsl:pmsl_abs
#  c:erai_djfm_pmsl:pmsl_abs
#  d:exper_djfm_pmsl - erai_djfm_pmsl:pmsl_diff
#  significance:(exper_djfv_pmsl*0.5+control_djfv_pmsl*0.5)/2.0
#  rms:bcd
#end:PMSL djf (v era-interim)

Patrick

comment:56 Changed 2 months ago by pmcguire

I am not sure if they have been running a different version of the code at the Met Office.
Annette

comment:57 Changed 2 months ago by pmcguire

sure, it's different.

This is my version that I got from Paul Earnshaw's Jasmin version on the old AutoAssess wiki page that I pointed you to.

I modified my version so that it has mostly variables that work.

The original set of variables is listed in:

aa_trunk_a1.0_pcm/lib/valnote/control/valorder.dat.orig

the a1.0 means the jasmin alpha version
the pcm are my initials meaning that I modified it.

I haven't tried the beta version b0.1.

Patrick

Last edited 2 months ago by pmcguire (previous) (diff)

comment:58 Changed 2 months ago by pmcguire

OK thanks well I will show the others what we've got and see what they say. Previously the validation of the N1280 runs has been done inside the Met Office. […]
Annette

comment:59 Changed 2 months ago by pmcguire

Do you have the N1280 AutoAssess output from the MetOffice?
Patrick

Last edited 2 months ago by pmcguire (previous) (diff)

comment:61 Changed 2 months ago by pmcguire

You might concentrate on plots that have all 4 subplots working right. If it doesn't have all 4 subplots, then skip to the next variable.
Patrick

comment:62 Changed 2 months ago by pmcguire

For example, in /gws/nopw/j04/ncas_climate_vol2/public/pmcguire/u-bq532q_v_u-br310_1989_004_4vars/images/
you'll see that I only did it for 4 variables, for 2-4 seasons, for 4 years of supermeans.

you know that you can just type display x.png on JASMIN, to display the image, rather than needing to download it?
Patrick

comment:63 Changed 2 months ago by pmcguire

Yes I have been using display.
Annette

comment:64 Changed 2 months ago by pmcguire

also, I don't know exactly the best way to run autoassess. But I have found that if I rerun it, I need to delete the files in these directories /group_workspaces/jasmin4/esmeval/cma/flags_v2b5 and /home/users/pmcguire/valnote_v2b5 before rerunning, unless I change the directory names.
Patrick

comment:65 Changed 2 months ago by pmcguire

Oh OK that is useful to know.
Annette

comment:66 Changed 2 months ago by pmcguire

To run with only 4-6 variables, you might use this file instead of valorder.dat:

autoassess2b/aa_trunk_a1.0_pcm/lib/valnote/control/valorder_short.dat.new

I know that most of those variables work.

The newest version of valorder.dat was my modified version of the original valorder.dat, and it was an experimental version, to try to get more of the original variables working right.
Patrick

comment:67 Changed 2 months ago by pmcguire

Ok thanks yes that is useful. And yes looking at some of the other plots from my run, other ones look fine.
Annette

comment:68 Changed 2 months ago by pmcguire

If you setup your directories properly, you can have multiple runs of autoassess going at the same time, for example, one with only 4-6 variables, and the other with all the variables being tried.
Patrick

comment:69 Changed 2 months ago by pmcguire

I am glad some of your plots look fine.
Patrick

comment:70 Changed 2 months ago by pmcguire

Oh wait and the original one looks OK now too! This is after re-running in the batch queue. I don't think I changed anything.
Annette

comment:71 Changed 2 months ago by pmcguire

yes. these purple plots are weird, and they don't happen all the time.

maybe it's a memory problem.
Patrick

comment:72 Changed 2 months ago by pmcguire

Or an issue with one of the python libraries? I had to load jaspy in the batch script and I changed the python path in bin/auto_assess script otherwise it tries to use /usr/bin/python2.7

Well that's good, thanks for your help. I am starting to understand a bit more about what it's doing. I have AutoAssess running in the batch queue so I will take a look at the plots tomorrow and see what they look like. […]
Annette

comment:73 Changed 2 months ago by pmcguire

Annette:
I just looked at your SLURM batch scripts. I might try them today or tomorrow. Thanks!
Patrick

comment:74 Changed 4 weeks ago by pmcguire

  • Reporter changed from pmcguire to annette

comment:75 Changed 8 days ago by ros

  • Resolution set to fixed
  • Status changed from accepted to closed
Note: See TracTickets for help on using tickets.