Opened 4 months ago

Last modified 3 months ago

#2800 accepted project-task

help with porting JULES SINCLAIR suite to CEDA JASMIN

Reported by: pmcguire Owned by: pmcguire
Component: JULES Keywords: porting, pre-build, JULES, Rose/Cylc
Cc: awright, kwilliams Platform: JASMIN
UM Version:

Description

Can you help with getting a copy of the JULES SINCLAIR suite u-am110 working on CEDA JASMIN?
It currently has a pre-built JULES requirement.
It works with enabling the fsmc_shape=1 option of JULES.

Attachments (2)

le.png (104.0 KB) - added by awright 3 months ago.
le.png
smc_tot.png (75.0 KB) - added by awright 3 months ago.
smc_tot.png

Download all attachments as: .zip

Change History (41)

comment:1 Changed 4 months ago by pmcguire

  • Status changed from new to accepted

comment:2 Changed 4 months ago by pmcguire

The copy of the JULES SINCLAIR suite u-am110 has been ported to CEDA JASMIN, including adding dynamic MOSRS extract and build to replace the pre-built JULES requirement. A branch of a branch of JULES has been created and edited to reflect the change of the path to the group workspace on JASMIN for jules from jasmin to jasmin2. The suite has completed the fcm_make and has gone on to successfully running 3 out of the 10 different jules processes. It is still running the other 7 processes, and it still needs to run the make_plots task/process.

comment:3 Changed 4 months ago by pmcguire

The make_plots postprocessing in the JULES SINCLAIR suite u-am110 still doesn't work on CEDA JASMIN.
The user and I have been chasing the bug, which seems to be due to iris/quickplot trying to use netcdftime instead of cftime.
We have been advised that maybe iris2.2 will work better than iris2.1 .
The user has used Python pip to upgrade to iris2.2 in a pip environment on CEDA JASMIN. The next step is to try this with the make_plot postprocessing task and in the rose/cylc suite.

comment:4 Changed 4 months ago by pmcguire

Hello Patrick,

I have run tmp1.py in sci1, with the activated virtual environment, I still get this error:

raise ValueError?: The datetime attribute of the CalendarDateTime? object must be of type netcdftime.datetime.

When I run tmp1.py in cylc I get this error:

import matplotlib as mpl

ImportError?: No module named matplotlib

I am available tomorrow all day if that suits you.

All the best,

Azin

comment:5 Changed 4 months ago by pmcguire

Hi Azin
So maybe the iris 2.2 doesn't fix things?
Can you try to install the nc-time-axis in your pip environment, like was suggested by V Predoi in https://github.com/ESMValGroup/ESMValTool/issues/407
That nc-time-axis is the one with hyphens, and not the underscores.

It won't run on jasmin-cylc as an interactive python process, so if you want to run it in python interactively (as opposed to a batch process on LOTUS), you will need to run it on jasmin-sci1 or jasmin-sci2 or jasmin-sci3.

If these two suggestions still don't help, then maybe we can make an appointment to work together interactively face-to-face.
Patrick

comment:6 Changed 4 months ago by pmcguire

  • Cc valeriu.predoi@… added

comment:7 Changed 4 months ago by pmcguire

Hi Azin
Thank you.
I looked in the dcoumentation for:
https://github.com/SciTools/nc-time-axis
and it suggests there in the main page that you need to install the package as nc-time-axis,
but that, in order to use the package, you should use 'import nc_time_axis'.
So, I think you might have it set up correctly.
I don't know what is wrong.
I asked V. Predoi about this, and he said he could look at the tmp1.py code.
Can you send me that tmp1.py code, together with the JULES NETCDF file that you are using in tmp1.py?
Patrick

comment:8 Changed 4 months ago by pmcguire

Thanks, Azin

I can read your files now. I can reproduce the error when using iris2.1. I have sent along the info to V. Predoi. Hopefully he can figure out what is wrong.

Patrick

comment:9 Changed 4 months ago by pmcguire

Hi Azin
V.Predoi was able to get it working on JASMIN. It turns out that you had at least one problem, in that you didn't have the latest nc-time-axis installed.

You said that when you tried to install nc-time-axis into your Py Pip environment, that it said nc-time-axis was already installed. But the JASMIN metomi installation of Python somehow needs to have the nc-time-axis over-ridden with the latest version of nc-time-axis, which apparently it's not letting you do with Py Pip.

V.Predoi suggests that you use miniconda instead of Py Pip. He says that this will work a lot better than Py Pip. And it's just a subset of the full anaconda, so it won't take quite as long to install and to use.
Here are his instructions:
"
1) So just grab miniconda2 https://docs.conda.io/en/latest/miniconda.html install it and
2) then create a nice virtual env, call it say plotting_environment, and
3) use with my env file for the working deps:

conda env create -n plotting_environment -f /home/users/valeriu/patrick_env.yml

"
Then you should be able to run the tmp1.py program in python2.7 or from the command line.
And you should be able to run the rose/cylc JULES/Sinclair suite.
Let me know how this works, ok?
Patrick

Last edited 4 months ago by pmcguire (previous) (diff)

comment:10 Changed 4 months ago by pmcguire

  • Type changed from help to project-task

comment:11 Changed 4 months ago by pmcguire

  • Cc awright added; azin.wright@… removed

comment:12 Changed 4 months ago by pmcguire

  • Cc valeriu added; valeriu.predoi@… removed

comment:13 Changed 4 months ago by pmcguire

Hello Patrick,

I have installed miniconda, and created the virtual environement (named: 'plotting_virtual_env_sitepackages ') and activated it, then I typed in the command to create the .yml file as follows:

conda env create -n plotting_virtual_env_sitepackages -f /home/users/valeriu/patrick_env.yml

and I receive this error message:
Solving environment: failed

ResolvePackageNotFound?:

  • libgdal==2.2.4=he036fc0_8

I have attached the screenshot as well.

All the best,

Azin

comment:14 Changed 4 months ago by pmcguire

Hi Azin
Have you tried conda install gdal? Or something like that? If you get stuck, maybe you can find a google search answer about
conda install gdal or ResolvePackageNotFound: - libgdal==2.2.4=he036fc0_8.
Patrick

comment:15 Changed 4 months ago by pmcguire

Hi Azin

V Predoi just looked at your email, and I explained to him the background of it.

So I am editing the steps given to you by me from V three days ago as follows:
"
1) So just grab miniconda2 `https://docs.conda.io/en/latest/miniconda.html and install it on JASMIN and
2) then create a nice virtual env, call it say plotting_environment, and use with V.'s env file for the working dependenciess:
conda env create -n plotting_environment -f /home/users/valeriu/patrick_env.yml
"
I had accidently split the 2nd step into steps 2 & 3. Now there is no 3rd step.The original 2nd step is now skipped.
You might have to remove the environmentplotting_environment or use an environment of a different name.
Patrick

Last edited 4 months ago by pmcguire (previous) (diff)

comment:16 Changed 4 months ago by pmcguire

Hi Azin

Does it work now?

Patrick

comment:17 Changed 4 months ago by awright

Hello Patrick,

I have created and activated a virtual environment as:
virtualenv —system-site-packages /home/users/azin/plotting_virtual_env_sitepackages
source /home/users/azin/plotting_virtual_env_sitepackages/bin/activate

then I typed in :
conda env create -n plotting_virtual_env_sitepackages -f /home/users/valeriu/patrick_env.yml

it gave an error message:
Solving environment: failed

ResolvePackageNotFound?:

  • libgdal==2.2.4=he036fc0_8

Then I tried to fix the problem by :
conda install -c conda-forge libgdal
and
conda install -c conda-forge gdal

Then I closed and opened my terminal, and activated my virtual environment. Then typed in :
conda env create -n plotting_virtual_env_sitepackages -f /home/users/valeriu/patrick_env.yml

and it still gives me the same error.

All the best,

Azin

comment:18 Changed 4 months ago by pmcguire

Hi Azin

Please skip the virtualenv step and the source activate step. Do not try to install gdal. Please do this step only:
conda env create -n plotting_virtual_env_sitepackages2 -f /home/users/valeriu/patrick_env.yml

I am calling it plotting_virtual_env_sitepackages2 to avoid conflict with your previous environments, which you might want to delete.

Patrick

comment:19 Changed 4 months ago by pmcguire

Hello Patrick,

[redacted] Karina [] answer[ed] my inquiry regarding adding fluxnet sites to Sinclair suite. She has suggested using the u-al752 suite (which has fluxnet sites) and change some settings to make it Sinclair.

Therefore, I don't know whether trying to plot in u-bg657 is still is a useful thing to do?

All the best,

Azin

comment:20 Changed 4 months ago by awright

Hello Patrick,

By just putting that command line, I still get the same error:

[azin@jasmin-sci1 ~]$ conda env create -n plotting_virtual_env_sitepackages2 -f /home/users/valeriu/patrick_env.yml
Collecting package metadata: done
Solving environment: failed

ResolvePackageNotFound?:

  • libgdal==2.2.4=he036fc0_8

Azin

comment:21 Changed 4 months ago by pmcguire

Hi Azin

Did you deactivate your prior environments before creating this environment?

Patrick

comment:22 Changed 4 months ago by awright

Hello Patrick,

I have deactivated it now, and closed and opened the terminal, but it is still the same.

Azin

comment:23 Changed 4 months ago by pmcguire

Hi Azin
Are you using the jasmin-sci* nodes or the jasmin-cylc VM?
Maybe it will only work on jasmin-sci*.
Patrick

comment:24 Changed 4 months ago by awright

Hello Patrick,

I was using jasmin-sci, i tried jasmin-cylc as well. I still get the same error. I think I just use the u-al752 suite and see how that one will go.

All the best,

Azin

comment:25 Changed 4 months ago by pmcguire

Hi Azin
Trying the JULES/FLUXNET suite to see how that one will work (with adding the JULES/Sinclair stuff) is a decent way to go.
One issue is that you won't be able to see the plots from the original JULES/Sinclair suite u-bg657.
Patrick

comment:26 Changed 3 months ago by pmcguire

Hi Azin
V Predoi has revised the patrick_env.yml to make a patrick_env_short.yml file,
and he has written a very nice guide about how to use miniconda (or alternatively, anaconda).
Can you look at this new file ? ( patrick_env_short.yml ) You will see it is much shorter.
Maybe it will work and handle your gdal problem properly in the JULES Sinclair suite.
Can you try the previous instructions using instead this new short file instead of the longer yml file?
Patrick


Hi Patrick,

I have re-created the conda env file in /home/users/valeriu/patrick_env_short.yml

If you have a look at it you will see it is very minimal:

name: base
channels:

  • conda-forge
  • defaults

dependencies:

  • iris=2.2.0=py27_1003
  • nc-time-axis=1.1.0=py_0

Here are some instructions that may be useful to you and can pass them on to Azin so you guys understand what's what with all this conda stuff:

  • it is a good habit to use conda (miniconda or anaconda) to create virtual environments where a suite of dependencies can live and certain specific dependency versions can be pinned (like in our case we know e need iris 2.2.0 and nc-time-axis 1.1.0; pinning means hard setting the version via = or hard setting a minimal version via >);
  • if we know we need just two specific packages (like here, iris and nc-tim-axis), a minimal environment file can be assembled and populated with only those packages and their respective versions; conda will install all the dependencies that the packages require - you will see that after creating and activating the environment, you will have a bunch of needed packages like scipy, numpy etc without specifically listing them in the environment file - and that's because eg iris needs them;
  • environment creation and activation is done after you have conda in path by:

conda env create -n iris_env -f patrick_env_short.yml
source activate iris_env

  • if you inspect your miniconda repo, you will see that under miniconda/envs/ there is a directory with the environment name, iris_env; that one stores the installed packages part of the iris_env environment and has a Python-specific install dir structure; one can simply remove the iris_env dir to get rid of the environment or can update it by installing new packages, if needed via:

conda install -c $CHANNEL $PACKAGE

where $CHANNEL is the name of the channel where the package is stored in the remote conda tree (most common is conda-forge);

That's about it, now the plotting script should work and also a lot of other things can be done since you have the latest iris.

Cheers,
V

comment:27 Changed 3 months ago by pmcguire

  • Cc kwilliams added

comment:28 Changed 3 months ago by awright

Hello Patrick and V,

Thank you very much for your help. I have manged to get my plots now.

All the best,

Azin

comment:29 Changed 3 months ago by pmcguire

That's excellent news, Azin!
Can you attach some example plots for the JULES/Sinclair suite to this ticket?
Patrick

comment:30 Changed 3 months ago by awright

Hello Patrick,

How do you normally attach jules plots to tickets? Do I copy them into scratch and then download them onto my laptop and attach them here, or are there any other more direct ways?

All the best,

Azin

comment:31 Changed 3 months ago by pmcguire

  • Cc valeriu removed

I am removing V Predoi from this ticket, since the Python/iris/dependency issues have been addressed and since they are apparently fixed.
Patrick

comment:32 Changed 3 months ago by pmcguire

  • Cc valeriu added

comment:33 Changed 3 months ago by pmcguire

I am removing V Predoi from this ticket, since the Python/iris/dependency issues have been addressed and since they are apparently fixed.
Patrick

comment:34 Changed 3 months ago by pmcguire

  • Cc valeriu removed

comment:35 Changed 3 months ago by pmcguire

Hi Azin
Yes, you can use scp to pull the PDF file from jasmin-xfer1 to your Mac. You will need to copy the files (preserving time stamp info) from scratch to your home directory or to a group workspace first (since one can't transfer externally to/from scratch).
There might be a more direct way, but I am not familiar with it.
Patrick

comment:36 Changed 3 months ago by awright

Hello Patrick,

Here are some of the plots produces by the JULES_Sinclair suite (u-bg657 in my space, which is copied from u-am110).
Do I need to commit anything now? Is it possible to commit the whole suite or do I have to do it file by file? I guess the only change we made was having V's iris_env.

I have noticed that I have to activate the iris_env everytime I open a new terminal. It does not automatically activate. Is that how it should be?

All the best,

Azin

comment:37 Changed 3 months ago by awright

Here are the plots:

Last edited 3 months ago by awright (previous) (diff)

Changed 3 months ago by awright

le.png

Changed 3 months ago by awright

smc_tot.png

comment:38 Changed 3 months ago by pmcguire

Hi Azin
Those JULES/Sinclair plots from running Karina's suite on CEDA JASMIN look very good and very interesting! Good work!

I will keep this ticket open for a little while, in case you have some further issues with running this suite that we ported to CEDA JASMIN.

Can you commit your changes to the suite to MOSRS with fcm ci? What is the suite number that you're using now?

I will try checking out and running your suite myself sometime in the next few weeks, so I will have to repeat the miniconda exercise that you did with V's input.
Patrick

comment:39 Changed 3 months ago by pmcguire

Hi Azin:

Furthermore, you wrote: "I have noticed that I have to activate the iris_env everytime I open a new terminal. It does not automatically activate. Is that how it should be?"

You can certainly set up your environment in your .bashrc file, for example, to activate the iris_env environment automatically. Maybe this is preferred. Maybe not. It depends. You might have other Python programs or Rose/Cylc suites that would need a different miniconda python environment. Or maybe you need the normal installation of Python on CEDA JASMIN instead of your specific miniconda environment. But if you always want to use the same miniconda environment, then you can do that.

Patrick

Note: See TracTickets for help on using tickets.