Opened 5 months ago

Closed 4 months ago

#2857 closed help (fixed)

ImportError: No module named matplotlib

Reported by: awright Owned by: pmcguire
Component: JULES Keywords: jasmin, python
Cc: Platform: JASMIN
UM Version:

Description

Hello Patrick,

I am running u-bh383. The make and run were fine, the plotting process fails due to :
File "/home/users/azin/cylc-run/u-bh383/bin/fluxnet_evaluation.py", line 10, in <module>

import matplotlib as mpl

ImportError?: No module named matplotlib

I have done some checks such as :

(base) [azin@jasmin-cylc ~]$ which python
~/miniconda2/bin/python
(base) [azin@jasmin-cylc ~]$ python2.7
Python 2.7.15 |Anaconda, Inc.| (default, Dec 14 2018, 19:04:19)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import matplotlib

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

ImportError?: No module named matplotlib

but I am not quite sure what to do now. Do I need a virtual machine? Do I just pip install matplotlib? Do I need to be in sci* environment? Apparently my environment is using the python in miniconda2? How do I check the version?

Best wishes,

Azin

Attachments (4)

Screen Shot 2019-04-04 at 13.02.41.png (68.8 KB) - added by awright 5 months ago.
Screen Shot 2019-04-05 at 14.17.35.png (111.6 KB) - added by awright 5 months ago.
Screen Shot 2019-04-05 at 14.17.07.png (73.0 KB) - added by awright 5 months ago.
Screen Shot 2019-04-05 at 16.46.49.png (160.6 KB) - added by awright 5 months ago.

Download all attachments as: .zip

Change History (24)

Changed 5 months ago by awright

comment:1 Changed 5 months ago by pmcguire

  • Status changed from new to accepted

comment:2 Changed 5 months ago by pmcguire

Hi Azin
Can you try this on jasmin-sci* instead of jasmin-cylc? It should work there. The Virtual Machine jasmin-cylc doesn't have the full Python capabilities, and is mostly to be used for submitting Rose/Cylc batch jobs.
Does it work on jasmin-sci*?
Patrick

comment:3 Changed 5 months ago by awright

Hello Patrick

I have tried importing matplotlib in sci* as follows, but it did not work.

(base) [azin@jasmin-sci1 ~]$ cd roses/u-bh383
(base) [azin@jasmin-sci1 u-bh383]$ ls
ancil app bin meta rose-suite.conf rose-suite.info site suite.rc
(base) [azin@jasmin-sci1 u-bh383]$ cd bin
(base) [azin@jasmin-sci1 bin]$ ls
archive_data.bat generate_conf_files.py list_sites_fluxnet2015.txt make_plots.py test_generate_conf_files.py
fluxnet2015_site_info.csv lba_site_info.csv list_sites_lba.txt run_fluxnet_evaluation_for_debugging.bat
(base) [azin@jasmin-sci1 bin]$ make_plots.py
-bash: make_plots.py: command not found
(base) [azin@jasmin-sci1 bin]$ python
Python 2.7.15 |Anaconda, Inc.| (default, Dec 14 2018, 19:04:19)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import matplotlib

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

ImportError?: No module named matplotlib

Can I come to you office with my laptop? It will be easier I think?!
All the best,
Azin

comment:4 Changed 5 months ago by awright

Hello Patrick

I think it is working now. I have created the virtual environment that V has kindly prepared. I still need to test it, will let you know.

Best wishes,

Azin

comment:5 Changed 5 months ago by awright

Hello Patrick,

To check the make_plots.py, I type in:

cd ~/roses/u-bh383/bin/
./make_plots.py

and this is the error I receive:
Traceback (most recent call last):

File "./make_plots.py", line 6, in <module>

import fluxnet_evaluation

ImportError?: No module named fluxnet_evaluation

How would you think I should go about that?

All the best,

Azin

comment:6 Changed 5 months ago by pmcguire

The fluxnet_evaluation.py module is checked out from MOSRS when the JULES/FLUXNET suite is run.
Once this rose/cylc suite is run, the location of this Python module is in ~/cylc-run/suitenum/bin.

Last edited 5 months ago by pmcguire (previous) (diff)

comment:7 Changed 5 months ago by pmcguire

Hi Azin
Also, you probably should run the make_plots.py module from a script (like the run*.bat scripts that I think we already used together). This will define the appropriate environment variables that make_plots.py needs. You might remember that we set up an example run*.bat file in either bin or bin2 in one of your cylc-run suite directories.
Patrick

comment:8 Changed 5 months ago by awright

Hello Patrick,

How would I implement the new address (/home/users/azin/cylc-run/u-bh383/bin/fluxnet_evaluation) into make_plots.py? I have tried some ways and it has not worked. Can I come to your office and show you? I think it would be a lot more effective to communicate face to face rather than by email.

All the best,

Azin

comment:9 Changed 5 months ago by pmcguire

Hi Azin

I would enjoy meeting you about this, but before we meet, can you try a little more?
I think you are on the right track. But maybe the word Fluxnet_evaluation should not be capitalized in your make_plots.py file in your roses\u-bh383\bin directory?

One way to test it is the most straightforward: Just run your suite again after making that change.

You can't just run the make_plots.py program by itself without the suite and without the run*.bat approach. You would need to modify your run*.bat file in your cylc_run\bin directory.
An old example that might not work anymore of a modified run*.bat file is in:
~pmcguire/cylc-run/u-al752jasmin-sci1/bin/run_fluxnet_evaluation_for_debugging.bat.
There are several items that you would need to change if you use a file like that. Hopefully you can figure out what you would need to change.
But it's a lot more straightforward to just run your suite again once you have the setup configured correctly ( like the capitalization issue, for example).
Patrick

Changed 5 months ago by awright

Changed 5 months ago by awright

comment:10 Changed 5 months ago by awright

Hello Patrick,

Thank you very much for your help. By getting rid of the capitalized letter F in Fluxnet_evaluation in make_plots.py, that error has been solved. I have a new error now:
UserWarning?: subset has not been recognised (error attached). The error is related to this section of~/cylc-run/u-bh383/bin/fluxnet_evaluation.py :

if obs == 'peg':

if var in VAR_NAMES_DICT.keys():

#obs_cube = get_daily_pre2015_obs_data(site, VAR_NAMES_DICT[var]obs_var_name_pre2015?)
obs_cube = get_daily_fluxnet2015_obs_data(site, var)

else:

obs_cube = None

elif obs == 'lba':

if var in VAR_NAMES_DICT.keys():

obs_cube = get_daily_lba_obs_data(site, var)

else:

obs_cube = None

elif obs is None:

obs_cube = None

else:

raise UserWarning?('subset has not been recognised')

return obs_cube

Do you have any suggestions?

All the best,

Azin

comment:11 Changed 5 months ago by pmcguire

Hi Azin

You have set the SUBSET to 'GF_Guy' in your rose-suite.conf file.
This option is not currently allowed (unless you want to add it).
You can see the list of options for SUBSET in roses\u-bh383\bin\make_plots.py.

Patrick

comment:12 Changed 5 months ago by awright

Hello Patrick,

I think you mean this list of SUBSETS: 'all', 'peg', 'lba', 'mead', 'med', 'imprex', 'china' and 'wetland'. I have changed GF_Guy to peg now and I am running the whole suite again.

I also made a .bat file called make_plots.bat:
OUTPUT_FOLDER=$HOME/roses/u-bh383/output_data
PLOT_FOLDER=$HOME/roses/u-bh383/plots
export OUTPUT_FOLDER
export PLOT_FOLDER
./make_plots.py

However, although I have changed the capital letter F to small letter f (in the word fluxnet_evaluation) in make_plots.py, when I run make_plots.bat, it gives this error:

(iris_env) [azin@jasmin-cylc bin]$ ./make_plots.bat
Traceback (most recent call last):

File "./make_plots.py", line 6, in <module>

import fluxnet_evaluation

ImportError?: No module named fluxnet_evaluation

All the best,

Azin

Changed 5 months ago by awright

comment:13 Changed 5 months ago by awright

Hello Patrick,

The jules run has just finished and I got a new error in plotting:
IOError: /group_workspaces/jasmin2/jules/pmcguire/fluxnet/kwilliam/suite_data/vn1.0/fluxnet_obs/daily_obs/AT_Neu-carbon-daily.dat not found.

I have attached a screenshot as well. Is that because I chose 'peg'? Ideally I would want to choose 'all'. Do you have any suggestions?

Many thanks,

Azin

comment:14 Changed 5 months ago by pmcguire

Hi Azin

I am glad that the whole fluxnet suite is running further now and that you have changed the SUBSET to an existing subset ('peg').
Sorry that it is not working yet. See the next message for some details about this.

If you want it to run faster, you could change it to 'china' or 'mead' or something. There are fewer sites there, as you can see in the list of sites in make_plots.py. You could even copy your suite, change your settings and output directories, and start this shorter run right now if you wanted.

In order for your offline make_plots.bat file to find the fluxnet_evaluation.py module, you will need to run the make_plots.bat file from within the ~azin\cylc-run\u-bh383\bin directory, because that is the location where MOSRS checked out the copy of fluxnet_evaluation.py program. This file fluxnet_evaluation.py does not exist in ~azin\roses\u-bh383\bin. So:
1) cd ~azin\cylc-run\u-bh383\bin
2) ./make_plots.bat

You should also be aware that you have not set the variables correctly in your make_plots.bat file.
There is no JULES data already in OUTPUT_FOLDER=$HOME/roses/u-bh383/output_data.
The directory that outputs the JULES data for this run is (as you can see in your rose-suite.conf file):
OUTPUT_FOLDER='/work/scratch/azin/fluxnet/run11a/jules_output'.
You might want to change your PLOT_FOLDER, too, since it's not the best idea to be putting model output or plots directly in the roses directory.

There are other variables that you might want to change too. You can see some example settings in:
~pmcguire/cylc-run/u-al752jasmin-sci1/bin/run_fluxnet_evaluation_for_debugging.bat.

comment:15 Changed 5 months ago by pmcguire

Hi Azin
Regarding your new error message, if you do:
ls -ltr /group_workspaces/jasmin2//jules/pmcguire/fluxnet/kwilliam/suite_data//vn1.0/fluxnet_obs/daily_obs/AT_Neu-carbon-daily.dat
you will see that the file does indeed not exist.
But if you do:
ls -ltr /group_workspaces/jasmin2//jules/pmcguire/fluxnet/kwilliam/suite_data//vn1.0/fluxnet_obs/daily_obs/AT_Neu-energyandcarbon-dailyUTC.dat
you will see that this file does exist.

The suite you are using now is the new one that has been updated in the past few months.
You can try the same with the newer fluxnet_evaluation.py program (at the different JULES revision number) or with the older suite (at the different MOSRS revision number), and it might work

You can also study the problem further, by doing (comparing my old copy on JASMIN to your copy of fluxnet_evaluation.py on JASMIN:
diff ~pmcguire/cylc-run/u-al752jasmin-sci1/bin/fluxnet_evaluation.py ~azin/cylc-run/u-bh383/bin/fluxnet_evaluation.py
where you will see that the files are the same.
And then you can do (comparing my new copy on JASMIN to the old version):
diff ~pmcguire/cylc-run/u-al752/bin/fluxnet_evaluation.py ~azin/cylc-run/u-bh383/bin/fluxnet_evaluation.py
and you can see that the files are very different.
This has to do with the version numbers being used for MOSRS for fluxnet_evaluation.py in your rose-suite.conf file being different than the latest version.

comment:16 Changed 5 months ago by awright

Hello Patrick,

As you recommended, I have copied the newer version of fluxnet_evaluation.py from ~pmcguire/cylc-run/u-al752/bin/fluxnet_evaluation.py in the place of ~azin/cylc-run/u-bh383/bin/fluxnet_evaluation.py. I have also changed the fluxnet sites from 'peg', to 'mead' to get a quicker answer. The suite (u-bh383) has run with no errors and I have some plots in my '/work/scratch/azin/fluxnet/run11a/plots' now.

Many thanks for all your help,

Azin

comment:17 Changed 5 months ago by pmcguire

Hi Azin
You're welcome! I am glad it is working now. But there's still an issue: if you commit your changes of your suite u-bh383 to the MOSRS archive with fcm ci, and then you or somebody else tries to check out your suite to a clean version, it probably won't work.

One way to fix this is to change the revision numbers of the Python files that you check out in your rose-suite.conf file to the same revision number as for the Python files in the latest version of the file roses\u-al752\rose-suite.conf.

In the latest version of u-al752, the Python revision numbers are 13496:

[file:bin/jules.py]
source=fcm:jules.x_br/pkg/karinawilliams/r6715_python_packages/share/jules.py@13496

[file:bin/make_time_coord.py]
source=fcm:jules.x_br/pkg/karinawilliams/r6715_python_packages/share/
     make_time_coord.py@13496

[file:bin/fluxnet_evaluation.py]
source=fcm:jules.x_br/pkg/karinawilliams/r6715_python_packages/share/
     fluxnet_evaluation.py@13496

In your file ~azin/roses/u-bh383/rose-suite.conf, the revision number is 11084:

[file:bin/jules.py]
source=fcm:jules.x_br/pkg/karinawilliams/r6715_python_packages/share/jules.py@11084

[file:bin/make_time_coord.py]
source=fcm:jules.x_br/pkg/karinawilliams/r6715_python_packages/share/
    make_time_coord.py@11084

[file:bin/fluxnet_evaluation.py]
source=fcm:jules.x_br/pkg/karinawilliams/r6715_python_packages/share/
    fluxnet_evaluation.py@11084

[file:bin/list_sites_fluxnet2015.txt]
source=fcm:jules.x_br/pkg/karinawilliams/r6715_python_packages/share/
    list_sites_fluxnet2015.txt@11084

[file:bin/list_sites_lba.txt]
source=fcm:jules.x_br/pkg/karinawilliams/r6715_python_packages/share/
    list_sites_lba.txt@11084

comment:18 Changed 5 months ago by awright

Thank you Patrick,
I have changed the revision number in roses\u-al752\rose-suite.conf to 13496 and committed the changes by: 'fcm commit u-bh383'.
Now I need to copy my output from scratch to somewhere which is suitable for storing files. Where do you recommend? Somewhere inside rose-cylc or roses or somewhere else?

Many thanks,

Azin

comment:19 Changed 5 months ago by pmcguire

Hi Azin
Excellent! How many changes did you need to make to your u-bh383 code to get it working?
You can keep things on scratch for a while, but it might get erased sometime.
You can copy things to a directory in your home directory or in a group workspace.
In your home directory, it's probably not best to put it in the roses directory, since I think it's better to only store source code there.
Also, the cylc-run directory, is a good place to keep temporary outputs, but it's better to store your long term outputs someplace else.
Patrick

comment:20 Changed 4 months ago by pmcguire

  • Resolution set to fixed
  • Status changed from accepted to closed
Note: See TracTickets for help on using tickets.