Opened 4 months ago

Closed 5 weeks ago

Last modified 4 weeks ago

#3265 closed help (answered)

SoilGrids data on JASMIN

Reported by: epinnington Owned by: pmcguire
Component: JULES Keywords: SoilGrids, NETCDF, JASMIN, Africa, JULES, TAMSAT, SMAP, Toth, pedotransfer
Cc: Platform: JASMIN
UM Version:

Description

Hi Patrick,

I just wondered if you had the SoilGrids data? And if so if you had it setup in a NetCDF format and stored on JASMIN somewhere? We are getting an Africa JULES suite setup to run with the TAMSAT precipitation data at 0.25-degree resolution and also looking to do some data assimilation with SMAP satellite soil moisture observations to improve the JULES soil ancillaries over Africa. I know you were using the SoilGrids data with the Toth pedotransfer functions, so just wondered if you had it handy?

Cheers,
Ewan

Change History (18)

comment:1 Changed 4 months ago by pmcguire

  • Status changed from new to accepted

comment:2 Changed 4 months ago by pmcguire

Hi Ewan:
Yes, I have been using code that extracts the 5km SoilGrids data, regrids/resamples it to 0.5 degrees resolution, and applies the various PTFs to them to compute the Van Genuchten parameters.
A mostly-complete history of the development (with links to code) that Imtiaz Dharssi, myself, and others have been doing is shown here:

http://cms.ncas.ac.uk/ticket/2777
https://code.metoffice.gov.uk/trac/ancil/ticket/713

I can help further with this and give further advice, if you want.
Patrick

comment:3 Changed 4 months ago by pmcguire

Hi Patrick
Ah wow this looks really cool! It would be great to get more help if possible.


I have the scripts to calculate the JULES soil ancillaries given a soil texture database, how tough would it be just to extract the soil texture and other property information using your scripts and then save this into a netcdf file (would be great to have this at the 5km resolution, then I can regrid it to the TAMSAT grid)? Or do you think it would be better to just use your scripts and vary the pedotransfer parameters within these? We would only need it over Africa so about -36 to 38 latitude and -18 to 52 longitude. Thanks!

Cheers,
Ewan

comment:4 Changed 4 months ago by pmcguire

Hi Ewan


Can you apply for access to the GWS on JASMIN called ncas_generic? Then I can point you to the directory where I have the code and the NETCDF files for the global 0.5-degree-grid SoilProperties files made using different PTFs for the 6 or 7 SoilGrids soil layers.

It wouldn't take a whole lot of work to make such NETCDF files at 0.25 degrees. It might be harder to go to much lower resolution than that, even for a subset of the global grid.

Patrick

comment:5 Changed 4 months ago by pmcguire

Hi Patrick,

That would be great, I have just been accepted for access to the ncas_generic GWS, where do I need to look for the NetCDF files? I can just regrid the 0.5 degree soil properties to start with if this is easier? It would be great to get them at 0.25 degree resolution too! Thanks for all the help so far!

Cheers,
Ewan

comment:6 Changed 4 months ago by pmcguire

Hi Ewan:
This is the working directory:

/gws/nopw/j04/ncas_generic/users/pmcguire/ants/soil_ancil3/

These are the 0.5deg soil properties ancillary files for 7 different SoilGrids soil-layers at different depths for Tóth et al PTF's 17and20 is at:

/gws/nopw/j04/ncas_generic/users/pmcguire/ants/soil_ancil3/soil_WFDEI_TothContin17and20_V1n_l?_2b.nc

The ? is a wildcard for layer number.

The soil_bulk_density field should be ignored.
The fields starting with'in_* are the regraded/resampled SoilGrids values, some of which were the inputs to the Tóth et al PTF to compute some of the other parameters.
This NETCDF file can be read into JULES as the soil properties ancillary.

The script:

/gws/nopw/j04/ncas_generic/users/pmcguire/ants/soil_ancil3/TothContin17and20_V1n_script.bsub

produced these NETDCF files, using JASMIN LOTUS batch processing, after activating the ANTS anaconda/python environment.
After activating that ANTS anaconda/python environment, the script is run with:

bsub < TothContin17and20_V1n_script.bsub

And this calls this Python code with the right setup parameters:

/gws/nopw/j04/ncas_generic/users/pmcguire/ants/soil_ancil3/ancil_soilGrid_V1n.py

Would it be OK if I made a CMS Helpdesk ticket for this work with you?
Patrick

comment:7 Changed 4 months ago by pmcguire

Hi Patrick,

Thanks for this! The files look great. Yes I am happy for you to make a ticket for this work!

You say the soil_bulk_density variable should be ignored but the in_bld field is also missing everywhere in the specified files, is there any files that contain the bulk density information (This is needed in the pedotransfer functions we were using)? Once we have that sorted I think I can just regrid these files to 0.25 degree for now and crop out Africa to start running some experiments which will be cool!

I will have a look at the scripts now! How do you activate the ANTS Anconda environment?

Thanks again!
Ewan

Last edited 4 months ago by pmcguire (previous) (diff)

comment:8 Changed 4 months ago by pmcguire

Hi Ewan

Yes, the BLD is in another file:

/gws/nopw/j04/ncas_generic/users/pmcguire/ants/soil_ancil3/soil_WFDEI_TothDisc19_V1m_l5_2b.nc

This is the file that uses the Tóth et al PTF 19 (which only uses the discrete texture classes from SoilGrids (in_text) as input), but it
has the other SoilGrids fields at 0.5 degrees as well.

It uses V1m of the scripts instead of V1n.

You probably want to make sure that all the in_* fields have the right values and units.
Patrick

comment:9 Changed 4 months ago by pmcguire

Hi Ewan

Yes, the BLD is in another file:

/gws/nopw/j04/ncas_generic/users/pmcguire/ants/soil_ancil3/soil_WFDEI_TothDisc19_V1m_l5_2b.nc

This is the file that uses the Tóth et al PTF 19 (which only uses the discrete texture classes from SoilGrids (in_text) as input), but it
has the other SoilGrids fields at 0.5 degrees as well.

It uses V1m of the scripts instead of V1n.

You probably want to make sure that all the in_* fields have the right values and units.
Patrick

comment:10 Changed 4 months ago by pmcguire

Hi Patrick:

That file looks great! How tough would it be to use the scripts to create a similar thing at 0.25-degree resolution?

Very good point, have you got a list of the units anywhere?

How are you interpolating the different layer depths to represent the 4 default layers of JULES? Or are you just running JULES with 7 layers?

Ewan

comment:11 Changed 4 months ago by pmcguire

Hi Ewan

I don't think it would be that hard to make the same files at 0.25 degrees. But I'd have to try, and I don't know how long it would take.

The units should be in the NETCDF file. You can uses ncinfo or ncview on JASMIN (jasmin-sci*) to view that info.

So far, we've been running (global) JULES with only 1 layer, corresponding to layer 5 (60-100cm). But the goal is to eventually use these files for 4-7 JULES soil layers. It would be best to use them with 7 soil-layer JULES.

To do the JULES sims with 4 soil layers would require more thought and possibly interpolation or resampling.

Patrick

comment:12 Changed 4 months ago by pmcguire

Hi Patrick,

Cool no worries, I have sent a text file (by email) describing the TAMSAT JULES model grid if this helps but no problem if you dont have time to try this out! I am sure regridding the 0.5-degree file should be fine! I might also take a look at the 5km .tiff files too.

Having a look at the files it seems like the variables for CEC, pH and BLD don't have any units? Do you know what these are?

Cool, yes I will have a think about the depth issue!

Thanks again!
Ewan

comment:13 Changed 4 months ago by pmcguire

Hi Ewan
The units for pH are standard: (no units), with a 1-14 range.

The units for BLD (soil bulk density) are g/cm^3.

The units for CEC are the units used by Tóth: (meq 100g^-1).
see the discussion in lines 401-403 of:

/gws/nopw/j04/ncas_generic/users/pmcguire/ants/soil_ancil3/ancil_soilGrid_V1n.py

About activating the ANTS anaconda/Python environment:
if this is not urgent, can I tell you about this another time?
I need to write a talk on another subject for tomorrow.

Also, if you could respond to the ticket by logging in to the CMS Helpdesk website, and responding there, instead of by email, that would be much preferred.
Patrick

Last edited 4 months ago by pmcguire (previous) (diff)

comment:14 Changed 4 months ago by epinnington

Hi Patrick,

Thanks for that!

No problem at all! Hope the talk tomorrow goes well.

Cheers,
Ewan

comment:15 Changed 2 months ago by pmcguire

Hi Ewan
Do you still need help with this? Is everything working now?
If there is additional information that you could add to this ticket, as to what you did to get things working, that would be helpful.
Patrick

comment:16 Changed 2 months ago by epinnington

Hi Patrick,

No it is all great currently thanks! All is working, I have just taken the SoilGrids? dataset from your directory and then used my own python functions of the Toth PTF's to create the JULES ancillaries and it seems to be producing good results compared to SMAP observations of soil moisture over Africa.

Thanks again for all the help and I will keep you updated when we start to write this up!

Ewan

comment:17 Changed 5 weeks ago by pmcguire

  • Resolution set to answered
  • Status changed from accepted to closed

comment:18 Changed 4 weeks ago by pmcguire

  • Cc epinnington removed
  • Reporter changed from pmcguire to epinnington
Note: See TracTickets for help on using tickets.