Opened 9 months ago

Closed 7 months ago

#2375 closed help (fixed)

convsh - reading in files from different sub-directories

Reported by: mvguarino Owned by: jeff
Priority: normal Component: UM Tools
Keywords: convsh, xconv Cc:
Platform: ARCHER UM Version:

Description

Hello,

Is there a way to read in multiple files stored in different directories/sub-directories using a loop in a convsh script?

maybe something analogue to this example that shows how to select files with different names (but in a same directory):

http://cms.ncas.ac.uk/documents/xconv/example3.html

Thanks,

Vittoria

Change History (18)

comment:1 Changed 9 months ago by jeff

  • Owner changed from um_support to jeff
  • Status changed from new to accepted

Hi Vittoria

It should be easy, but would depend on what exactly you wanted to do. You can put wildcards (*) in both the file and directory name and the glob command would expand them out. A simple program to output the actual file names used would be

#! /usr/bin/env convsh

foreach file [glob */*] {
    puts $file
}

You could put your particular directory/file name structure in this program and run it to make sure it is selecting the correct files.

If this doesn't help get back to me with more details of what you want to do.

Jeff.

comment:2 Changed 9 months ago by mvguarino

Hi Jeff,

Thank you,
that was helpful and I managed to write a working script.

Vittoria

comment:3 Changed 9 months ago by jeff

  • Resolution set to answered
  • Status changed from accepted to closed

comment:4 Changed 9 months ago by mvguarino

Maybe I spoke too early… Although my script is running, it doesn't do what I want it to do.
This is the script I am running:

#! /usr/bin/env convsh1.93

#  Convsh script getst.tcl
#

#  Write out netCDF file
set outformat netcdf

#  Automatically work out input file type
set filetype 0

#  Convert one-variable field only to netCDF
#  (use Xconv on one file to get Variable field numbers)
# -1 for all the fileds in the nc file, otherwise specify the single file number, if a list: "1 2 3"
set fieldlist "13 14" 

#  path to parent directory
set dir /nerc/n02/n02/vittoria/u-as245/18500201T0000Z*


set outfile NEMO_water_salinity_theta_monthly.nc


#  Read in files

foreach infile [glob $dir/nemo_as245o_1m_*-T.nc] {

  
#  Read input file
   readfile $filetype $infile

}
#  Write out fields to a netCDF file
writefile $outformat $outfile $fieldlist



and this is the .out file:

####################################
######### START PROLOGUE  ##########
####################################

User=vittoria
Job=22164.rdf-xcat
Group=n02
Queue=nscluster
Account=n02-NEK004514
Budget Check Passed: User = vittoria, Project = n02-NEK004514
file /nerc/n02/n02/vittoria/u-as245/18500201T0000Z/nemo_as245o_1m_18500101-18500201_grid-T.nc is a HDF5 or NetCDF4 file 
can only read classic data model format NetCDF4 files
file /nerc/n02/n02/vittoria/u-as245/18500201T0000Z/nemo_as245o_1m_18500101-18500201_grid-T.nc is a HDF5 or NetCDF4 file 
can only read classic data model format NetCDF4 files
file /nerc/n02/n02/vittoria/u-as245/18500201T0000Z/nemo_as245o_1m_18500101-18500201_grid-T.nc is a HDF5 or NetCDF4 file 
can only read classic data model format NetCDF4 files
file /nerc/n02/n02/vittoria/u-as245/18500201T0000Z/nemo_as245o_1m_18500101-18500201_grid-T.nc is a HDF5 or NetCDF4 file 
can only read classic data model format NetCDF4 files
file /nerc/n02/n02/vittoria/u-as245/18500201T0000Z/nemo_as245o_1m_18500101-18500201_grid-T.nc is a HDF5 or NetCDF4 file 
can only read classic data model format NetCDF4 files
file /nerc/n02/n02/vittoria/u-as245/18500201T0000Z/nemo_as245o_1m_18500101-18500201_grid-T.nc is a HDF5 or NetCDF4 file 
can only read classic data model format NetCDF4 files
file /nerc/n02/n02/vittoria/u-as245/18500201T0000Z/nemo_as245o_1m_18500101-18500201_grid-T.nc is a HDF5 or NetCDF4 file 
can only read classic data model format NetCDF4 files
file /nerc/n02/n02/vittoria/u-as245/18500201T0000Z/nemo_as245o_1m_18500101-18500201_grid-T.nc is a HDF5 or NetCDF4 file 
can only read classic data model format NetCDF4 files
file /nerc/n02/n02/vittoria/u-as245/18500201T0000Z/nemo_as245o_1m_18500201-18500301_grid-T.nc is a HDF5 or NetCDF4 file 
can only read classic data model format NetCDF4 files
file /nerc/n02/n02/vittoria/u-as245/18500201T0000Z/nemo_as245o_1m_18500201-18500301_grid-T.nc is a HDF5 or NetCDF4 file 
can only read classic data model format NetCDF4 files
file /nerc/n02/n02/vittoria/u-as245/18500201T0000Z/nemo_as245o_1m_18500201-18500301_grid-T.nc is a HDF5 or NetCDF4 file 
can only read classic data model format NetCDF4 files
file /nerc/n02/n02/vittoria/u-as245/18500201T0000Z/nemo_as245o_1m_18500201-18500301_grid-T.nc is a HDF5 or NetCDF4 file 
can only read classic data model format NetCDF4 files
file /nerc/n02/n02/vittoria/u-as245/18500201T0000Z/nemo_as245o_1m_18500201-18500301_grid-T.nc is a HDF5 or NetCDF4 file 
can only read classic data model format NetCDF4 files
file /nerc/n02/n02/vittoria/u-as245/18500201T0000Z/nemo_as245o_1m_18500201-18500301_grid-T.nc is a HDF5 or NetCDF4 file 
can only read classic data model format NetCDF4 files
file /nerc/n02/n02/vittoria/u-as245/18500201T0000Z/nemo_as245o_1m_18500201-18500301_grid-T.nc is a HDF5 or NetCDF4 file 
can only read classic data model format NetCDF4 files
file /nerc/n02/n02/vittoria/u-as245/18500201T0000Z/nemo_as245o_1m_18500201-18500301_grid-T.nc is a HDF5 or NetCDF4 file 
can only read classic data model format NetCDF4 files
Creating classic format NetCDF file NEMO_water_salinity_theta_monthly.nc 
Creating classic format NetCDF file NEMO_water_salinity_theta_monthly.nc 
Creating classic format NetCDF file NEMO_water_salinity_theta_monthly.nc 
Creating classic format NetCDF file NEMO_water_salinity_theta_monthly.nc 
Creating classic format NetCDF file NEMO_water_salinity_theta_monthly.nc 
Creating classic format NetCDF file NEMO_water_salinity_theta_monthly.nc 
Creating classic format NetCDF file NEMO_water_salinity_theta_monthly.nc 
Creating classic format NetCDF file NEMO_water_salinity_theta_monthly.nc 

It seems to be going in a loop and re-writing the same file over and over.
What I want to get is a single netcdf file containing selected variables.

Thanks,

Vittoria

comment:5 Changed 9 months ago by jeff

  • Resolution answered deleted
  • Status changed from closed to reopened

Hi Vittoria

Can you change the file permissions so I can look at your directories on /nerc. Running

chmod -R g+rX /nerc/n02/n02/vittoria

will give access to your directory to anybody in the n02 group. I noticed directory u-as245 has an access control list (ACL) defined on it, which gives read permission to user cjrw09 and no group permission on the directory, is there a reason for this?

Jeff.

comment:6 Changed 9 months ago by mvguarino

Hi Jeff,

I have done it, let me know if you can access it now.

I was trying to give permission to cjrw09 so that he could access some model outputs, but I was unsuccessful as the permissions of u-as245 appear to be weird.

Vittoria

comment:7 Changed 9 months ago by jeff

No I can't access the directory, did you run the command I gave you?

Jeff.

comment:8 Changed 9 months ago by mvguarino

yes, I did.
Have done it another time now, just in case.

Vittoria

comment:9 Changed 9 months ago by jeff

I believe the problem is with the acl you have on the directory, do you mind deleting this? If so run

setfacl -b u-as245

in the /nerc/n02/n02/vittoria directory. Then run the chmod command again.

Jeff.

comment:10 Changed 9 months ago by mvguarino

I have run:

setfacl -b u-as245
chmod -R g+rX /nerc/n02/n02/vittoria

let me know if it worked,

thanks

vittoria

comment:11 Changed 9 months ago by jeff

Hi Vittoria

Thanks I can access that directory now.

Your script seems to be working as designed. It looks like you are running the same convsh script 8 times, are you running this in a parallel queue?

Jeff.

comment:12 Changed 9 months ago by mvguarino

Hi Jeff,

Thanks for looking into that.
It was silly of me to leave uncommented the mpi command in the submit.pbs file, I was trying to understand whether I could run also things in parallel.
Now I understand why the getNEMO.tcl was executed multiple times…
However, this doesn’t solve my problem.

Although the script runs, the output file contains “sea water salinity” and “sea water potential temperature” (fields: 14 and 15) only of one of the two input files, and not for both of them.
What I need to get is single nc file with 2 time levels.

Vittoria

Last edited 9 months ago by mvguarino (previous) (diff)

comment:13 Changed 9 months ago by mvguarino

An update on this:

I think the problem is not the script itself but xconv. I tried manually to join the two files together and, just like when I run the script, xconv gets stuck after the requested file is created and keeps on running without actually doing anything.
This happens both when I use xconv from /nerc/n02/n02/vittoria , and when I use xconv1.93 once I logged into the DAC (note that I had to download and install xconv1.93 because it wasn’t previously installed).

Vittoria

comment:14 Changed 9 months ago by jeff

Hi Vittoria

The problem is both files have the same time value (0) so xconv/convsh won't be able to put them into a single file.

Jeff.

comment:15 Changed 9 months ago by jeff

Hi Vittoria

Looking at your files further it seems they do have different time values, but they are using a CF netCDF convention that xconv doesn't support. You could try using CF-python (https://cfpython.bitbucket.io/) to do the conversion. This tool is available on the rdf,jasmin and archer.

Jeff.

comment:16 Changed 9 months ago by mvguarino

Hi Jeff,

Thanks, I'll give it a go.

Vittoria

comment:17 Changed 8 months ago by willie

  • Component changed from Other to UM Tools
  • UM Version <select version> deleted

comment:18 Changed 7 months ago by willie

  • Resolution set to fixed
  • Status changed from reopened to closed
Note: See TracTickets for help on using tickets.