Opened 5 years ago

Closed 5 years ago

#1252 closed help (completed)

Transferring data from Hector to Archer

Reported by: charlie Owned by: annette
Component: Data Keywords: HECToR, data, file sizes
Cc: Platform: ARCHER
UM Version: <select version>

Description

Hello,

I'm trying to transfer all of my data on /home and /work from Hector to Archer, ready for the Hector shutdown next week, but struggling to know how best to do this.

I can't use sftp because this doesn't allow transfer of directories (and files therein), and given that some of my directories have numerous subdirectories it would take forever to manually transfer each one.

I was using scp but, although this works, not all the data appears to have transferred correctly, e.g.

scp -r xiogc cjrw14@…:/home/n02/n02/cjrw14/um

Then, on Hector:

hector-xe6-7:cjrw09$ du -skh xiogc
393M xiogc

but, on Archer:

eslogin001:cjrw14$ du -skh xiogc
229M xiogc

Why has it only transferred some of the data?

Apologies for the basic question,

Charlie

Change History (5)

comment:1 Changed 5 years ago by annette

Hi Charlie,

I have also noticed with some of my transfers that the data sizes looked a bit different. If you retry the transfer with the rsync command it will check the files are copied over correctly and add the additional ones if necessary.

To do this on Archer (pulling the data over from HECToR):

rsync -avzhe ssh <hector-userid>@login.hector.ac.uk:<path-to-folder-on-hector>/ <path-to-folder-on-archer>/

Note the '/' on the end of the directory names.

If the rsync command succeeds without transferring anything it means the directories match.

Hope this helps,
Annette

comment:2 Changed 5 years ago by charlie

Thanks very much, Annette. I have tried transferring my data using both scp and rsync, and both seem to work. However, the file sizes are still quite different. I have checked both very closely, and the individual number of files (and their sizes) match on both machines. When using du -skh * they don't. See below for an example. As I said, however, I have looked closely at the individual files, and they have all transferred correctly. So is it possible that there is simply a problem with du?

Charlie

—-

on Hector:

hector-xe6-4:cjrw09$ du -skh *
1.8M d_mods
34G umui_out
393M xiogc
393M xiogd
393M xioge
393M xiogf
393M xjfja
393M xjfjb
393M xjfjc
393M xjfjd
394M xjjda
394M xjjdb
394M xjjdc
394M xjjdd

whereas on Archer:

eslogin003:cjrw14$ du -skh *
656K d_mods
4.0K umui_out
229M xiogc
229M xiogd
229M xioge
229M xiogf
229M xjfja
229M xjfjb
229M xjfjc
229M xjfjd
229M xjjda
229M xjjdb
229M xjjdc
229M xjjdd

comment:3 Changed 5 years ago by annette

  • Keywords HECToR, data, file sizes added
  • Owner changed from um_support to annette
  • Platform changed from HECToR to ARCHER
  • Status changed from new to assigned

Hi Charlie,

This is probably one for the Archer helpdesk - you can post a query through SAFE:

https://www.hector.ac.uk/safe/

Annette

comment:4 Changed 5 years ago by annette

By the way rsync does a checksum to ensure the source and target files are the same, so I wouldn't worry about that. Must be something system specific between the machines.

Annette

comment:5 Changed 5 years ago by annette

  • Resolution set to completed
  • Status changed from assigned to closed

Hi Charlie,

I'm going to close this ticket now, but if you want to add any further information feel free.

Annette

Note: See TracTickets for help on using tickets.