Opened 3 years ago

Closed 3 years ago

Last modified 2 years ago

#1959 closed defect (answered)

Self-copying fcm.bld.lock

Reported by: Emre Owned by: um_support
Priority: high Component: UM Model
Keywords: Cc:
Platform: ARCHER UM Version: 8.4

Description

Dear People,
I ran into a strange situation where my jobs (xmufc, version 8.4) suddenly started failing in the compilation process. When I checked in archer, I found that fcm.bld.lock files were generated which interrupted and terminated the compilation (as I learned from the previous tickets <thanks>). The problem is, even after I delete the *lock files in the umatmos and umrecon folders, they somehow get re-generated each time when the exact same jobs are sent. These are the jobs which used to compile fine. So, for some reason the submission process produces *lock files by itself from the beginning. In some jobs I was able manually get into the folder and delete the *lock file while the compilation was still running and at the end they compiled fine. But it is cumbersome to repeat the same deletion process in each job. Besides, sometimes, it does not work.


Change History (10)

comment:1 Changed 3 years ago by Emre

Thanks very much for your times

comment:2 Changed 3 years ago by Emre

  • Owner changed from um_support to Gren
  • Status changed from new to assigned

comment:3 Changed 3 years ago by Emre

  • Owner changed from Gren to UM_support

comment:4 Changed 3 years ago by Emre

  • Owner changed from UM_support to um_support

comment:5 Changed 3 years ago by Emre

To do some more further checks I copied the job xmufc to xmuff. Now, strangely, (due to my manual fixing attempts) xmufc job creates fcm.bld.lock file only in the umrecon folder while xmuff creates fcm.bld.lock files in the umatmos folder. If you look at the time of these *lock files you will see that they are generated at the same time together with other files (such as the Makefile).

comment:6 Changed 3 years ago by Emre

I should also provide you the location of the directory
/home/n02/n02/emre/um/xmufc AND /home/n02/n02/emre/um/xmuff

Similar problems occur in other jobs as well

comment:7 Changed 3 years ago by ros

Hi Emre,

The fcm.bld.lock file is created when a compile begins in order to prevent multiple compiles running simultaneously. The umatmos is first followed by the umrecon, hence why you don't see an fcm.bld.lock file in both directories simultaneously. When the compile finishes the fcm.bld.lock is automatically removed, however, if the compile is abruptly terminated, e.g. if you kill the job or it runs out of wallclock time the fcm.bld.lock file is left behind and has to be manually removed before you can continue the compile.

Looking at your .comp.leave files I can see that all your jobs (xmufc, xmufa, xmufd and xmuff) have failed at somepoint due to wallclock being exceeded. This would be why you encountered problems with the fcm.bld.lock file. I also notice that you have some compilation errors in a couple of the jobs which will need fixing.

Please increase the compilation time for your jobs from 1hour to 3hours and resubmit. The compilation time can vary from day to day depending on the load on the serial nodes. Sometimes a compile will complete in 1hour and at others it make take a bit longer.

Regards,
Ros.

comment:8 Changed 3 years ago by ros

  • Resolution set to answered
  • Status changed from assigned to closed

I assume, due to lack of activity, that this fixed your problem so will now close this ticket.

Regards,
Ros.

comment:9 Changed 2 years ago by Emre

I finally found this ticket. Why does it not appear in my TRAC?
Thank you very much Rose and Grenville. I got the answer.
I am not sure if my thanks will reach you as the ticket is closed ?

comment:10 Changed 2 years ago by ros

Hi Emre,

You should have received an email when we respond to a ticket. I assume the TRAC view you are referring to is when you logged in and at this URL: http://cms.ncas.ac.uk/query - Is that correct? If so this view by default lists all tickets that are owned by you (you are the reporter not the owner) and also if the ticket status isn't closed. You can adjust the list by changing the filters at the top of the page and clicking update.

Cheers,
Ros.

Note: See TracTickets for help on using tickets.