Avoid tidying in batch

[mandresm]

Email from @mjy1011:

Hi, Miguel

Thank you so much for the "intermediate file movements" work. It is so nice of you to spend much time on this.
I just have started testing it. I'll let you know how it goes.

By the way,
Is is possible to run "compute" only on LSF batch ? Prepcompute and tidy job takes the same time whether in or out of LSF batch system.
KMA machine users always have to yield the jobs (killed) whenever an operational job is submitted. These days, the node usage is crazy, almost 99.9 %.
So before EGU, I could run the experiment with two or three parallel jobs (300 nodes), but after EGU, I can't use even 100 nodes and this job is often killed.
The worst part is after 9 hours of compute, during the tidy job, it is often killed, then all the data are lost, sometimes restart files.. then I have to re-run again and again...
Crazy.. right?

There are too many obstacles here in KMA so if my question is not a proper one, then it's fine.
Just curious and want to ask you opinion how to reduce time of experiment with the intermediate file movements!

Thank you so~~~~ much!

Best,
Jayeon

[mandresm]

Dear @mjy1011,

Thank you so much for the "intermediate file movements" work. It is so nice of you to spend much time on this.
I just have started testing it. I'll let you know how it goes.

You're welcome :)

Is is possible to run "compute" only on LSF batch ?

Yes it is. Not a fancy nice way, but there is a manual way:

1. Your standard esm_runscripts command with -c flag: esm_runscripts <your_runscript.yaml> -e <expid> --open-run <your other flags> -c
2. After new run folder is prepared, cd <PATH_to_exp>/<expid>/run_<DATE>/scripts/
3. Edit the .run file there, commenting the esm_runscripts line below the launching of the model.
4. Launch the .run file with your job scheduler. In slurm it would be sbatch <my_script>.run
5. Once compute is finished, launch from the login node the esm_runscripts command you commented from the .run script and make sure you add a -c to it so that it stops before submitting the simulation.
6. repeat steps 2 to 5 for the following run.

That being said, if there is a problem during tidying you can always resume from tidying without the need to compute, by running the esm_runscripts command in the .run script. That should solve the problem. If tidy has finished before it broke the date and run in <PATH_to_exp>/<expid>/scripts/*.date has been updated and you will need to manually reverted back manually, both the date and the run number before you submit the esm_runscripts ... -t observe ... command. The part of the manual date update will be solved automatically in the future, I'll let you know when that is integrated into release. I am also working in a parallelization of file copying, that I have already on testing phase. That only parallelizes things in a single node. In the future I will work also in a parallelization in multiple nodes.

Please, let me know if that make sense to you, and that I am not misinterpreting your email.

@JanStreffing

[mjy1011]

This is what I've expected. Thank you so much!

[mjy1011]

Can I ask one more question?
If rcf and waminfo files are lost during the run, is there a way to manually create them (copying from previous file and modify) ?

[JanStreffing]

Yes, i would suggest you look at the difference between two old runs and try to apply that difference on top of your last files. in the end, these are just a bunch of dates and names that change.

try to ensure, that the differences you compute and the files you apply them to have the same leap year status.

[mjy1011]

Thank you, Jan. It's a bit complicated with the numbers but I'll try :)