Run time suddenly explodes in certain iteration

[simonschaefers]

Hi everyone,

I am advecting particles on repeated year velocity data (~10^5 particles, 120 years) for different time averages. As for computational limits, I only advect for 2 years at once and repeat the simulation with the latest position of the particles.

This works fine in general, but it has occurred to me multiple times, that at one iteration, the expected run time jumps from the "normal" ~30min to >100h.

Current simulation year:  26
[INFO] using particle set from previous particles
[INFO] detecting & (re)moving stuck particles
[INFO] particle status:
previously_stuck :  27398
horizontally_stuck :  28635
vertically_stuck :  28248
too_far_north :  843
deleted :  348
[INFO] moved 3118 stuck particles to (lon: 300, lat: 0, depth: 1000) 
INFO: Output files are stored in <zarr.storage.MemoryStore object at 0x15554738e9b0>.

  0%|          | 0/63072000.0 [00:00<?, ?it/s]
  0%|          | 86400.0/63072000.0 [00:05<1:10:57, 14793.87it/s]
  0%|          | 172800.0/63072000.0 [00:08<50:45, 20652.01it/s] 
  0%|          | 259200.0/63072000.0 [00:11<44:15, 23652.58it/s]
  1%|          | 345600.0/63072000.0 [00:14<41:25, 25236.16it/s]
  1%|          | 432000.0/63072000.0 [00:17<39:37, 26341.74it/s]
  1%|          | 518400.0/63072000.0 [00:21<38:32, 27049.14it/s]
  1%|          | 604800.0/63072000.0 [00:24<37:50, 27516.33it/s]

Current simulation year:  28
[INFO] using particle set from previous particles
[INFO] detecting & (re)moving stuck particles
[INFO] particle status:
previously_stuck :  30516
horizontally_stuck :  31682
vertically_stuck :  31451
too_far_north :  785
deleted:  314
[INFO] moved 3039 stuck particles to (lon: 300, lat: 0, depth: 1000) 
INFO: Output files are stored in <zarr.storage.MemoryStore object at 0x15554738e9b0>.

  0%|          | 0/63072000.0 [00:00<?, ?it/s]
  0%|          | 86400.0/63072000.0 [08:25<102:18:49, 171.00it/s]
  0%|          | 86400.0/63072000.0 [08:40<102:18:49, 171.00it/s]
  0%|          | 172800.0/63072000.0 [16:35<100:22:39, 174.06it/s]
  0%|          | 172800.0/63072000.0 [16:50<100:22:39, 174.06it/s]
  0%|          | 259200.0/63072000.0 [25:03<101:21:41, 172.14it/s]
  0%|          | 259200.0/63072000.0 [25:20<101:21:41, 172.14it/s]
  1%|          | 345600.0/63072000.0 [34:11<104:50:40, 166.19it/s]

In previous simulations, I could cope with the error by displacing any irregular particles to land coordinates, but now the error is back.

I avoid boundary-related errors and take care of stuck particles, however, I am clueless on how the runtime length performs this sudden jump before ramping up. The computation nodes I am working on remain constant and restarting one iteration earlier gives the same result (one step is working fine, but the next step (same as before) fails).

Next to particle deletion, the following error occurs from time to time, however, this has not influenced the run time in the first iterations:

WARNING: Deleting particle 47xxxx because of non-recoverable error

As I have not changed anything from one iteration to the next, is there any reasonable mechanism in the parcels code that could produce such errors, or has anyone experienced similar issues?

Thanks in advance!

(I can provide code if helpful)

[erikvansebille]

Thanks for flagging, @simonschaefers; I must say I have never seen this extreme type of jump in ETA before. And I almost never see particles being deleted because of non-recoverable errors... Which version of parcels do you run on?

For some background about the efficiency of Parcels, check out Christian Kehl's paper at https://doi.org/10.1016/j.cageo.2023.105322. There may be some inspiration on what could cause the issue in your case in there?

One question: why do you move particles on land? Why not delete them? Or, alternatively, set particle.status=StatusCode.StopExecution (see the tutorial here)?

[simonschaefers]

Hi Erik,
thank you for answering! I run on version 3.0.2.

I assume there is some connection to particles being close to a boundary error, as the failed simulations seem to be those with lower accuracy of the particle movement (additional displacement, averaged fields, see table).

feature \ velocity configuration	daily mean	seasonal mean	yearmean
no manipulation	success	success	failed
displacement at boundaries *	success	failed	not yet tried

• following https://docs.oceanparcels.org/en/latest/examples/documentation_unstuck_Agrid.html

Regarding the paper, I optimized my simulation to the nodes of Levante (https://docs.dkrz.de/doc/levante), especially targeting the I/O traffic. This was a major issue before and presumably, there is a lot more to optimize, but I don't think this causes the issue. But I'll look into further optimization, maybe the issue disappears as a side effect.

I move the particles to land as I restart the simulation with the last positions of the particles (using .from_particlefile caused issues with the time coordinate). It is not the most sophisticated solution, but in this way, I was able to maintain the particle IDs and didn't have to cope with nan values.
Of course, it amasses particles at one position, but it has not affected the run time so far.
Stopping execution sounds like an easy way to disentangle my workaround, thank you!

[eavellashaw]

Hi @simonschaefers, I actually experienced the same warning for the non-recoverable error when decreasing the timestep calculation (dt) from 1 hours to 12 hours or to 1 day. I am still waiting for the jobs to finish to see if that impacted the runs in some ways, I'll let you know!