(no) Pre batch processing task state (Option 3)

[dwsutherland]

closes #6567, closes #6860, closes #6859

Tasks batched (for DB read) during a graph walk, and processed at the end of that walk.

Window resizes typically include many walks (for each active task), so the DB will have that many calls.
However, no state-less tasks will exist.

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Code covered by existing tests.
• Changelog entry added.
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[oliver-sanders]

This seems like the right sort of fix (i.e, get the task status right first time).

If I'm reading this correctly:

• Before we were performing all the DB lookups at once.
• This change splits it up into smaller batches.

If I'm understanding correctly, the root cause is that the data store is trying to push out incomplete deltas before it has actually gathered all of the required information. This (only?) happens on reload.

If so, could we not just delay sending the deltas (after reload) until they are complete (i.e, after we have finished the first walk)?

[dwsutherland]

If I'm understanding correctly, the root cause is that the data store is trying to push out incomplete deltas before it has actually gathered all of the required information. This (only?) happens on reload.

it applies stateless deltas before the historical batch is applied, yes.

This change splits it up into smaller batches.

It also processes the batch before any associated deltas are applied too.
The batches are 1 graph window walk.

[dwsutherland]

Have rebased this against 8.6.x, it fixes the bug.

[oliver-sanders]

If I'm understanding correctly, the root cause is that the data store is trying to push out incomplete deltas before it has actually gathered all of the required information. This (only?) happens on reload.

it applies stateless deltas before the historical batch is applied, yes.

I'm not convinced this is the case.

The data store shouldn't yield any deltas until the process has completed (both logically and from the code):

cylc-flow/cylc/flow/data_store_mgr.py

Line 707 in a12a458

self.publish_deltas = self.get_publish_deltas()

I have also managed to reproduce this example without any historical data to load - #6567 (comment)

Unfortunately, the issue still reproduces on this branch :(

[dwsutherland]

The data store shouldn't yield any deltas until the process has completed (both logically and from the code):

Yes, but TUI doesn't get publish deltas, it asks for a dump..
However, yes, I don't think it should be possible.. what get's applied to the data-store should be added deltas (empty state) that have had the historical info applied (actual state).. So what TUI queries should be a coherent data-store where there are no stateless tasks ..

Maybe I need to go back to the drawing board of what I think is happening

[oliver-sanders]

(unassigning this to re-focus my review queue, feel free to re-assign when undrafted)