perf(aggregation): recursive XMSS aggregation exceeds slot-interval budget, stalling finalization under load

[mananuf]

Summary

On a single-node run (slots 0→362, ~23 min), the node finalized healthily through ~slot 244, then finalization froze (justified stuck at 279, finalized at 274) for the final ~75 slots while the head kept advancing to 355. Root cause: XMSS recursive aggregation duration crosses from milliseconds into multiple seconds around slot 260, blowing the per-slot proving budget. This is a performance/scaling ceiling, not a correctness bug — no ERROR/PANIC, no state-transition failures, proc_time healthy, peers stable at 2 the whole run.

Evidence

Aggregation duration blows the slot budget. A slot is 4s / 5×800ms intervals; aggregation runs in interval 2 and shares the ProvingGate. Max [signature] aggregate ... duration= per 40-slot bucket:

slots	max aggregation
200–239	916 ms
240–279	2816 ms
280–319	3328 ms
320–359	2726 ms

Worst single aggregations: 3.33s (slot 296), 2.95s (312), 2.82s (261) — 3–4× the 800ms interval, ~70–80% of the whole 4s slot.

Cascade (all downstream of the above):

1. Vote starvation → finalization stall. [signature] aggregation session hit budget: ... produced=1 begins at slot 261 — exactly the stall onset. Only 1 aggregate published per slot → network never reaches the ⅔ supermajority → justified freezes at 279, finalized at 274. Attestations-packed-per-block drop to 0.5 (slots 240–279) from ~1.0–1.2 earlier.
2. Node stops proposing. Last successful [validator] proposed block is slot 314; it keeps attempting (317, 320 … 353) but every one hits proposal worker busy / discarding proposal — the ProvingGate is held for ~3s by the aggregation.
3. Node stops attesting. [validator] duty gate closed: local view is stale, duty=attestation (4×) — the stale-view guard suppresses attestation duty as the node's own view lags, further shrinking the vote pool.

Timeline

Phase	Slots	State
Healthy	0–244	finalized tracks justified ~3 behind, both follow current slot
Degrading	~250–284	justification stalls at 238–239, recovers once to 274
Stalled	284–362	justified frozen at 279, finalized frozen at 274, head → 355

Ruled out

• Peers: held at 2 the entire run (dropped to 1 only in the final status box) — not a partition.
• Connectivity/gossip: received block steady 4–6/min to the end; the ~33 negotiate protocol: context deadline exceeded req/resp timeouts are thin, spread throughout, and don't correlate with the stall onset.
• Tick/state: proc_time median 49 ms, max 706 ms (< 800 ms); no decode/oversized-block issues; 12 reorgs, all depth 1–2 (normal).

Why it matters

This is the recursive-aggregation-vs-slot-interval limit: as per-slot aggregation cost grows, a single aggregation eventually exceeds the interval budget, and the node degrades into a non-finalizing, non-proposing, partially-attesting state even though blocks keep flowing. It defines the load at which one gean node stops finalizing (~slot 260, aggregation ≳2s here).

Proposed direction

• Bound per-slot recursive aggregation cost so it fits the ~800ms interval window (cap tree depth / batch size, or shed work), rather than letting produced collapse to 1 under load.
• Investigate whether aggregation can be parallelized / precomputed off the interval so the ProvingGate isn't held for multiple seconds (which also starves the proposal worker).
• Add a metric/alert on aggregation duration p95 vs. the interval budget (plus produced=1 occurrences) so the finalization stall is predictable rather than found post-hoc.

Follow-ups / open questions

• Pull the raw= / children= / total= fields from the slow aggregations to attribute the cost spike (attestation count vs. proof-tree depth).
• Confirm whether this run used real XMSS proving or the shadow rate model — the threshold is useful either way, but it changes how directly the durations map to production hardware.

(Analysis from a single-node log, slots 0–362, 2026-07-13 15:09–15:32.)