Use coordinator state discovery and improve bot diagnostics (#720)

* a

* Format coordinator discovery test

* Fix dashboard audit findings

* Harden bot error redaction and legacy recovery

- Redact URLs and relative paths from public poll errors
- Recover replacement credits for legacy positions using quorum log scans

* Harden canonical recovery snapshots

Author: KillariDev

bots/open-oracle-arbitrager/README.md

@@ -79,7 +79,13 @@ for the report lifecycle assumptions and economics used by the arbitrager.
 ## Requirements
 
 - Bun and this project's frozen dependencies.
-- An RPC endpoint for Ethereum mainnet or Sepolia. An archive-capable endpoint is recommended when
+- An RPC endpoint for Ethereum mainnet or Sepolia. Approved-coordinator discovery
+  uses current contract state and does not require historical log access. Legacy
+  journals without a persisted dispute index are the bounded exception: after a
+  restart, replacement-credit recovery scans only that report's dispute logs from
+  its entry block until the immediate successor. Execution RPCs must therefore
+  retain log history back to the oldest open legacy position's entry block. Archive
+  access can also be useful for coordinator-free diagnostic mode when
   `runtime.lookbackBlocks` reaches beyond the provider's retained log history.
 - The deployed OpenOracle contract address.
 - At least one reviewed Zoltar `OpenOraclePriceCoordinator` address for every
@@ -268,36 +274,33 @@ available when RPC validation is temporarily unavailable, reports
 `connectivity-degraded`, and retries with bounded backoff. The bot checks the chain
 before every scan.
 
-Startup enters **Syncing** while the bot scans the configured historical lookback in
-100-block chunks. The deliberately bounded response size prevents permissionless
-OpenOracle event volume from turning one historical RPC response into an unbounded
-memory spike. Once caught up, it polls the latest head and covers every unseen
-height in the OpenOracle event-log query; if several blocks arrive between polls, no
-event-log height is skipped. Opportunity evaluation and pool sampling run once at
-the newest observed head, not once at every intermediate historical height. A
-12-block overlap is re-read and reconciled for shallow reorgs in event-derived
-report state. With no new head the bot remains **Running** without re-evaluating or
-writing duplicate price samples.
+With approved coordinators configured, startup discovers reports by reading each
+coordinator's `pendingReportId` at one fixed block and then reads the corresponding
+stored OpenOracle state. It repeats those current-state reads at each new head and
+does not query historical OpenOracle logs. Execution mode requires a quorum to agree
+on the block and report snapshot. Opportunity evaluation and pool sampling run once
+at the newest agreed head. With no new head the bot remains **Running** without
+re-evaluating or writing duplicate price samples.
+
+Coordinator-free diagnostic mode retains the historical fallback: startup scans
+`runtime.lookbackBlocks` in 100-block log-query chunks, then reads a 12-block overlap
+at each new head for shallow reorganization handling. The deliberately bounded
+response size prevents permissionless event volume from producing an unbounded RPC
+response.
 
 ### Data freshness and retention
 
-Startup lookback backfills OpenOracle events, but it does not backfill historical
-pool prices. The price file can retain a sample from a block displaced by a reorg;
-the 12-block reconciliation applies only to report events. The bot reads at most the
-latest 8 MiB, loads and charts the latest 2,000 valid price records, and atomically
-compacts the file to those records after it crosses 8 MiB. Approved-coordinator report
-paths are reconstructed in memory from the startup lookback plus events observed by
-the current process. Consequently a path can begin at a dispute when its submission
-predates the lookback, and a settlement-only report is not shown when no earlier
-event for that report was observed. For each active report, the scanner retains one
-pre-overlap state anchor plus every event in the 12-block reorg window; older dispute
-steps are compacted instead of replayed forever. Settled paths remain through that
-reorg window and are then removed from the live scanner; confirmed transaction
-history remains in the execution history file. Unapproved reports are not retained
-in the execution cache. In diagnostic mode without configured coordinators, at most
+The price file can retain a sample from a block displaced by a reorganization. The
+bot reads at most the latest 8 MiB, loads and charts the latest 2,000 valid price
+records, and atomically compacts the file to those records after it crosses 8 MiB.
+Approved-coordinator monitoring keeps the complete current state of each pending
+report but does not reconstruct its historical dispute path from logs. Reports that
+are no longer pending are removed from the live cache; confirmed bot transaction
+history remains in the execution history file. In coordinator-free diagnostic mode,
+the startup lookback backfills report events but not historical pool prices. At most
 256 reports and 64 permissionlessly observed tokens are retained so event spam
-cannot create ever-growing per-block work. Increase `runtime.lookbackBlocks` when
-complete active-game context is operationally important.
+cannot create ever-growing per-block work. Increase `runtime.lookbackBlocks` only
+when broader diagnostic event history is operationally important.
 
 ## Run on Sepolia
 
@@ -630,9 +633,8 @@ The dashboard shows:
 - Selected network, expected chain ID, read/public RPC controls, and endpoint checks.
 - A local signer control, connected address, and its ETH/WETH/REP balances.
 - ETH, WETH, REP, executable REP value, and estimated portfolio value.
-- Native ETH stakes, WETH stakes, and ETH settler rewards locked in active games
-  observed within the configured event lookback. The combined figure treats 1 WETH
-  as 1 ETH and can undercount games created before that lookback.
+- Native ETH stakes, WETH stakes, and ETH settler rewards locked in reports currently
+  pending on configured coordinators. The combined figure treats 1 WETH as 1 ETH.
 - Current opportunities, token-metadata-normalized inventory requirements, deadline
   window, token-specific direction, pool, and decision.
 - Durable positions with actual hedge execution, entry and lifecycle gas, exact
@@ -658,8 +660,10 @@ The dashboard shows:
   address links to the selected-network explorer. The [market discovery section](#token-and-pool-discovery)
   owns the venue, price, and liquidity semantics. A token with no supported pool is
   explicitly labeled instead of disappearing.
-- The submitted/disputed/settled events observed for each OpenOracle report,
-  including blocks, reporters, raw locked amounts, and transaction links. See
+- In coordinator-free diagnostic mode, the submitted/disputed/settled events
+  observed for each OpenOracle report, including blocks, reporters, raw locked
+  amounts, and transaction links. Configured-coordinator mode shows current report
+  state without reconstructing these historical paths. See
   [data freshness and retention](#data-freshness-and-retention) for lookback limits.
 - A per-asset current-head price-history chart with one series per supported pool,
   axes, point tooltips, and a recent exact-value table. Samples persist across
@@ -1050,7 +1054,7 @@ scan. The same values live under `strategy` in the complete configuration:
 | TWAP window | `1800 seconds` | `twapSeconds` | Controls the Uniswap manipulation-resistance window. Minimum: 60 seconds. |
 | Remaining time | `36 seconds` | `minimumRemainingSeconds` | Inclusion buffer for timestamp-based games. |
 | Remaining blocks | `3 blocks` | `minimumRemainingBlocks` | Inclusion buffer for block-based games. |
-| Head poll interval | `1000 ms` | `pollMilliseconds` | Delay between latest-head checks. Every unseen event-log height is queried. |
+| Head poll interval | `1000 ms` | `pollMilliseconds` | Delay between latest-head checks. Coordinator-free diagnostic mode queries every unseen event-log height. |
 
 Increasing profit thresholds reduces execution frequency. Increasing the TWAP
 window or remaining-time buffers is generally more conservative, while decreasing
@@ -1271,10 +1275,10 @@ entry from depending on wallet inventory already committed to recovery.
 - Private delivery reduces public-mempool exposure but does not guarantee
   confidentiality, inclusion, fair ordering, or relay/builder behavior. Configuring
   multiple relays shares the signed payload with every listed operator.
-- A 12-block event overlap is replayed whenever a new head is processed to tolerate
-  short reorganizations. The retained pre-overlap block hash is checked on every
-  poll; a deeper reorganization stops execution and requires restart so the complete
-  lookback is rebuilt. Operators still need independent alerting.
+- Approved-coordinator reports are reread from a fixed block whenever a new head is
+  processed. A retained block hash is checked on every poll; a deeper reorganization
+  stops execution and requires restart. Coordinator-free diagnostic mode separately
+  replays a 12-block event overlap. Operators still need independent alerting.
 - Continuous mode retries transient poll failures with bounded exponential backoff.
   The dashboard exposes per-endpoint health and the latest error, `/healthz` supports
   container supervision, and Compose restarts an unexpectedly exited process.

bots/open-oracle-arbitrager/scripts/capture-docs-screenshots.mts

@@ -3,7 +3,7 @@ import { tmpdir } from 'node:os'
 import { join } from 'node:path'
 import { getAddress, keccak256, toHex } from '#ethereum'
 import { startDashboardServer } from '#dashboard/dashboard-server'
-import { operatorSnapshot, type OperatorSnapshot, type OperatorState } from '#state/operator-state'
+import { operatorSnapshot, publicOperatorFailure, publicPollFailure, type OperatorSnapshot, type OperatorState } from '#state/operator-state'
 import type { PositionRecord } from '#state/position-store'
 
 const address = (value: number) => getAddress(`0x${value.toString(16).padStart(40, '0')}`)
@@ -20,6 +20,13 @@ const pool = address(0x3000)
 const hash = transactionHash('open-oracle-documentation-fixture')
 const checkedAt = sampledAt(0)
 const protectedFailureMarker = 'operator-secret'
+const longProviderFailureDetail = ` ${'provider response detail '.repeat(30).trim()}`
+const rawRpcFailure = `Read RPC https://operator:${protectedFailureMarker}@rpc.example failed at block 23842152:${longProviderFailureDetail}`
+const rawRelayFailure = `Private relay https://operator:${protectedFailureMarker}@relay.example rejected the transaction:${longProviderFailureDetail}`
+const expectedRpcPollFailure = publicPollFailure(rawRpcFailure)
+const expectedRpcOperatorFailure = publicOperatorFailure(rawRpcFailure)
+const expectedRelayOperatorFailure = publicOperatorFailure(rawRelayFailure)
+const expectedStateUnavailableFailure = `${publicPollFailure('fixture state endpoint unavailable', 'load the latest operator state for the dashboard')} Use Refresh to retry now.`
 let fixtureStatus: OperatorSnapshot['status'] = 'running'
 let paused = false
 let fixtureAttention: 'error' | 'none' | 'recovery' | 'transaction' = 'none'
@@ -740,15 +747,21 @@ async function captureScreenshots(chromium: string, origin: string, outputDirect
 									attentionText: attention?.textContent,
 									bodyScrollWidth: document.body.scrollWidth,
 									clientWidth: document.documentElement.clientWidth,
+									disputePathsEmptyText: document.querySelector('#dispute-paths-empty')?.textContent,
 									hash: window.location.hash,
 									label: badge?.textContent,
 									noticeCopy: document.querySelector('#notice-copy')?.textContent,
 									bodyContainsCredential: document.body.textContent?.includes('operator-secret') === true,
 									endpointText: document.querySelector('#endpoint-checks')?.textContent,
 									noticeTitle: document.querySelector('#notice-title')?.textContent,
 									noticeTone: notice instanceof HTMLElement ? notice.dataset.tone : undefined,
+									operationDetailsWidth: document.querySelector('#operations-body td[data-label="Details"]')?.getBoundingClientRect().width,
+									operationReasonWidth: document.querySelector('#operations-body td[data-label="Why"]')?.getBoundingClientRect().width,
+									operationTableWidth: document.querySelector('#operations-body')?.closest('table')?.getBoundingClientRect().width,
 									operationText: document.querySelector('#operations-body')?.textContent,
 									status: badge instanceof HTMLElement ? badge.dataset.status : undefined,
+									transactionTableWidth: document.querySelector('#transactions-body')?.closest('table')?.getBoundingClientRect().width,
+									transactionTargetWidth: document.querySelector('#transactions-body td[data-label="Target results"]')?.getBoundingClientRect().width,
 									transactionText: document.querySelector('#transactions-body')?.textContent,
 								}
 							})()`,
@@ -758,6 +771,9 @@ async function captureScreenshots(chromium: string, origin: string, outputDirect
 					)
 					const value = typeof state === 'object' && state !== null && 'result' in state && typeof state.result === 'object' && state.result !== null && 'value' in state.result ? state.result.value : undefined
 					if (typeof value !== 'object' || value === null || !('status' in value) || value.status !== status) throw new Error(`Run badge did not render ${status}`)
+					if (!('disputePathsEmptyText' in value) || value.disputePathsEmptyText !== 'No historical dispute path is available. Configured coordinator mode reads current reports directly, while coordinator-free diagnostic mode reconstructs paths from its configured event lookback.') {
+						throw new Error('Empty dispute paths did not explain both discovery modes')
+					}
 					if (
 						status === 'error' &&
 						(!('attentionHref' in value) ||
@@ -773,18 +789,34 @@ async function captureScreenshots(chromium: string, origin: string, outputDirect
 							!('noticeTitle' in value) ||
 							value.noticeTitle !== 'Latest poll failed' ||
 							!('noticeCopy' in value) ||
-							value.noticeCopy !== 'RPC connectivity or canonical chain reads failed. Automatic retry remains active.' ||
+							value.noticeCopy !== expectedRpcPollFailure ||
 							!('bodyContainsCredential' in value) ||
 							value.bodyContainsCredential !== false ||
 							!('endpointText' in value) ||
 							typeof value.endpointText !== 'string' ||
-							!value.endpointText.includes('RPC connectivity or canonical chain reads failed. Automatic retry remains active.') ||
+							!value.endpointText.includes(expectedRpcOperatorFailure) ||
+							!('operationDetailsWidth' in value) ||
+							typeof value.operationDetailsWidth !== 'number' ||
+							value.operationDetailsWidth > 480 ||
+							!('operationReasonWidth' in value) ||
+							typeof value.operationReasonWidth !== 'number' ||
+							value.operationReasonWidth > 480 ||
+							!('operationTableWidth' in value) ||
+							typeof value.operationTableWidth !== 'number' ||
+							value.operationDetailsWidth > value.operationTableWidth ||
+							value.operationReasonWidth > value.operationTableWidth ||
 							!('operationText' in value) ||
 							typeof value.operationText !== 'string' ||
-							!value.operationText.includes('Transaction confirmation or delivery tracking failed. Review transaction activity while automatic retry remains active.') ||
+							!value.operationText.includes(expectedRelayOperatorFailure) ||
+							!('transactionTargetWidth' in value) ||
+							typeof value.transactionTargetWidth !== 'number' ||
+							value.transactionTargetWidth > 480 ||
+							!('transactionTableWidth' in value) ||
+							typeof value.transactionTableWidth !== 'number' ||
+							value.transactionTargetWidth > value.transactionTableWidth ||
 							!('transactionText' in value) ||
 							typeof value.transactionText !== 'string' ||
-							!value.transactionText.includes('Transaction confirmation or delivery tracking failed. Review transaction activity while automatic retry remains active.'))
+							!value.transactionText.includes(expectedRelayOperatorFailure))
 					)
 						throw new Error('Error state did not expose its attention and recovery context')
 					if (mobile && 'bodyScrollWidth' in value && typeof value.bodyScrollWidth === 'number' && value.bodyScrollWidth > width) throw new Error(`${status} header overflows its ${width.toString()}px viewport`)
@@ -851,6 +883,10 @@ async function captureScreenshots(chromium: string, origin: string, outputDirect
 					initialFailure.mode !== 'Mode unavailable' ||
 					!('network' in initialFailure) ||
 					initialFailure.network !== 'Network unavailable' ||
+					!('noticeCopy' in initialFailure) ||
+					initialFailure.noticeCopy !== expectedStateUnavailableFailure ||
+					!('noticeTitle' in initialFailure) ||
+					initialFailure.noticeTitle !== 'Dashboard disconnected' ||
 					!('pauseDisabled' in initialFailure) ||
 					initialFailure.pauseDisabled !== true ||
 					!('runStatus' in initialFailure) ||
@@ -882,7 +918,9 @@ async function captureScreenshots(chromium: string, origin: string, outputDirect
 					!('network' in postSuccessFailure) ||
 					postSuccessFailure.network !== 'mainnet · 1 · last known' ||
 					!('noticeCopy' in postSuccessFailure) ||
-					postSuccessFailure.noticeCopy !== 'State polling failed. Automatic retry remains active; use Refresh to retry now.' ||
+					postSuccessFailure.noticeCopy !== expectedStateUnavailableFailure ||
+					!('noticeTitle' in postSuccessFailure) ||
+					postSuccessFailure.noticeTitle !== 'Dashboard disconnected' ||
 					!('pauseDisabled' in postSuccessFailure) ||
 					postSuccessFailure.pauseDisabled !== false ||
 					!('runStatus' in postSuccessFailure) ||
@@ -1485,8 +1523,6 @@ if (
 
 function currentFixtureSnapshot(): OperatorSnapshot {
 	const fixturePositions = fixtureAttention === 'recovery' ? snapshot.positions.map((position, index) => (index === 0 ? { ...position, status: 'recovery-required' as const } : position)) : snapshot.positions
-	const rawRpcFailure = `Read RPC https://operator:${protectedFailureMarker}@rpc.example failed at block 23842152`
-	const rawRelayFailure = `Private relay https://operator:${protectedFailureMarker}@relay.example rejected the transaction`
 	const fixtureTransactions = snapshot.transactionActivity.map((transaction, index) => {
 		if (index !== 0) return transaction
 		if (fixtureAttention === 'transaction') return { ...transaction, status: 'confirmation-unknown' as const }

bots/open-oracle-arbitrager/scripts/check-docs.mts

@@ -51,6 +51,8 @@ assert.match(guide, /Uniswap V3 remains the\s+reference and TWAP anchor/)
 assert.match(readme, /Uniswap V2, V3, or hookless V4/)
 assert.match(readme, /### Executor public surface/)
 assert.match(readme, /`dispute` is a lower-level, unhedged funding helper/)
+assert.match(readme, /Legacy\s+journals without a persisted dispute index are the bounded exception/)
+assert.match(readme, /retain log\s+history back to the oldest open legacy position's entry block/)
 
 await assertLocalLinksResolve(guidePath, guide)
 const fixturePath = path.join(projectRoot, 'docs', 'market-fixture.html')

bots/open-oracle-arbitrager/src/contracts/abi.ts

@@ -190,6 +190,7 @@ export const erc20Abi = [
 
 export const openOraclePriceCoordinatorAbi = [
 	{ type: 'function', name: 'openOracle', stateMutability: 'view', inputs: [], outputs: [{ name: '', type: 'address' }] },
+	{ type: 'function', name: 'pendingReportId', stateMutability: 'view', inputs: [], outputs: [{ name: '', type: 'uint256' }] },
 	{ type: 'function', name: 'reputationToken', stateMutability: 'view', inputs: [], outputs: [{ name: '', type: 'address' }] },
 	{ type: 'function', name: 'weth', stateMutability: 'view', inputs: [], outputs: [{ name: '', type: 'address' }] },
 	{ type: 'function', name: 'settlementTime', stateMutability: 'view', inputs: [], outputs: [{ name: '', type: 'uint48' }] },
@@ -253,6 +254,21 @@ const timingComponents = [
 
 export const openOracleAbi = [
 	{ type: 'function', name: 'oracleGame', stateMutability: 'view', inputs: [{ name: '', type: 'uint256' }], outputs: [{ name: '', type: 'bytes32' }] },
+	{
+		type: 'function',
+		name: 'disputeHistory',
+		stateMutability: 'view',
+		inputs: [
+			{ name: '', type: 'uint256' },
+			{ name: '', type: 'uint256' },
+		],
+		outputs: [
+			{ name: 'amount1', type: 'uint128' },
+			{ name: 'amount2', type: 'uint128' },
+			{ name: 'baseFee', type: 'uint128' },
+			{ name: 'reportTimestamp', type: 'uint48' },
+		],
+	},
 	{ type: 'function', name: 'storedGame', stateMutability: 'view', inputs: [{ name: '', type: 'uint256' }], outputs: gameComponents },
 	{
 		type: 'function',