feat: Remote Seal - #971
Draft
magik6k wants to merge 80 commits into
Draft
Conversation
Resolve conflicts in task_treed.go and task_treerc.go CanAccept methods: adopt main's QueryRow/array_agg/storage_path performance improvement while preserving UNION ALL with rseal_provider_pipeline for remote seal two-path awareness.
Root cause: local swag binary was v1.16.6 while CI installs v1.16.4, producing different swagger output for AllocationId format field. The marketgen target now runs 'go install swag@v1.16.4' before generating to ensure consistency regardless of local environment.
…ndition The ConstructCurioTest function was registering storage paths via the RPC API (StorageInit/StorageAddLocal) AFTER tasks.StartTasks() had already begun. This created a race where the task engine could pick up sectors before any storage path was known, causing 'storage claim failed' errors for SyntheticProofs. Once a claim fails, STORAGE_FAILURE_TIMEOUT (3 min) blocks retries, leading to the 10-min test timeout. Move storage registration (sectorstore.json + OpenPath + SetStorage) to before StartTasks, using the deps directly instead of the RPC API.
rseal_provider_pipeline does not have task_id_synth or after_synth columns (the provider only does SDR+trees, not synth proofs). The UNION ALL queries referencing these non-existent columns caused SQL errors in taskToSector(), which is called during TaskStorage.Claim(), resulting in 'storage claim failed' for every SyntheticProofs task.
Three issues prevented TestRemoteSealHappyPath from working: 1. HTTP servers were never started (cfg.HTTP.Enable was not set). Fix: enable HTTP with DelegateTLS=true and ListenAddress=127.0.0.1:0 on both provider and client instances. 2. attachRouters panicked on must.One(d.EthClient.Get()) when DealMarket was not enabled. Fix: gate deal market routers (retrieval, IPNI, libp2p, market handler) behind cfg.Subsystems.EnableDealMarket. 3. partner_url and provider_url pointed to wrong addresses (localhost:0 and RPC port respectively). Fix: use actual HTTPListenAddr discovered after binding the listener. Supporting changes: - HTTP server now binds a net.Listener before starting the goroutine, making the actual listen address available via deps.HTTPListenAddr (supports port 0 for tests). - ConstructCurioTest sets dependencies.Cfg before PopulateRemainingDeps so HTTP config overrides take effect, and returns *deps.Deps so the caller can read HTTPListenAddr.
All provider-side and client-side endpoints validated token existence but did not verify the authenticated partner/provider owns the requested sector. Add partner_id/provider_id constraints to every SQL query in handleStatus, handleSealedData, handleCacheData, handleCommit1, handleFinalize, handleCleanup, handleTicket, and handleComplete.
- Remove RSealClientC1Exchange task entirely; C1 output is now fetched on-demand by Local.GeneratePoRepVanillaProof via a c1.url file written during the fetch stage, avoiding 50MiB JSON storage in PostgreSQL - Refactor RSealDelegate to move HTTP calls (CheckAvailable, SendOrder) from schedule() into Do(), keeping the IAmBored scheduling path fast with only DB operations - Refactor RSealClientPoll.Do() to loop internally with 30s poll interval instead of returning false/nil and cycling through the full task scheduling machinery. Uses CanYield for graceful shutdown support. - Remove PoRepSnarkWithVanilla codepath; unified PoRepSnark handles both local and remote-sealed sectors transparently - Bump commit-phase1-output size constraint from 20MB to 128MB
The SDR ticket is derived from public chain randomness, so any chain participant can compute it. Remove the entire RSealProviderTicket task and /ticket API endpoint. The provider's SDR task now computes the ticket directly from its own chain node (which it already did, but previously a redundant pre-fetch was required as a scheduling gate). The ticket now flows from provider to client via the /complete notification and /status response, eliminating the need for ticket_epoch/ticket_value columns in rseal_client_pipeline. Changes: - Delete RSealProviderTicket task entirely - Remove pollerProvTicketFetch from provider poller enum - Provider poller goes directly to SDR (no ticket gate) - Add ticket fields to CompleteNotification and StatusResponse - RSealProviderNotify includes ticket in completion callback - Move applyRemoteCompletion to sealmarket.ApplyRemoteCompletion (shared between poll task and /complete handler) - Remove TicketAPI interface and api field from SealMarket - Remove /ticket route and handleTicket handler - Remove ticket columns from rseal_client_pipeline schema
…ipeline GC Add BEFORE DELETE triggers on sectors_sdr_pipeline and rseal_provider_pipeline that cascade-delete matching batch_sector_refs rows based on pipeline_source. This replaces the FK that was dropped to support remote sectors in batch refs. Also add cleanupRemoteSealProvider to PipelineGC to delete rseal_provider_pipeline rows that have completed cleanup and been idle for 24+ hours, preventing indefinite accumulation.
The commit-phase1-output (C1 vanilla proof) is ~50-128 MiB of binary data. JSON-encoding it base64-encodes the []byte field, adding ~33% size overhead for no benefit. Switch to application/octet-stream for the /commit1 response and io.ReadAll on the client side.
Three documentation files: - remote-seal.md: Overview of the architecture, setup flow, and end-to-end sealing flow - remote-seal-provider.md: Provider setup, configuration, pipeline states, API endpoints, storage considerations - remote-seal-client.md: Client setup, configuration, task lifecycle, C1 mechanism, failure handling, network requirements Plus a graphviz dot diagram rendered to SVG showing the full task dependency graph across both client and provider nodes, including HTTP calls between them.
…al CI test The RSealProviderPoller's promise slots for SDR, TreeD, and TreeRC were never populated with an AddTaskFunc, causing rseal_provider_pipeline rows to be permanently stuck at after_sdr=false. The provider poller's pollStartSDR/TreeD/TreeRC methods silently returned because IsSet() was always false. Fix by: - Adding exported SetPollerSDR/TreeD/TreeRC methods on RSealProviderPoller - Adding ProviderPollerSDR/TreeD/TreeRC interfaces in the seal package - Passing the provider poller to SDR/TreeD/TreeRC task constructors so their Adder() methods register with both SealPoller and the provider poller - Creating the provider poller before task construction in tasks.go - Using typed nil interface values to avoid Go nil-interface pitfall Also adds test-itest-remoteseal to the CI workflow matrix, which was the reason this bug was never caught (remoteseal_test.go was never executed).
…ypes The SealPoller would schedule PoRep for remote sectors as soon as precommit succeeded, without waiting for RSealClientFetch to download the sealed data and cache from the provider. PoRep would then fail at runtime trying to AcquireSector (files not on disk), burning retry attempts. Fix by selecting COALESCE(c.after_fetch, TRUE) from the rseal_client_pipeline LEFT JOIN (TRUE for local sectors, actual value for remote) and requiring AfterFetch in pollStartPoRep. Commit msg, finalize, and move storage are transitively blocked via after_porep. PreCommit msg is correctly not gated since it only needs tree CIDs from ApplyRemoteCompletion. Also removes unused availableProvider and candidateSector types from task_client_delegate.go (caught by golangci-lint).
Add TestMain to itests/ that starts a YugabyteDB container via testcontainers-go when CURIO_HARMONYDB_HOSTS is not set, enabling zero-setup local test runs. CI is unaffected since it pre-sets the env var.
…r remote seal client The provider's C1 handler (handleCommit1) calls SealCommitPhase1 which requires syn-porep-vanilla-proofs.dat. The provider runs SDR+trees but skips the normal Synth task (which clears layers). Add EnsureSyntheticProofs to generate synthetic proofs without clearing cache, and call it before C1. Enable the Finalize task when EnableRemoteSealClient is set - the client skips SDR/Trees but still needs Finalize after PoRep. Fix RSealProvFinalize task name to RSealProvFinal (16-char limit).
Fix ListenAndServe error in ConstructCurioTest: when running multiple Curio instances they share a hardcoded listen port, causing EADDRINUSE. Log the error instead of failing the test since the RPC server is not needed for the remote seal test pipeline. Strip all YSQL_*/YCQL_* env vars from testcontainers YugabyteDB config so it starts with trust auth (matching CI). Simplify remote seal test cleanup to use defers.
…leanup timeout Sealed file download now uses aria2c for multi-connection resumable downloads (16 connections, 100 retries, --continue for resume), with a Go HTTP fallback that supports Range headers. Same pattern as lib/fastparamfetch. Add RemoteSealProviderMaxTasks config to limit concurrent provider-side remote seal tasks (Notify/Finalize/Cleanup). Default 0 (unlimited). Add RemoteSealCleanupTimeout config (default 72h) replacing the hardcoded interval in the provider notify task. Controls how long the provider keeps sealed data before auto-cleanup if the client doesn't respond.
aria2 is needed at runtime for resumable multi-connection sealed file downloads. time (GNU /usr/bin/time) is needed by make gen for timing go generate runs.
The test was stuck at SDR because no subsystem flags were set in the config. Enable EnableSealSDR, EnableSealSDRTrees, EnableSendPrecommitMsg, EnablePoRepProof, EnableSendCommitMsg, EnableMoveStorage, and UseSyntheticPoRep so the full sealing pipeline runs.
…byteDB Replace manually-managed Docker containers in CI with testcontainers-go so that every test package self-provisions its own YugabyteDB instance. This fixes the pre-existing test-all CI failure where lib/paths and market/indexstore tests couldn't connect to a database. Key changes: - Add shared dbtest.StartYugabyte helper with dynamic port allocation - Add TestMain to lib/paths and market/indexstore - Refactor itests/testmain_test.go to use the shared helper - Make harmonydb test port configurable via CURIO_HARMONYDB_PORT env var - Make CQL port configurable via CURIO_HARMONYDB_CQL_PORT env var - Remove Docker container lifecycle steps from CI workflow
When specifying individual .go files to 'go test', TestMain from testmain_test.go is not included in the compilation. Switch all itest matrix entries to use '-run <pattern> ./itests/' so the full package (including TestMain with testcontainers setup) is compiled and run.
…ablets
Two optimizations to reduce YugabyteDB schema migration overhead in tests:
1. Reduce tablet count to 1 per tserver (--yb_num_shards_per_tserver=1,
--ysql_num_shards_per_tserver=1) so each DDL creates just one tablet
instead of the default based on CPU count.
2. Create a colocated database ('curio_test') after container startup.
Colocated databases store all tables in a single shared tablet,
eliminating per-table tablet creation overhead entirely. Tests connect
to this database via the CURIO_HARMONYDB_DB environment variable.
harmonydb.NewFromConfigWithITestID now reads the database name from
CURIO_HARMONYDB_DB (defaulting to 'yugabyte') so tests automatically
use the colocated database when started via testcontainers.
TestLocalStorage: 84s -> 64s (~24% faster)
…l batch The SupraSeal path generated a fresh ticket for remote sectors but never persisted ticket_epoch and ticket_value back to rseal_provider_pipeline, leaving them NULL. This caused RSealProvNotify to fail when scanning the NULL ticket_epoch into a non-pointer int64.
This reverts commit f31f074.
Use temporary files/directories with .tmp suffix for downloading sealed sector files and cache data. Only rename to final destination after successful download. This ensures partial/incomplete data is never exposed under the final sector path name. Also add proper cleanup on error - temp files and aria2 control files are removed if any error occurs during download, preventing lingering incomplete data from failed attempts. Changes: - fetchWithAria2c: downloads to .tmp, renames on success, cleans up on error - fetchWithGoHTTP: downloads to .tmp with resume support, renames on success - fetchCacheTar: extracts to .tmp dir, renames on success, cleans up on error
aria2c may leave a pre-allocated sparse file on failure that stats as full size but contains null-byte holes for unfetched chunks. Falling back to Go HTTP with such a file causes the resume logic to treat it as complete (416 Range Not Satisfiable), silently producing corrupted sector data. Fix: if aria2c is installed, use it exclusively. On failure, the task retries with aria2c from scratch (temp files are cleaned up). Go HTTP is only used when aria2c is not in PATH at all. Also add --file-allocation=none to aria2c to prevent sparse pre-allocation as defense-in-depth.
- Lower --lowest-speed-limit from 16K to 4K to tolerate brief dips - Increase --timeout to 120s for stall tolerance on 32 GiB files - Set --max-tries=0 (infinite) with --retry-wait=30 for persistent retry - Add --auto-file-renaming=false and --allow-overwrite=true to prevent duplicate file copies on retry
Resolve conflicts across CI, HTTP server, config defaults, tests, docs, and modules. Combine remote-seal with main: deal-market-gated routes with denylist retrieval, PDP and sealmarket wiring, cuzk PoRep path with remote seal comment, IPNI StartPublishing from main, helpers YBCQLPort for index store. CI uses main runner matrix plus remoteseal job. Accept main removal of market_deal_dynamic_test. Upgrade testcontainers-go to v0.42 with moby/api container types for dbtest. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
@magik6k this seems to need a test correction for its own test. can you look at that? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Verified to work mostly