feat: batch job pulling for native workers [ee]#8244
Draft
Conversation
Reduce DB polling overhead for native workers by batch-fetching jobs server-side and serving them from an in-memory buffer via HTTP. - Add batch_pull() in windmill-queue: single SELECT...FOR UPDATE SKIP LOCKED LIMIT N - Add batch pull SQL helpers (make_batch_pull_query, format_batch_pull_query) - OSS stubs for agent-workers accept batch_buffer parameter (4-tuple return) - Native workers self-sign JWT and pull jobs via HTTP when co-located with server - Add uses_batch_http_pull column to worker_ping for server-side tracking - Worker pull loop: HTTP batch pull when client available, SQL otherwise Co-Authored-By: Claude Opus 4.6 <[email protected]>
Deploying windmill with
|
| Latest commit: |
e033c73
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://b75f9115.windmill.pages.dev |
| Branch Preview URL: | https://batch-pulling.windmill.pages.dev |
Native workers in Mode::Worker (no co-located server) can now use HTTP batch pull when BASE_INTERNAL_URL is explicitly set pointing to the remote server. The batch buffer itself only runs on the server side. Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Replace BASE_INTERNAL_URL overloading with dedicated BATCH_PULL_URL env var for native workers' HTTP pull endpoint - Add exp claim to JWT token (required by jsonwebtoken validation) - Token expires in 30 days, renewed on worker restart Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Extract bunnative→nativets tag logic into ScriptLang::as_worker_tag() - Add benchmark results for batch pull vs direct SQL (1W and 3W) - Add throughput model script comparing batch vs SQL at scale - Add nativets_sleep benchmark script support Co-Authored-By: Claude Opus 4.6 <[email protected]>
hugocasa
changed the title
Co-Authored-By: Claude Opus 4.6 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Batch job pulling for native workers — server batch-fetches N jobs in one
SELECT...FOR UPDATE SKIP LOCKED LIMIT Nquery instead of N individual queries from N subworkers.worker_ping)pull_jobHTTP endpoint (no DB hit per pull)BATCH_PULL_URLenv var (auto-detected in standalone mode)expclaim for native worker authScriptLang::as_worker_tag()helper to centralize bunnative→nativets tag logicBenchmark Results (3 native workers = 24 subworkers)
nativets — 10,000 fast jobs:
nativets — 1,000 fast jobs:
nativets_sleep (500ms avg) — 1,000 jobs:
Projected scaling (from model calibrated on real data):
SQL throughput plateaus around 15-20 native workers due to O(N²) SKIP LOCKED contention. Batch keeps scaling linearly.
Test plan
BATCH_PULL_URL🤖 Generated with Claude Code