Skip to content

WIP fine-grained task concurrency#6955

Draft
lihaoyi wants to merge 11 commits intocom-lihaoyi:mainfrom
lihaoyi:concurrency
Draft

WIP fine-grained task concurrency#6955
lihaoyi wants to merge 11 commits intocom-lihaoyi:mainfrom
lihaoyi:concurrency

Conversation

@lihaoyi
Copy link
Copy Markdown
Member

@lihaoyi lihaoyi commented Mar 23, 2026

This code is intended to remove the global out/ folder lock in favor of finer grained per-task or per-bootstrap-phase read/write locks, with handling for the various "global" out/ folder files (e.g. mill-chrome-profile.json,
mill-profile.json, mill-console-tail, etc.) to preserve their usefulness in the presence of concurrent runs.

  • During task execution, we take a read lock for each task, if it needs to be re-evaluated we then take a write lock, and when evaluation is done we keep the read lock to ensure it remains stable for use by downstream tasks until execution completes

  • When evaluating meta-builds, we take a write lock for any meta build that is being actively evaluated, and convert to a read lock after it is done so it remains stable while we use it

  • The various out/mill-* files are turned into symlinks to out/mill-run-*/* files: the out/mill-* file pointing at the most recent out/mill-run-*/* file to be created, but the older out/mill-run-*/* files kept around in case people need them (e.g. due to concurrent use)

  • RunnerState handling is updated to support concurrent runs, with the same workerCache shared across all runs

  • This change is meant to only affect daemon mode, since we don't have reliable filesystem locking we can rely on for non-daemon mode (e.g. docker folder mounts don't always support locks properly). mill-out-lock is turned into a process-level lock, such that either a single MillDaemonMain process or MillNoDaemonMain process can claim it. If a MillDaemonMain claims it then it can support multiple concurrent runs with fine-grained locking internally, but if a MillNoDaemonMain claims it then only that one run can take effect

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant