WIP fine-grained task concurrency#6955
Draft
lihaoyi wants to merge 11 commits intocom-lihaoyi:mainfrom
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This code is intended to remove the global out/ folder lock in favor of finer grained per-task or per-bootstrap-phase read/write locks, with handling for the various "global" out/ folder files (e.g. mill-chrome-profile.json,
mill-profile.json, mill-console-tail, etc.) to preserve their usefulness in the presence of concurrent runs.
During task execution, we take a read lock for each task, if it needs to be re-evaluated we then take a write lock, and when evaluation is done we keep the read lock to ensure it remains stable for use by downstream tasks until execution completes
When evaluating meta-builds, we take a write lock for any meta build that is being actively evaluated, and convert to a read lock after it is done so it remains stable while we use it
The various
out/mill-*files are turned into symlinks toout/mill-run-*/*files: theout/mill-*file pointing at the most recentout/mill-run-*/*file to be created, but the olderout/mill-run-*/*files kept around in case people need them (e.g. due to concurrent use)RunnerStatehandling is updated to support concurrent runs, with the sameworkerCacheshared across all runsThis change is meant to only affect daemon mode, since we don't have reliable filesystem locking we can rely on for non-daemon mode (e.g. docker folder mounts don't always support locks properly).
mill-out-lockis turned into a process-level lock, such that either a singleMillDaemonMainprocess orMillNoDaemonMainprocess can claim it. If aMillDaemonMainclaims it then it can support multiple concurrent runs with fine-grained locking internally, but if aMillNoDaemonMainclaims it then only that one run can take effect