Fix CI flakiness: async race conditions, disposal crash, and stdout truncation#9939
Open
Fix CI flakiness: async race conditions, disposal crash, and stdout truncation#9939
Conversation
…runcation
Root Cause Analysis of ~63% CI failure rate on main branch:
1. Parallel.For async void anti-pattern (TemplatePackageManager.cs)
- Parallel.For(0, N, async (i) => ...) converts lambda to async void
- Parallel.For returns after synchronous portion, before awaits complete
- scanResults[] left partially null, causing OrderOfScanningIsCorrect failures
- Fix: Replace with await Task.WhenAll(Enumerable.Range(...).Select(async ...))
2. GlobalSettings.FileChanged disposal race condition (GlobalSettings.cs)
- FileChanged is async void callback from file watcher
- Original Dispose() called _watcher.Dispose() before setting _disposed = true
- Callback could fire between these, hitting ObjectDisposedException
- Unhandled exception in async void crashes entire xunit test runner process
- Fix: Set _disposed = true before watcher disposal, add disposed checks
and try-catch in FileChanged callback
3. Console output truncation in CLI commands (ExecutableCommand.cs)
- SimpleConsoleLogger uses background queue thread for output
- LoggerFactory.Dispose() drains queue but Console.Out pipe buffer not flushed
- Process exit before flush causes mid-line truncation in integration tests
- Fix: Explicit Console.Out.Flush()/Console.Error.Flush() after disposal
Validation: 10 consecutive CI passes (builds 1324248-1324572) on this branch,
compared to ~63% failure rate (20/32 failures) on main branch.
Co-authored-by: Copilot <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes ~63% CI failure rate on the main branch caused by three independent race conditions.
Root Causes Identified
1.
Parallel.Forasync void anti-pattern (TemplatePackageManager.cs)Parallel.For(0, N, async (i) => ...)converts the lambda toasync void.Parallel.Forreturns immediately after the synchronous portion of each iteration completes (up to the firstawait), not after the async work finishes. This leavesscanResults[]partially populated with nulls, causingOrderOfScanningIsCorrecttest failures (expectedsample99, got random values likesample69).Fix: Replace with
await Task.WhenAll(Enumerable.Range(...).Select(async ...))2.
GlobalSettings.FileChangeddisposal race condition (GlobalSettings.cs)FileChangedis anasync voidcallback from a file watcher. The originalDispose()called_watcher.Dispose()before setting_disposed = true. The callback could fire between these two operations, hittingObjectDisposedExceptionon theSettingsChangedevent handler. Since the exception is unhandled inasync void, it crashes the entire xunit test runner process ΓÇö killing ALL Edge.UnitTests, not just the file watcher test.Fix: Set
_disposed = truebefore watcher disposal, add disposed check andObjectDisposedExceptioncatch inFileChanged3. Console output truncation in CLI commands (ExecutableCommand.cs)
SimpleConsoleLoggeruses a background queue thread for console output.LoggerFactory.Dispose()drains the queue but theConsole.Outpipe buffer may not be flushed before process exit. This causes mid-line output truncation inValidateCommand_BasicTestintegration tests.Fix: Explicit
Console.Out.Flush()/Console.Error.Flush()after logger factory disposalValidation
10 consecutive CI passes on this branch (builds 1324248-1324572), compared to ~63% failure rate (20/32 failures) on the main branch over the same period.
Files Changed
src/Microsoft.TemplateEngine.Edge/Settings/TemplatePackageManager.csΓÇö Parallel.For to Task.WhenAllsrc/Microsoft.TemplateEngine.Edge/BuiltInManagedProvider/GlobalSettings.csΓÇö Disposal ordering + FileChanged guardstest/Microsoft.TemplateEngine.Edge.UnitTests/GlobalSettingsTests.csΓÇö Robustified TestFileWatcher event handlertools/Microsoft.TemplateEngine.Authoring.CLI/Commands/ExecutableCommand.csΓÇö Console flush after logger disposal