feat(ContextPropagator): Replace imperative set/unset context propagation with lexically-scoped runWithContext (v3.13.x backport) - #1075
Conversation
…tion Backport of the equivalent fix on the master PR (cadence-workflow#1074) to this v3.13.x backport branch (cadence-workflow#1075). Why: Review feedback on cadence-workflow#1074 (github.com/cadence-workflow/pull/1074#pullrequestreview-4929567768) pointed out a real regression risk in the runWithContext redesign: because a ContextPropagator's runWithContext(context, task) now directly wraps the call to task.run(), a buggy implementation can structurally catch and suppress (or replace) whatever the task throws -- including workflow cancellation and thread-destruction signals from WorkflowThread. The old imperative setCurrentContext/unsetCurrentContext shape made this impossible, since propagators never touched the call site of the wrapped task. Change: - ContextThreadLocal.runWithContext now wraps the innermost task in a canary that records any Throwable it throws before rethrowing it. After the composed propagator chain returns, if the canary recorded a Throwable but none escaped the chain, some propagator in it swallowed the exception -- in which case a new ContextPropagatorSwallowedExceptionError is thrown (listing the propagators applied to that call) instead of silently continuing as if the task had succeeded. - ContextPropagatorSwallowedExceptionError extends Error, matching this codebase's existing convention for framework-level, must-not-be-retried conditions (NonDeterminisicWorkflowError, DestroyWorkflowThreadError, WorkflowRejectedExecutionError). This also means it composes correctly with existing handling: WorkflowThreadImpl already special-cases `catch (Error e)` to abort the decision rather than fail the workflow, and PollTaskExecutor routes any Throwable escaping an activity handler to the configured uncaughtExceptionHandler instead of a normal activity failure response. - Strengthened the ContextPropagator#runWithContext javadoc to state the contract explicitly: implementations must only wrap task.run() in try/finally, never try/catch. - Added a test with a deliberately misbehaving propagator that catches and suppresses the task's exception, verifying the new error is thrown with the original exception as its cause. This does not catch every misbehavior -- a propagator that catches and replaces the exception with a different one can't be reliably distinguished from a propagator intentionally translating exception types, so that half of the original concern is addressed by the javadoc contract instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ong number of times Backport of the equivalent fix on the master PR (cadence-workflow#1074) to this v3.13.x backport branch (cadence-workflow#1075). Why: Follow-up to the swallow-detection fix, addressing github.com/cadence-workflow/pull/1074#discussion_r3778176907 (gitar-bot): the swallow-detection canary's `thrown` AtomicReference was set on the first task failure and never cleared, so a propagator that catches the task's exception and then retries it, eventually succeeding, would incorrectly be flagged as having swallowed the earlier failure. Rather than making the canary tolerate retries, retrying task is itself a contract violation -- ContextPropagator#runWithContext must call task.run() exactly once. So instead this adds a second, independent check: an invocation counter that must equal exactly one. It covers both a propagator that skips calling task entirely and one that calls it more than once (including the retry-then-succeed case), and it is checked before the existing swallow check, so a retry-then-succeed propagator is now correctly reported as an invocation-count violation rather than a misleading "swallowed exception" one. Change: - ContextThreadLocal.runWithContext tracks an AtomicInteger invocationCount alongside the existing AtomicReference<Throwable> thrown, incrementing it in the same canary that wraps task.run(). After the composed propagator chain returns normally, invocationCount != 1 is checked first (covers zero and multiple invocations), then the existing swallowed-exception check. - Renamed ContextPropagatorSwallowedExceptionError to ContextPropagatorContractViolationError (extends Error, same as before) since it now models two distinct contract violations via named static factory methods (swallowedException, unexpectedInvocationCount) instead of a single constructor. - Updated ContextPropagator#runWithContext's javadoc contract to state task.run() must be called exactly once -- never skipped, never retried. - Added a test with a propagator that retries task after catching its exception and eventually succeeds, verifying it is reported as an invocation-count violation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ion-count violation Backport of the equivalent fix on the master PR (cadence-workflow#1074) to this v3.13.x backport branch (cadence-workflow#1075). Why: Follow-up to the invocation-count check, addressing github.com/cadence-workflow/pull/1074#discussion_r3778369888 (gitar-bot): when a propagator retries the task after an exception (invocationCount != 1), ContextPropagatorContractViolationError was thrown with a null cause, even though `thrown` already held the exception from the attempt that triggered the retry. Attaching it makes it much easier to debug a misbehaving propagator, since the resulting stack trace shows both the contract violation and the original failure that provoked it. Change: - ContextPropagatorContractViolationError.unexpectedInvocationCount now takes the last captured Throwable and passes it through as the cause, instead of always passing null. - ContextThreadLocal.runWithContext passes thrown.get() into that call. - Updated the retry test to assert the original exception is preserved as the cause. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…h lexically-scoped runWithContext Why: JDK 25 finalizes ScopedValue (JEP 481), a lexically-scoped, immutable alternative to ThreadLocal for propagating context: a value is bound and the task runs inside that binding in a single call (`ScopedValue.where(...).call(task)`), with no separate unset step -- the binding is torn down automatically when the call returns. The previous ContextPropagator API couldn't express that: context was installed via `setCurrentContext`/`deserializeContext` at the top of a handler (e.g. ActivityWorker.propagateContext, LocalActivityWorker.restoreContext, WorkflowThreadImpl's propagateContextToCurrentThread) and torn down via `unsetCurrentContext` in a `finally` block possibly several methods away, which doesn't map onto ScopedValue's single-call binding model. Restructuring around a single `runWithContext(context, task)` call gets us: - Simplicity: a ScopedValue-backed propagator only needs to override `runWithContext` and call `ScopedValue.where(VALUE, context).run(task)` -- no separate set/get/unset methods to keep consistent with each other. - Immutable context data: ScopedValue bindings are write-once for the dynamic extent of the call; nothing downstream can accidentally mutate a shared context object the way a mutable ThreadLocal value can be. - No unpaired unset: because the binding and the task execution happen in the same call frame, cleanup is guaranteed by stack unwinding. The old shape's risk -- an early return, a thrown exception before the try block, or a future refactor splitting set/unset further apart leaving a propagator's context set with no matching unset and leaking it onto whatever task a pooled thread picks up next -- goes away entirely. Change: - Add `ContextPropagator.runWithContext(Object context, ContextRunnable task)`, a default method that wraps `setCurrentContext`/`task.run()`/`unsetCurrentContext` in a single try/finally, so set and unset are always paired in one call frame. - `ContextThreadLocal.runWithContext(...)` composes all configured propagators into a single nested call chain and executes the task inside it, replacing the old `propagateContextToCurrentThread` + manual `unsetCurrentContext` pair. - Update the three call sites (ActivityWorker, LocalActivityWorker, WorkflowThreadImpl) to run their task through this single call instead of bracketing it with separate propagate/unset calls. Existing propagators that implement `setCurrentContext`/`unsetCurrentContext` are unaffected; `runWithContext` has a default implementation built on those methods, so this is additive for current implementations and opt-in for ScopedValue-based ones. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Haofeng Lei <haofeng.lei@uber.com>
…tion Backport of the equivalent fix on the master PR (cadence-workflow#1074) to this v3.13.x backport branch (cadence-workflow#1075). Why: Review feedback on cadence-workflow#1074 (github.com/cadence-workflow/pull/1074#pullrequestreview-4929567768) pointed out a real regression risk in the runWithContext redesign: because a ContextPropagator's runWithContext(context, task) now directly wraps the call to task.run(), a buggy implementation can structurally catch and suppress (or replace) whatever the task throws -- including workflow cancellation and thread-destruction signals from WorkflowThread. The old imperative setCurrentContext/unsetCurrentContext shape made this impossible, since propagators never touched the call site of the wrapped task. Change: - ContextThreadLocal.runWithContext now wraps the innermost task in a canary that records any Throwable it throws before rethrowing it. After the composed propagator chain returns, if the canary recorded a Throwable but none escaped the chain, some propagator in it swallowed the exception -- in which case a new ContextPropagatorSwallowedExceptionError is thrown (listing the propagators applied to that call) instead of silently continuing as if the task had succeeded. - ContextPropagatorSwallowedExceptionError extends Error, matching this codebase's existing convention for framework-level, must-not-be-retried conditions (NonDeterminisicWorkflowError, DestroyWorkflowThreadError, WorkflowRejectedExecutionError). This also means it composes correctly with existing handling: WorkflowThreadImpl already special-cases `catch (Error e)` to abort the decision rather than fail the workflow, and PollTaskExecutor routes any Throwable escaping an activity handler to the configured uncaughtExceptionHandler instead of a normal activity failure response. - Strengthened the ContextPropagator#runWithContext javadoc to state the contract explicitly: implementations must only wrap task.run() in try/finally, never try/catch. - Added a test with a deliberately misbehaving propagator that catches and suppresses the task's exception, verifying the new error is thrown with the original exception as its cause. This does not catch every misbehavior -- a propagator that catches and replaces the exception with a different one can't be reliably distinguished from a propagator intentionally translating exception types, so that half of the original concern is addressed by the javadoc contract instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Haofeng Lei <haofeng.lei@uber.com>
…ong number of times Backport of the equivalent fix on the master PR (cadence-workflow#1074) to this v3.13.x backport branch (cadence-workflow#1075). Why: Follow-up to the swallow-detection fix, addressing github.com/cadence-workflow/pull/1074#discussion_r3778176907 (gitar-bot): the swallow-detection canary's `thrown` AtomicReference was set on the first task failure and never cleared, so a propagator that catches the task's exception and then retries it, eventually succeeding, would incorrectly be flagged as having swallowed the earlier failure. Rather than making the canary tolerate retries, retrying task is itself a contract violation -- ContextPropagator#runWithContext must call task.run() exactly once. So instead this adds a second, independent check: an invocation counter that must equal exactly one. It covers both a propagator that skips calling task entirely and one that calls it more than once (including the retry-then-succeed case), and it is checked before the existing swallow check, so a retry-then-succeed propagator is now correctly reported as an invocation-count violation rather than a misleading "swallowed exception" one. Change: - ContextThreadLocal.runWithContext tracks an AtomicInteger invocationCount alongside the existing AtomicReference<Throwable> thrown, incrementing it in the same canary that wraps task.run(). After the composed propagator chain returns normally, invocationCount != 1 is checked first (covers zero and multiple invocations), then the existing swallowed-exception check. - Renamed ContextPropagatorSwallowedExceptionError to ContextPropagatorContractViolationError (extends Error, same as before) since it now models two distinct contract violations via named static factory methods (swallowedException, unexpectedInvocationCount) instead of a single constructor. - Updated ContextPropagator#runWithContext's javadoc contract to state task.run() must be called exactly once -- never skipped, never retried. - Added a test with a propagator that retries task after catching its exception and eventually succeeds, verifying it is reported as an invocation-count violation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Haofeng Lei <haofeng.lei@uber.com>
…ion-count violation Backport of the equivalent fix on the master PR (cadence-workflow#1074) to this v3.13.x backport branch (cadence-workflow#1075). Why: Follow-up to the invocation-count check, addressing github.com/cadence-workflow/pull/1074#discussion_r3778369888 (gitar-bot): when a propagator retries the task after an exception (invocationCount != 1), ContextPropagatorContractViolationError was thrown with a null cause, even though `thrown` already held the exception from the attempt that triggered the retry. Attaching it makes it much easier to debug a misbehaving propagator, since the resulting stack trace shows both the contract violation and the original failure that provoked it. Change: - ContextPropagatorContractViolationError.unexpectedInvocationCount now takes the last captured Throwable and passes it through as the cause, instead of always passing null. - ContextThreadLocal.runWithContext passes thrown.get() into that call. - Updated the retry test to assert the original exception is preserved as the cause. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Haofeng Lei <haofeng.lei@uber.com>
080d51b to
5ed4dae
Compare
…ssertion ReplayDecider#executeLocalActivities caps how many local activities land in one decision batch by comparing elapsed processing time against 80% of the decision task timeout. With the test's 5s timeout that budget is exactly 4000ms -- precisely 2x the fixed 2000ms sleepActivity duration used here -- so whether a 3rd activity also completes within the first batch is decided by a few milliseconds of scheduling jitter around that exact tie, not anything the test should be asserting on. Tolerate the batch boundary landing on either side instead of asserting a single hardcoded snapshot, while still verifying that querying mid-execution returns real, non-decreasing progress before the workflow completes. Signed-off-by: Haofeng Lei <haofeng.lei@uber.com>
… span count TChannel finishes its own per-request outbound span (in addition to the explicit "cadence-..." span WorkflowServiceTChannel creates) from a future completion callback registered on the TFuture. That callback is not guaranteed to run before the blocking RPC call returns to the caller, so the assertion on finishedSpans().size() could observe only 1 of the 2 expected spans immediately after enqueueSignalWithStart() returns, depending on which thread wins the race. Poll briefly for the expected span count before asserting instead of checking immediately. Signed-off-by: Haofeng Lei <haofeng.lei@uber.com>
CI failed: Unit test failures occurred across CI jobs due to failing test assertions in the test suite.OverviewCI runs encountered test failures in the Gradle FailuresGradle Unit Test Failures (confidence: high)
Summary
Code Review ✅ Approved 1 resolved / 1 findingsBackports lexically-scoped runWithContext propagation to replace imperative set/unset calls, addressing the invocation-count violation exception-discarding finding. No issues found. ✅ 1 resolved✅ Quality: Invocation-count violation discards the triggering exception
Tip Comment OptionsAuto-apply is off → Gitar will not commit updates to this branch. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |
Backport of #1074 to the
v3.13.xbranch, so the team can pick it up without needing a dirty cherry-pick against the divergedmasterhistory.Why
JDK 25 finalizes ScopedValue (JEP 481), a lexically-scoped, immutable alternative to ThreadLocal for propagating context: a value is bound and the task runs inside that binding in a single call (
ScopedValue.where(...).call(task)), with no separate unset step -- the binding is torn down automatically when the call returns. The previousContextPropagatorAPI couldn't express that: context was installed viasetCurrentContext/deserializeContextat the top of a handler and torn down viaunsetCurrentContextin afinallyblock possibly several methods away, which doesn't map onto ScopedValue's single-call binding model.Restructuring around a single
runWithContext(context, task)call gets us:runWithContextand callScopedValue.where(VALUE, context).run(task)-- no separate set/get/unset methods to keep consistent with each other.Change
ContextPropagator.runWithContext(Object context, ContextRunnable task), a default method that wrapssetCurrentContext/task.run()/unsetCurrentContextin a single try/finally, so set and unset are always paired in one call frame.ContextThreadLocal.runWithContext(...)composes all configured propagators into a single nested call chain and executes the task inside it, replacing the oldpropagateContextToCurrentThread+ manualunsetCurrentContextpair.ActivityWorker,LocalActivityWorker,WorkflowThreadImpl) to run their task through this single call instead of bracketing it with separate propagate/unset calls.Existing propagators that implement
setCurrentContext/unsetCurrentContextare unaffected;runWithContexthas a default implementation built on those methods, so this is additive for current implementations and opt-in for ScopedValue-based ones.Test plan
./gradlew test --tests "com.uber.cadence.internal.context.*" --tests "com.uber.cadence.internal.worker.*" --tests "com.uber.cadence.internal.sync.*"passes againstv3.13.x(Docker build, JDK 11 / thrift 0.9.3 toolchain matching this repo's CI image; version string confirms3.13.3-1-g01b4ed5).🤖 Generated with Claude Code
Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com