Skip to content

feat(ContextPropagator): Replace imperative set/unset context propagation with lexically-scoped runWithContext (v3.13.x backport) - #1075

Merged
shijiesheng merged 6 commits into
cadence-workflow:v3.13.xfrom
rlei:codex/cadence-scoped-propagation-v3.13.3
Aug 14, 2026
Merged

feat(ContextPropagator): Replace imperative set/unset context propagation with lexically-scoped runWithContext (v3.13.x backport)#1075
shijiesheng merged 6 commits into
cadence-workflow:v3.13.xfrom
rlei:codex/cadence-scoped-propagation-v3.13.3

Conversation

@rlei

@rlei rlei commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Backport of #1074 to the v3.13.x branch, so the team can pick it up without needing a dirty cherry-pick against the diverged master history.

Why

JDK 25 finalizes ScopedValue (JEP 481), a lexically-scoped, immutable alternative to ThreadLocal for propagating context: a value is bound and the task runs inside that binding in a single call (ScopedValue.where(...).call(task)), with no separate unset step -- the binding is torn down automatically when the call returns. The previous ContextPropagator API couldn't express that: context was installed via setCurrentContext/deserializeContext at the top of a handler and torn down via unsetCurrentContext in a finally block possibly several methods away, which doesn't map onto ScopedValue's single-call binding model.

Restructuring around a single runWithContext(context, task) call gets us:

  • Simplicity: a ScopedValue-backed propagator only needs to override runWithContext and call ScopedValue.where(VALUE, context).run(task) -- no separate set/get/unset methods to keep consistent with each other.
  • Immutable context data: ScopedValue bindings are write-once for the dynamic extent of the call; nothing downstream can accidentally mutate a shared context object the way a mutable ThreadLocal value can.
  • No unpaired unset: because the binding and the task execution happen in the same call frame, cleanup is guaranteed by stack unwinding. The old shape's risk -- an early return, a thrown exception before the try block, or a future refactor splitting set/unset further apart leaving a propagator's context set with no matching unset and leaking it onto whatever task a pooled thread picks up next -- goes away entirely.

Change

  • Add ContextPropagator.runWithContext(Object context, ContextRunnable task), a default method that wraps setCurrentContext/task.run()/unsetCurrentContext in a single try/finally, so set and unset are always paired in one call frame.
  • ContextThreadLocal.runWithContext(...) composes all configured propagators into a single nested call chain and executes the task inside it, replacing the old propagateContextToCurrentThread + manual unsetCurrentContext pair.
  • Update the three call sites (ActivityWorker, LocalActivityWorker, WorkflowThreadImpl) to run their task through this single call instead of bracketing it with separate propagate/unset calls.

Existing propagators that implement setCurrentContext/unsetCurrentContext are unaffected; runWithContext has a default implementation built on those methods, so this is additive for current implementations and opt-in for ScopedValue-based ones.

Test plan

  • ./gradlew test --tests "com.uber.cadence.internal.context.*" --tests "com.uber.cadence.internal.worker.*" --tests "com.uber.cadence.internal.sync.*" passes against v3.13.x (Docker build, JDK 11 / thrift 0.9.3 toolchain matching this repo's CI image; version string confirms 3.13.3-1-g01b4ed5).

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com

@rlei rlei changed the title [cadence-client] Replace imperative set/unset context propagation with lexically-scoped runWithContext (v3.13.x backport) [refactor] Replace imperative set/unset context propagation with lexically-scoped runWithContext (v3.13.x backport) Aug 13, 2026
rlei added a commit to rlei/cadence-java-client that referenced this pull request Aug 13, 2026
…tion

Backport of the equivalent fix on the master PR (cadence-workflow#1074) to this v3.13.x
backport branch (cadence-workflow#1075).

Why:
Review feedback on cadence-workflow#1074 (github.com/cadence-workflow/pull/1074#pullrequestreview-4929567768)
pointed out a real regression risk in the runWithContext redesign: because
a ContextPropagator's runWithContext(context, task) now directly wraps the
call to task.run(), a buggy implementation can structurally catch and
suppress (or replace) whatever the task throws -- including workflow
cancellation and thread-destruction signals from WorkflowThread. The old
imperative setCurrentContext/unsetCurrentContext shape made this impossible,
since propagators never touched the call site of the wrapped task.

Change:
- ContextThreadLocal.runWithContext now wraps the innermost task in a canary
  that records any Throwable it throws before rethrowing it. After the
  composed propagator chain returns, if the canary recorded a Throwable but
  none escaped the chain, some propagator in it swallowed the exception --
  in which case a new ContextPropagatorSwallowedExceptionError is thrown
  (listing the propagators applied to that call) instead of silently
  continuing as if the task had succeeded.
- ContextPropagatorSwallowedExceptionError extends Error, matching this
  codebase's existing convention for framework-level, must-not-be-retried
  conditions (NonDeterminisicWorkflowError, DestroyWorkflowThreadError,
  WorkflowRejectedExecutionError). This also means it composes correctly
  with existing handling: WorkflowThreadImpl already special-cases
  `catch (Error e)` to abort the decision rather than fail the workflow,
  and PollTaskExecutor routes any Throwable escaping an activity handler
  to the configured uncaughtExceptionHandler instead of a normal activity
  failure response.
- Strengthened the ContextPropagator#runWithContext javadoc to state the
  contract explicitly: implementations must only wrap task.run() in
  try/finally, never try/catch.
- Added a test with a deliberately misbehaving propagator that catches and
  suppresses the task's exception, verifying the new error is thrown with
  the original exception as its cause.

This does not catch every misbehavior -- a propagator that catches and
replaces the exception with a different one can't be reliably distinguished
from a propagator intentionally translating exception types, so that half
of the original concern is addressed by the javadoc contract instead.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
rlei added a commit to rlei/cadence-java-client that referenced this pull request Aug 13, 2026
…ong number of times

Backport of the equivalent fix on the master PR (cadence-workflow#1074) to this v3.13.x
backport branch (cadence-workflow#1075).

Why:
Follow-up to the swallow-detection fix, addressing
github.com/cadence-workflow/pull/1074#discussion_r3778176907
(gitar-bot): the swallow-detection canary's `thrown` AtomicReference was set
on the first task failure and never cleared, so a propagator that catches
the task's exception and then retries it, eventually succeeding, would
incorrectly be flagged as having swallowed the earlier failure.

Rather than making the canary tolerate retries, retrying task is itself a
contract violation -- ContextPropagator#runWithContext must call task.run()
exactly once. So instead this adds a second, independent check: an
invocation counter that must equal exactly one. It covers both a propagator
that skips calling task entirely and one that calls it more than once
(including the retry-then-succeed case), and it is checked before the
existing swallow check, so a retry-then-succeed propagator is now correctly
reported as an invocation-count violation rather than a misleading
"swallowed exception" one.

Change:
- ContextThreadLocal.runWithContext tracks an AtomicInteger invocationCount
  alongside the existing AtomicReference<Throwable> thrown, incrementing it
  in the same canary that wraps task.run(). After the composed propagator
  chain returns normally, invocationCount != 1 is checked first (covers
  zero and multiple invocations), then the existing swallowed-exception
  check.
- Renamed ContextPropagatorSwallowedExceptionError to
  ContextPropagatorContractViolationError (extends Error, same as before)
  since it now models two distinct contract violations via named static
  factory methods (swallowedException, unexpectedInvocationCount) instead
  of a single constructor.
- Updated ContextPropagator#runWithContext's javadoc contract to state
  task.run() must be called exactly once -- never skipped, never retried.
- Added a test with a propagator that retries task after catching its
  exception and eventually succeeds, verifying it is reported as an
  invocation-count violation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
rlei added a commit to rlei/cadence-java-client that referenced this pull request Aug 13, 2026
…ion-count violation

Backport of the equivalent fix on the master PR (cadence-workflow#1074) to this v3.13.x
backport branch (cadence-workflow#1075).

Why:
Follow-up to the invocation-count check, addressing
github.com/cadence-workflow/pull/1074#discussion_r3778369888
(gitar-bot): when a propagator retries the task after an exception
(invocationCount != 1), ContextPropagatorContractViolationError was thrown
with a null cause, even though `thrown` already held the exception from the
attempt that triggered the retry. Attaching it makes it much easier to
debug a misbehaving propagator, since the resulting stack trace shows both
the contract violation and the original failure that provoked it.

Change:
- ContextPropagatorContractViolationError.unexpectedInvocationCount now
  takes the last captured Throwable and passes it through as the cause,
  instead of always passing null.
- ContextThreadLocal.runWithContext passes thrown.get() into that call.
- Updated the retry test to assert the original exception is preserved as
  the cause.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@rlei rlei changed the title [refactor] Replace imperative set/unset context propagation with lexically-scoped runWithContext (v3.13.x backport) refactor: Replace imperative set/unset context propagation with lexically-scoped runWithContext (v3.13.x backport) Aug 13, 2026
@rlei rlei changed the title refactor: Replace imperative set/unset context propagation with lexically-scoped runWithContext (v3.13.x backport) feat(ContextPropagator): Replace imperative set/unset context propagation with lexically-scoped runWithContext (v3.13.x backport) Aug 13, 2026
rlei and others added 4 commits August 13, 2026 23:02
…h lexically-scoped runWithContext

Why:
JDK 25 finalizes ScopedValue (JEP 481), a lexically-scoped, immutable
alternative to ThreadLocal for propagating context: a value is bound and the
task runs inside that binding in a single call (`ScopedValue.where(...).call(task)`),
with no separate unset step -- the binding is torn down automatically when the
call returns. The previous ContextPropagator API couldn't express that: context
was installed via `setCurrentContext`/`deserializeContext` at the top of a
handler (e.g. ActivityWorker.propagateContext, LocalActivityWorker.restoreContext,
WorkflowThreadImpl's propagateContextToCurrentThread) and torn down via
`unsetCurrentContext` in a `finally` block possibly several methods away, which
doesn't map onto ScopedValue's single-call binding model.

Restructuring around a single `runWithContext(context, task)` call gets us:
- Simplicity: a ScopedValue-backed propagator only needs to override
  `runWithContext` and call `ScopedValue.where(VALUE, context).run(task)` --
  no separate set/get/unset methods to keep consistent with each other.
- Immutable context data: ScopedValue bindings are write-once for the dynamic
  extent of the call; nothing downstream can accidentally mutate a shared
  context object the way a mutable ThreadLocal value can be.
- No unpaired unset: because the binding and the task execution happen in the
  same call frame, cleanup is guaranteed by stack unwinding. The old shape's
  risk -- an early return, a thrown exception before the try block, or a
  future refactor splitting set/unset further apart leaving a propagator's
  context set with no matching unset and leaking it onto whatever task a
  pooled thread picks up next -- goes away entirely.

Change:
- Add `ContextPropagator.runWithContext(Object context, ContextRunnable task)`,
  a default method that wraps `setCurrentContext`/`task.run()`/`unsetCurrentContext`
  in a single try/finally, so set and unset are always paired in one call frame.
- `ContextThreadLocal.runWithContext(...)` composes all configured propagators
  into a single nested call chain and executes the task inside it, replacing the
  old `propagateContextToCurrentThread` + manual `unsetCurrentContext` pair.
- Update the three call sites (ActivityWorker, LocalActivityWorker,
  WorkflowThreadImpl) to run their task through this single call instead of
  bracketing it with separate propagate/unset calls.

Existing propagators that implement `setCurrentContext`/`unsetCurrentContext`
are unaffected; `runWithContext` has a default implementation built on those
methods, so this is additive for current implementations and opt-in for
ScopedValue-based ones.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Haofeng Lei <haofeng.lei@uber.com>
…tion

Backport of the equivalent fix on the master PR (cadence-workflow#1074) to this v3.13.x
backport branch (cadence-workflow#1075).

Why:
Review feedback on cadence-workflow#1074 (github.com/cadence-workflow/pull/1074#pullrequestreview-4929567768)
pointed out a real regression risk in the runWithContext redesign: because
a ContextPropagator's runWithContext(context, task) now directly wraps the
call to task.run(), a buggy implementation can structurally catch and
suppress (or replace) whatever the task throws -- including workflow
cancellation and thread-destruction signals from WorkflowThread. The old
imperative setCurrentContext/unsetCurrentContext shape made this impossible,
since propagators never touched the call site of the wrapped task.

Change:
- ContextThreadLocal.runWithContext now wraps the innermost task in a canary
  that records any Throwable it throws before rethrowing it. After the
  composed propagator chain returns, if the canary recorded a Throwable but
  none escaped the chain, some propagator in it swallowed the exception --
  in which case a new ContextPropagatorSwallowedExceptionError is thrown
  (listing the propagators applied to that call) instead of silently
  continuing as if the task had succeeded.
- ContextPropagatorSwallowedExceptionError extends Error, matching this
  codebase's existing convention for framework-level, must-not-be-retried
  conditions (NonDeterminisicWorkflowError, DestroyWorkflowThreadError,
  WorkflowRejectedExecutionError). This also means it composes correctly
  with existing handling: WorkflowThreadImpl already special-cases
  `catch (Error e)` to abort the decision rather than fail the workflow,
  and PollTaskExecutor routes any Throwable escaping an activity handler
  to the configured uncaughtExceptionHandler instead of a normal activity
  failure response.
- Strengthened the ContextPropagator#runWithContext javadoc to state the
  contract explicitly: implementations must only wrap task.run() in
  try/finally, never try/catch.
- Added a test with a deliberately misbehaving propagator that catches and
  suppresses the task's exception, verifying the new error is thrown with
  the original exception as its cause.

This does not catch every misbehavior -- a propagator that catches and
replaces the exception with a different one can't be reliably distinguished
from a propagator intentionally translating exception types, so that half
of the original concern is addressed by the javadoc contract instead.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Haofeng Lei <haofeng.lei@uber.com>
…ong number of times

Backport of the equivalent fix on the master PR (cadence-workflow#1074) to this v3.13.x
backport branch (cadence-workflow#1075).

Why:
Follow-up to the swallow-detection fix, addressing
github.com/cadence-workflow/pull/1074#discussion_r3778176907
(gitar-bot): the swallow-detection canary's `thrown` AtomicReference was set
on the first task failure and never cleared, so a propagator that catches
the task's exception and then retries it, eventually succeeding, would
incorrectly be flagged as having swallowed the earlier failure.

Rather than making the canary tolerate retries, retrying task is itself a
contract violation -- ContextPropagator#runWithContext must call task.run()
exactly once. So instead this adds a second, independent check: an
invocation counter that must equal exactly one. It covers both a propagator
that skips calling task entirely and one that calls it more than once
(including the retry-then-succeed case), and it is checked before the
existing swallow check, so a retry-then-succeed propagator is now correctly
reported as an invocation-count violation rather than a misleading
"swallowed exception" one.

Change:
- ContextThreadLocal.runWithContext tracks an AtomicInteger invocationCount
  alongside the existing AtomicReference<Throwable> thrown, incrementing it
  in the same canary that wraps task.run(). After the composed propagator
  chain returns normally, invocationCount != 1 is checked first (covers
  zero and multiple invocations), then the existing swallowed-exception
  check.
- Renamed ContextPropagatorSwallowedExceptionError to
  ContextPropagatorContractViolationError (extends Error, same as before)
  since it now models two distinct contract violations via named static
  factory methods (swallowedException, unexpectedInvocationCount) instead
  of a single constructor.
- Updated ContextPropagator#runWithContext's javadoc contract to state
  task.run() must be called exactly once -- never skipped, never retried.
- Added a test with a propagator that retries task after catching its
  exception and eventually succeeds, verifying it is reported as an
  invocation-count violation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Haofeng Lei <haofeng.lei@uber.com>
…ion-count violation

Backport of the equivalent fix on the master PR (cadence-workflow#1074) to this v3.13.x
backport branch (cadence-workflow#1075).

Why:
Follow-up to the invocation-count check, addressing
github.com/cadence-workflow/pull/1074#discussion_r3778369888
(gitar-bot): when a propagator retries the task after an exception
(invocationCount != 1), ContextPropagatorContractViolationError was thrown
with a null cause, even though `thrown` already held the exception from the
attempt that triggered the retry. Attaching it makes it much easier to
debug a misbehaving propagator, since the resulting stack trace shows both
the contract violation and the original failure that provoked it.

Change:
- ContextPropagatorContractViolationError.unexpectedInvocationCount now
  takes the last captured Throwable and passes it through as the cause,
  instead of always passing null.
- ContextThreadLocal.runWithContext passes thrown.get() into that call.
- Updated the retry test to assert the original exception is preserved as
  the cause.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Haofeng Lei <haofeng.lei@uber.com>
@rlei
rlei force-pushed the codex/cadence-scoped-propagation-v3.13.3 branch from 080d51b to 5ed4dae Compare August 13, 2026 21:07
rlei added 2 commits August 14, 2026 10:25
…ssertion

ReplayDecider#executeLocalActivities caps how many local activities land
in one decision batch by comparing elapsed processing time against 80%
of the decision task timeout. With the test's 5s timeout that budget is
exactly 4000ms -- precisely 2x the fixed 2000ms sleepActivity duration
used here -- so whether a 3rd activity also completes within the first
batch is decided by a few milliseconds of scheduling jitter around that
exact tie, not anything the test should be asserting on.

Tolerate the batch boundary landing on either side instead of asserting
a single hardcoded snapshot, while still verifying that querying
mid-execution returns real, non-decreasing progress before the workflow
completes.

Signed-off-by: Haofeng Lei <haofeng.lei@uber.com>
… span count

TChannel finishes its own per-request outbound span (in addition to the
explicit "cadence-..." span WorkflowServiceTChannel creates) from a
future completion callback registered on the TFuture. That callback is
not guaranteed to run before the blocking RPC call returns to the
caller, so the assertion on finishedSpans().size() could observe only
1 of the 2 expected spans immediately after enqueueSignalWithStart()
returns, depending on which thread wins the race.

Poll briefly for the expected span count before asserting instead of
checking immediately.

Signed-off-by: Haofeng Lei <haofeng.lei@uber.com>
@gitar-bot

gitar-bot Bot commented Aug 14, 2026

Copy link
Copy Markdown
CI failed: Unit test failures occurred across CI jobs due to failing test assertions in the test suite.

Overview

CI runs encountered test failures in the Gradle :test task (1 test failure in the standard unit test run and 3 test failures in the Docker service test run), directly related to the changes introduced in the PR.

Failures

Gradle Unit Test Failures (confidence: high)

  • Type: test
  • Affected jobs: 94580624034, 94827982305
  • Related to change: yes
  • Root cause: Unit tests failed during execution of the Gradle :test task (1 failing test in standard tests, 3 failing tests in Docker-based sticky-off tests), likely caused by test assertions affected by the new runWithContext ContextPropagator implementation or related changes.
  • Suggested fix: Review the Gradle test report files at build/reports/tests/test/index.html to identify the failing tests, verify the test assertions and logic against the updated context propagation behavior, and fix any incorrect expectations or code issues.

Summary

  • Change-related failures: 2 job failure groups involving unit test failures in the Gradle build.
  • Infrastructure/flaky failures: 0
  • Recommended action: Inspect the local or artifact test reports for the exact stack traces and failing test names, then update the tests or implementation to ensure proper context propagation and correct test expectations.
Code Review ✅ Approved 1 resolved / 1 findings

Backports lexically-scoped runWithContext propagation to replace imperative set/unset calls, addressing the invocation-count violation exception-discarding finding. No issues found.

✅ 1 resolved
Quality: Invocation-count violation discards the triggering exception

📄 src/main/java/com/uber/cadence/internal/context/ContextThreadLocal.java:118-121 📄 src/main/java/com/uber/cadence/internal/context/ContextPropagatorContractViolationError.java:49-60
When a propagator retries the task after catching a failure, thrown holds the original exception that triggered the retry, but unexpectedInvocationCount(applied, count) is called with a null cause (ContextThreadLocal.java:118-120, ContextPropagatorContractViolationError.java:49-60), so the diagnostic stack trace of the underlying failure is lost. Consider passing thrown.get() as the cause when non-null to aid debugging of the misbehaving propagator. The existing test asserts e.getCause()==null, so this is a deliberate choice, but attaching the cause would make production diagnosis easier.

Tip

Comment Gitar fix CI or enable auto-apply: gitar auto-apply:on

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Auto-apply Compact
gitar auto-apply:on         
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

@shijiesheng
shijiesheng merged commit e2de672 into cadence-workflow:v3.13.x Aug 14, 2026
5 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants