Skip to content

Worker occasionally dies with "UpdateForWorker stream closed early" #2115

@rejuvenile

Description

@rejuvenile
  2026-01-21T20:58:46.749339Z ERROR nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the requirements, will attempt to do it in the background., operation_id: 9c671a53-b7fc-4c2b-ba51-5639f1b064b5
    at nativelink-worker/src/running_actions_manager.rs:1415
    in nativelink_worker::local_worker::worker_start_action
    in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "9c671a53-b7fc-4c2b-ba51-5639f1b064b5", digest_function: BLAKE3
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"

  2026-01-21T20:58:46.749389Z ERROR nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the requirements, will attempt to do it in the background., operation_id: 6cac1188-bff7-4038-996f-4612bb138fe2
    at nativelink-worker/src/running_actions_manager.rs:1415
    in nativelink_worker::local_worker::worker_start_action
    in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "6cac1188-bff7-4038-996f-4612bb138fe2", digest_function: BLAKE3
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"

  2026-01-21T20:58:59.487142Z ERROR nativelink_worker::local_worker: ERROR_MSG: "Actions in transit did not reach zero before we disconnected from the scheduler"
    at nativelink-worker/src/local_worker.rs:701
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"


thread 'main' (24873374) panicked at src/bin/nativelink.rs:702:9:
Error { code: Internal, messages: ["UpdateForWorker stream closed early", "Actions in transit did not reach zero before we disconnected from the scheduler"] }

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions