Container CPU Stuck at ~200% After Task Fails on Large Text Variable

[Dimmvvv]

Describe the issue

Description:

When passing a large text output (e.g. 500,000 lines) into another task via variables {{ outputs... }}, Kestra runs out of allocated RAM and fails the task as expected. However, after the failure, the system does not clean up the memory properly. The container enters a background cleanup loop, keeping host CPU usage stuck at 150–180% indefinitely.

Test Flow (YAML):

id: memory_leak_test
namespace: qa.prod.test

tasks:
  - id: export_huge_csv
    type: io.kestra.plugin.scripts.shell.Commands
    runner: PROCESS
    commands:
      - yes "1001,John Doe,user@example.com,Active,2026-08-19" | head -n 500000

  - id: process_csv_in_memory
    type: io.kestra.plugin.core.log.Log
    message: "Data: {{ outputs.export_huge_csv.stdout }}"

Steps to Reproduce:

1. Run the memory_leak_test flow.
2. Wait for the second step (process_csv_in_memory) to fail with an error.
3. Check container resource consumption (docker stats) 15 seconds after the step has failed.

Expected Result:

The second step fails due to variable size limits, but the container frees up memory and CPU usage drops back to baseline (~0–5%).

Actual Result:

The task fails, but CPU usage remains stuck at 180–200% long after execution ends, causing system slowdowns until the container is restarted.

Impact & Business Risks:

• Risk of Denial of Service (DoS): Worker node CPU starvation degrades overall system throughput and delays parallel flow executions across other pipelines and teams.

• Increased Cloud Infrastructure Costs: Continuous background CPU utilization on cloud instances (AWS / GCP / Azure) directly drives up cloud billing due to wasted compute resources.

• Required Manual Intervention: The system fails to self-heal after execution completion, requiring manual container restarts—a critical issue for zero-downtime, 24/7 Production environments.

Environment

• Kestra Version: Kestra v1.0.56
  OS: MacOS 26.6.1

[loicmathieu]

On Kestra 2.0, this would be disallow by default: a too big message will fail the task without retaining the message.