[BUG] CoPilot sets terminationGracePeriodSeconds in nanoseconds instead of seconds, causing pods to hang in Terminating state

[mitja-kleider]

Describe the bug

In flyteplugins/go/tasks/pluginmachinery/flytek8s/copilot.go, the CoPilot timeout is assigned directly to TerminationGracePeriodSeconds without converting from nanoseconds to seconds:

coPilotPod.TerminationGracePeriodSeconds = (*int64)(&cfg.Timeout.Duration)

cfg.Timeout.Duration is a time.Duration, which stores nanoseconds as int64. Kubernetes expects terminationGracePeriodSeconds in seconds.

For example, a 1-hour copilot timeout produces terminationGracePeriodSeconds: 3600000000000.

Expected behavior

The duration should be converted to seconds before assignment:

seconds := int64(cfg.Timeout.Duration.Seconds())
coPilotPod.TerminationGracePeriodSeconds = &seconds

How to reproduce

1. Run a Flyte task with CoPilot enabled (i.e., with output handling) which ignores SIGTERM
2. Cancel or timeout the execution
3. Observe the pod enters Terminating state and never completes termination until manual force-deletion
4. Inspect the pod: kubectl get pod <name> -o jsonpath='{.spec.terminationGracePeriodSeconds}' returns 3600000000000
5. Inspect the deletion timestamp: it will be decades/centuries in the future

Environment

• FlytePropeller version: v1.16.3 (also present on master/v2.0.9)

[runllm]

It could be possible that this bug is valid: in copilot.go, cfg.Timeout.Duration is a time.Duration (nanoseconds), but Kubernetes expects terminationGracePeriodSeconds in seconds. Assigning the raw value would result in an extremely large grace period, matching your observation.

Though I'm not entirely certain, it might be that the correct fix is to convert the duration to seconds before assignment, as you suggested. However, the specific assignment to TerminationGracePeriodSeconds is not explicitly shown in the provided data sources, so I can't confirm the exact line. Would you like more detail on how to implement or test this fix?

Sources:

• copilot.go

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!