Kubernetes Executor : Quota exceeded, tasks fail

[datasc24]

Hello,
We were using AIrflow 2.7 with Kubernetes Executor and when we used to launch a lot of tasks, they were staying queued. But after migrating to Airflow 3.2, if we exceed our namespace quota, tasks are failing.
Do you have any solution to deal with this ? Is there a way to force task to stay queued, wiaiting for ressources instead of failing ?

[cookesan]

Kubernetes quota rejection happens before the worker pod exists, so Airflow can only retry or throttle pod creation; it cannot make Kubernetes accept pods beyond the namespace quota.

In the current Kubernetes Executor source, 403 errors whose message contains exceeded quota are retried only while task_publish_max_retries allows it. If that value is 0, or if the retry budget is exhausted while the quota is still full, the task is failed.

I would check these first:

• AIRFLOW__KUBERNETES_EXECUTOR__TASK_PUBLISH_MAX_RETRIES
• worker_pods_creation_batch_size
• global parallelism
• DAG/task concurrency
• pools for the high-fanout work
• worker pod CPU/memory requests versus the namespace quota

The durable fix is to keep Airflow's maximum pod demand below the namespace quota, or raise the quota. If you want the old "stay queued until quota frees" behavior, a finite or infinite publish retry setting is the closest Airflow-side lever, but it should be paired with concurrency limits so the scheduler is not constantly asking Kubernetes for pods it cannot create.