Fix tag-based concurrency batching delay

[devin-ai-integration]

Addresses a performance issue where tag-based concurrency limits cause tasks to execute in discrete batches instead of a smooth waterfall. When concurrency slots are full, all waiting tasks sleep for the same fixed duration (~30s), then all wake up and compete simultaneously — repeating the pattern and significantly increasing total runtime.

Two changes:

1. Lower tag_concurrency_slot_wait_seconds default from 30s to 10s — aligns with the value already used in Prefect Cloud (nebula). The 30s default was a major contributor to the batching problem.

2. Use avg_slot_occupancy_seconds for server-side delay — instead of always delaying by the fixed setting value, the orchestration rule now uses the most contended V2 limit's avg_slot_occupancy_seconds (capped at the setting max). This makes the delay proportional to actual slot turnover, so tasks retry sooner when slots free up quickly. Falls back to the configured max when avg_slot_occupancy_seconds is zero (e.g. newly created limits).

Note: the V1 concurrency limit path is unchanged since V1 limits don't track avg_slot_occupancy_seconds.

Existing tests were updated to reflect the new 10s default for tag limit decay rates and retry-after bounds.

closes https://linear.app/prefect/issue/OSS-7078/tag-based-concurrency-adds-a-noticeable-delay-with-gcls

Checklist

• This pull request references any related issue by including "closes <link to issue>"
• If this pull request adds new functionality, it includes unit tests that cover the changes
• If this pull request removes docs files, it includes redirect settings in mint.json.
• If this pull request adds functions or classes, it includes helpful docstrings.

Things for reviewers to verify

• The active_slots / limit ratio as the heuristic for picking the "most contended" limit — note this will raise ZeroDivisionError if a limit has limit=0, though that should not happen for active V2 limits
• avg_slot_occupancy_seconds or max_wait uses Python truthiness: a value of 0.0 (cold start / no samples yet) falls back to max_wait, which is intentional
• Whether the 30→10 default change warrants a changelog/migration note for OSS users who relied on the previous default
• No new unit tests were added for the adaptive delay selection logic in SecureTaskConcurrencySlots — existing concurrency tests (80 + 14 integration) all pass, and test expectations were updated where the default changed

Link to Devin session: https://app.devin.ai/sessions/f27fc316ae2d4818bce18966344b368b
Requested by: @desertaxle

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[codspeed-hq]

Merging this PR will not alter performance

✅ 2 untouched benchmarks

---

_{Comparing devin/OSS-7078-1773804269 (68e93d2) with main (860f8ca)}