fix(metrics): normalize persistence metric labels for Prometheus

[ribaraka]

What changed?
Normalized persistence metric label sets emitted by the metered persistence wrappers so Prometheus does not reject metrics registered under the same name with different label keys.

• Always emitting is_retry on metered persistence wrapper paths.
• Defaulting task_category="none" on direct persistence metrics when no task category applies.
• Renaming collision-prone persistence rollup metrics so rollups do not emit into already-used direct metric names.

Why?
#7844
Prometheus requires every metric name to use a consistent label-name set. Cadence persistence metrics could emit the same metric name from different wrapper paths with different labels, causing Prometheus reporter warnings and dropped metrics.

How did you test it?
Added a Prometheus-backed regression test that exercises persistence metered wrapper paths with different label dimensions and verifies no Prometheus registration errors are emitted.
go test ./common/persistence/wrappers/metered

Potential risks
This changes some persistence rollup metric names, which may require dashboard or alert updates for users relying on the old aggregate persistence metric names.

Release notes
This PR changes the persistence metrics contract.
Persistence rollup metrics were renamed to avoid prometheus label-set conflicts with directly emitted persistence metrics:

• persistence_requests rollup is now persistence_requests_rollup
• persistence_latency rollup is now persistence_latency_rollup
• persistence_latency_ns rollup is now persistence_latency_ns_rollup

The direct persistence_requests / persistence_latency / persistence_latency_ns metrics still exist, but they are no longer reused as rollup targets. During a mixed-version rollout, dashboards may need to account for both old and new rollup names until all clusters are upgraded. This PR also normalizes labels on direct persistence metrics, adding labels like is_retry and task_category="none" where they were previously missing. Selectors by metric name should still match, but dashboards/alerts that aggregate or group by labels may need to explicitly aggregate away those new dimensions.

Documentation Changes

[neil-xie]

Hey, the code changes look good to me, I am checking we will need to do with the metrics rename internally since we have bunch of alerts/dashboards based on it, and our deployment will take time to roll out to all production clusters

[ribaraka]

@neil-xie Thanks for checking.

I'll add this clearly to the release notes so the rollout impact is visible.

[neil-xie]

LGTM, thanks for fixing it.
As discussed, I will merge when we make the changes internally since it will impact existing dashboards/alerts

[gitar-bot]

Code Review ✅ Approved 1 resolved / 1 findings

Normalizes persistence metric labels and renames rollup metrics to resolve Prometheus label-set conflicts. The previously missing task category tag in the call() path is also addressed, resulting in consistent metric reporting.

✅ 1 resolved

✅ Bug: Per-domain path in call() still lacks ensureTaskCategoryTag

📄 common/persistence/wrappers/metered/base.go:164-168 📄 common/persistence/wrappers/metered/base.go:217
In call(), when perDomain == true (line 165-168), ensureTaskCategoryTag is NOT applied to tags. This means PersistenceRequestsPerDomain / PersistenceLatencyPerDomain metrics can still be emitted with inconsistent label sets:

• GetHistoryTasks with a domain provides task_category via MetricTags() → emits with {domain, is_retry, task_category}
• GetWorkflowExecution with a domain does NOT provide task_category → emits with {domain, is_retry}

Similarly, in callWithDomainAndShardScope(), domainMetricsScope (line 217) uses raw additionalTags without ensuring task_category, so PersistenceRequestsPerDomain emitted there will also have inconsistent labels depending on whether the request's MetricTags() includes task_category.

This is the same class of Prometheus registration error this PR aims to fix, but for the per-domain metric path.

Rules ✅ All requirements met

Repository Rules

✅ GitHub Issue Linking Requirement: The PR description includes a valid link to a cadence-workflow issue (#7844).

✅ PR Description Quality Standards: The PR description follows the required structure and guidance, providing clear technical rationale, reproduction steps for tests, and well-detailed release notes.

_{2 rules not applicable. Show all rules by commenting gitar display:verbose.}

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Auto-apply	Compact
gitar auto-apply:on	gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[Shaddoll]

metered_execution.tmpl is not updated