How can I fix jagged metrics when using aggregate transforms?

[syedg1]

Question

i setup an aggregate transform for incremental metrics, but noticed that the metrics are very jagged when i view them in datadog (emmitting to datadog metrics sink)., and there seems to be some underreporting of data (blue: baseline metric, purple: vector aggregated metric). any tips on how i can smoothen the graph when using the aggregate transform and prevent data loss?

[schahal]

This may be related to #24421

@pront @jszwedko @thomasqueirozb thoughts? That seems like a safe PR, especially if its theory is sound?

[kinthaiofficial]

Jagged metrics from the aggregate transform usually come from one of these causes:

Window boundary artifacts
The aggregate transform flushes at interval boundaries (e.g., every 60s). If your upstream source emits events at irregular intervals, the count per window varies — not because of actual spikes but because different windows capture different numbers of events. This looks "jagged" on a graph.

Fix: use a longer aggregation window to smooth out irregularities, or add a reduce transform after aggregate to compute a rolling average.

Multiple Vector instances without deduplication
If you're running Vector in a sidecar/daemonset pattern with multiple replicas all sending to the same downstream, each instance has its own aggregate window that doesn't align with others. The receiving side sees overlapping windows and double-counts.

Fix: use a single Vector aggregator instance (not sidecar) for metrics aggregation, or use Prometheus remote_write with deduplication at the receiver.

Flush interval vs scrape interval mismatch
If your aggregate flush interval (60s) doesn't align with your downstream scrape interval (15s), you'll see 3 scrapes with nothing and 1 scrape with 60s worth of data. That's a 4x spike pattern, which looks jagged.

Fix: set aggregate flush interval to match or be a multiple of your scrape interval.

The aggregate transform reset behavior
After flushing, aggregate resets counters to 0. If you're graphing cumulative values, you need to convert to a rate or use cum_sum post-aggregate. Graphing raw aggregate output as if it were a gauge produces sawtooth patterns.

Is the jaggedness periodic (spike every N seconds) or random?

[syedg1]

I was actually able to resolve this issue using #24421. The event time aggregation is what we needed to ensure that metrics are consistent with baseline metrics. Note, we did need to add a timestamp_shift to prevent the event time aggregation logic from overwriting data when reflushing buckets (more details in that PR).