Skip to content

Promql cpp functions#272

Open
cherep58 wants to merge 88 commits into
ppfrom
promql_cpp
Open

Promql cpp functions#272
cherep58 wants to merge 88 commits into
ppfrom
promql_cpp

Conversation

@cherep58
Copy link
Copy Markdown
Collaborator

@cherep58 cherep58 commented Mar 24, 2026

Summary

Implement C++ pushdown optimizations for PromQL functions at the storage layer. When the query engine provides SelectHints.Func, the storage can now reduce data before it reaches the Go PromQL engine — returning only the samples needed for the specific function instead of the full series.

New decorator iterators

  • Aggregation: MinOverTimeIterator, MaxOverTimeIterator, LastOverTimeIterator, SumOverTimeIterator (via OverTimeFuncIterator template with pluggable handlers)
  • Counter/rate family: RateIterator (rate, increase), IRateIterator (irate, idelta)
  • Transform: DeltaIterator, ChangesIterator, ResetsIterator
  • Downsampling: DownsamplingDecodeIterator — interval-based sample reduction

Refactored decode iterator infrastructure

  • CRTP-based DecodeIteratorTrait with seek(), seek_to(), invalidate(), set() methods
  • SeekResult enum for composable seek logic (kUpdateSample, kNext, kStop, kUpdateSampleNextAndStop)
  • UniversalDecodeIterator extended with seek(), seek_to(), invalidate(), set() that dispatch through std::visit
  • Outer DecodeIterator variant wrapping all iterator types for the Go binding layer

Function dispatch

  • gperf-generated perfect hash (FunctionNamesHash) for O(1) function name lookup
  • create_decode_iterator() dispatches on SelectHints.Func to construct the appropriate iterator

Go bridge changes

  • SelectHints passed through cppbridge via ABI-compatible GenericSelectHints<Go::String, Go::SliceView>
  • downsamplingMs parameter added to Query() and ChunkRecoder paths
  • DataStorageSerializedDataIterator control block simplified: direct Timestamp()/Value() accessors replacing raw field access
  • Namespace change: entrypoint::headentrypoint::series_data

Testing

  • Parametric C++ unit tests for each new iterator (boundary conditions, StaleNaN handling, time interval filtering, reset scenarios)
  • Go integration tests for all dispatched PromQL functions (downsampling, min/max/last/sum_over_time, rate, increase, changes, delta, irate, idelta, resets)

@cherep58 cherep58 marked this pull request as ready for review April 29, 2026 14:15
Comment thread pp/entrypoint/series_data_data_storage.cpp Outdated
Comment thread pp/go/storage/querier/series_bench_test.go
Comment thread pp/entrypoint/series_data/serialization.h
Comment thread pp/entrypoint/series_data_data_storage.h Outdated
Comment thread pp/prometheus/query.h
Comment thread pp/series_data/decoder/decorator/changes_iterator.h
Copy link
Copy Markdown
Collaborator

@vporoshok vporoshok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: WindowFunctionIterator sub-interval logic vs ADR-001

Analyzed the sub-interval splitting algorithm in WindowFunctionIterator against ADR-001 (range function iteration algorithm). The code works correctly for the most common case (s < w < 2s), but has critical issues in two other cases.

Summary of findings

# Severity Finding
1 Bug/Critical advance_interval() breaks for w > 2s — produces invalid interval (min > max), potential infinite loop
2 Bug/Critical Incorrect boundaries for w < s — aggregation captures samples from gaps between windows
3 Bug/Moderate Possible double-count for sum_over_time at interval boundaries (depends on seek_to inclusive/exclusive semantics)
4 Design count_over_time missing from dispatch (falls back to universal iterator — correct but no pushdown)
5 Design sum_over_time result timestamp is last raw sample's ts, not sub-interval end
6 Observation min/max sub-interval collapsing optimization from ADR not implemented (correct but sub-optimal)

Comment thread pp/series_data/decoder/decorator/window_function_iterator.h Outdated
Comment thread pp/series_data/decoder/decorator/window_function_iterator.h Outdated
Comment thread pp/series_data/decoder/decorator/sum_over_time.h
Comment thread pp/entrypoint/series_data/decode_iterator.h
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants