Skip to content

feat(observability): profiles end-to-end (gateway + UI panel)#5382

Draft
Ma77Ball wants to merge 109 commits into
apache:mainfrom
Ma77Ball:obs/pr8/profiles
Draft

feat(observability): profiles end-to-end (gateway + UI panel)#5382
Ma77Ball wants to merge 109 commits into
apache:mainfrom
Ma77Ball:obs/pr8/profiles

Conversation

@Ma77Ball

@Ma77Ball Ma77Ball commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Continuous-profiling retrieval through the gateway plus a flame-graph UI panel.
Backend:

  • ParcaClient: gRPC-Web client to the Parca server, with a profiles query builder and parseProfiles.
  • ProfilesResource exposing scoped profile queries (optional workflow and execution id), registered in TexeraWebApplication.
    Frontend:
  • Profiles panel with a flame-chart component, scoped by workflow and execution id, with explicit empty states for "no samples yet" and for environments where profiling is disabled.

Any related issues, documentation, or discussions?

Closes: #5374
Part of #4070. Stacked on #5381.

How was this PR tested?

  • Backend specs for ParcaClient, the profiles query builder, and the parser; sbt scalafmtCheckAll passes.
  • Frontend profiles-panel and flame-chart specs; prettier-eslint and eslint pass.
  • Compile and the full test suites run in this PR's CI.

Was this PR authored or co-authored using generative AI tooling?

Co-authored with Claude Opus 4.8 in compliance with ASF

Ma77Ball and others added 8 commits June 5, 2026 04:49
…, SDK bootstrap (default-off)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s panel

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nt scope, health, routing)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ca/eBPF profiling

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tracing primitives

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…I flame-graph panel

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… trace tree panel

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…UI metrics panel (ECharts)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added engine dependencies Pull requests that update a dependency file frontend Changes related to the frontend GUI docs Changes related to documentations dev common labels Jun 5, 2026
@codecov-commenter

codecov-commenter commented Jun 5, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 60.35645% with 823 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.69%. Comparing base (dfa0434) to head (917e46d).

Files with missing lines Patch % Lines
...observability/gateway/ObservabilityResources.scala 0.00% 264 Missing ⚠️
...texera/web/observability/gateway/ParcaClient.scala 33.71% 109 Missing and 7 partials ⚠️
...observability/logs-panel/logs-panel.component.html 56.92% 55 Missing and 1 partial ⚠️
...ala/org/apache/texera/observability/OtelInit.scala 63.30% 43 Missing and 8 partials ⚠️
...apache/texera/web/observability/gateway/dtos.scala 76.96% 37 Missing and 7 partials ⚠️
...texera/web/observability/gateway/AuditLogger.scala 0.00% 41 Missing ⚠️
...xera/web/observability/gateway/ScopeResolver.scala 19.44% 27 Missing and 2 partials ⚠️
...ra/web/observability/gateway/ResponseParsers.scala 81.06% 2 Missing and 23 partials ⚠️
...era/web/observability/gateway/GatewayContext.scala 0.00% 19 Missing ⚠️
...rvability/traces-panel/traces-panel.component.html 44.11% 19 Missing ⚠️
... and 20 more
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #5382      +/-   ##
============================================
- Coverage     52.91%   50.69%   -2.23%     
- Complexity     2626     2670      +44     
============================================
  Files          1090      985     -105     
  Lines         42188    36793    -5395     
  Branches       4531     4950     +419     
============================================
- Hits          22324    18651    -3673     
+ Misses        18555    16726    -1829     
- Partials       1309     1416     +107     
Flag Coverage Δ
access-control-service 70.91% <ø> (ø)
agent-service ?
amber 52.92% <51.65%> (-0.09%) ⬇️
computing-unit-managing-service 1.65% <ø> (ø)
config-service 56.71% <ø> (ø)
file-service 57.06% <ø> (ø)
frontend 49.01% <79.32%> (+1.08%) ⬆️
pyamber ?
python ?
workflow-compiling-service 58.69% <ø> (ø)

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 4 better · 🔴 2 worse · ⚪ 9 noise (<±5%) · 0 without baseline

Compared against main dfa0434 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🟢 bs=10 sw=10 sl=64 397 0.242 24,868/34,503/34,503 us 🟢 -53.3% / ⚪ within ±5%
🔴 bs=100 sw=10 sl=64 950 0.58 104,592/134,607/134,607 us 🔴 +21.4% / 🟢 -6.8%
bs=1000 sw=10 sl=64 1,125 0.687 887,141/940,604/940,604 us ⚪ within ±5% / 🟢 -8.8%
Baseline details

Latest main dfa0434 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 397 tuples/sec 363 tuples/sec 410.82 tuples/sec +9.4% -3.4%
bs=10 sw=10 sl=64 MB/s 0.242 MB/s 0.222 MB/s 0.251 MB/s +9.0% -3.5%
bs=10 sw=10 sl=64 p50 24,868 us 24,354 us 23,785 us +2.1% +4.6%
bs=10 sw=10 sl=64 p95 34,503 us 73,948 us 34,980 us -53.3% -1.4%
bs=10 sw=10 sl=64 p99 34,503 us 73,948 us 34,980 us -53.3% -1.4%
bs=100 sw=10 sl=64 throughput 950 tuples/sec 993 tuples/sec 891.94 tuples/sec -4.3% +6.5%
bs=100 sw=10 sl=64 MB/s 0.58 MB/s 0.606 MB/s 0.544 MB/s -4.3% +6.5%
bs=100 sw=10 sl=64 p50 104,592 us 100,961 us 112,277 us +3.6% -6.8%
bs=100 sw=10 sl=64 p95 134,607 us 110,878 us 139,802 us +21.4% -3.7%
bs=100 sw=10 sl=64 p99 134,607 us 110,878 us 139,802 us +21.4% -3.7%
bs=1000 sw=10 sl=64 throughput 1,125 tuples/sec 1,131 tuples/sec 1,041 tuples/sec -0.5% +8.1%
bs=1000 sw=10 sl=64 MB/s 0.687 MB/s 0.69 MB/s 0.635 MB/s -0.4% +8.1%
bs=1000 sw=10 sl=64 p50 887,141 us 896,307 us 972,714 us -1.0% -8.8%
bs=1000 sw=10 sl=64 p95 940,604 us 939,985 us 1,023,057 us +0.1% -8.1%
bs=1000 sw=10 sl=64 p99 940,604 us 939,985 us 1,023,057 us +0.1% -8.1%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,503.91,200,128000,397,0.242,24867.98,34503.38,34503.38
1,100,10,64,20,2105.09,2000,1280000,950,0.580,104591.60,134607.09,134607.09
2,1000,10,64,20,17777.18,20000,12800000,1125,0.687,887140.75,940604.30,940604.30

Ma77Ball added 25 commits June 14, 2026 03:42
  ResponseParsersSpec (fresh-build compile failure)
  ResponseParsersSpec (fresh-build compile failure)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common dependencies Pull requests that update a dependency file dev docs Changes related to documentations engine frontend Changes related to the frontend GUI platform Non-amber Scala service paths

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Observability] Continuous profiling: gateway endpoint and flame-graph panel

2 participants