Approach: landing Modern Observability (#4070) as a stack of small, self-contained PRs

[Ma77Ball]

Proposal: land "Modern Observability" (#4070) as a stack of small, self-contained PRs

Opening this to get feedback on the approach and procedure before raising the PRs.

Problem

The observability work for #4070 grew into one large branch (~16k insertions across ~230 files): OTel bootstrap + log/metric/trace/profile pipelines, a tenant-scoped query gateway, an Angular dashboard, docker-compose backends — plus a codebase-wide logging/readability pass that rode along. That is far too big to review or merge safely as one PR.

Approach

Decompose it into 26 small branches in two independent tracks, each:

• ≤ ~1000 lines of logic, compiles green on its own, and is non-breaking (the feature is default-off via OTEL_SDK_DISABLED, and the dashboard is health-gated), and
• adds one coherent slice toward the larger feature.

Track A — the feature (18 stacked PRs): foundations (log sanitizer → log bridge → OTel bootstrap → metrics → tracing) → deploy (compose/collector, Parca/eBPF) → query gateway (dtos → core → logs / metrics / traces / profiles each its own PR) → UI (shell → logs / metrics / traces / profiles panels).

Track B — logging & a security fix (8 PRs, independent, off main): human-readable lifecycle/error logging across engine, web layer, dashboard resources, workflow-core storage, the microservices, common/auth, common/dao; plus a fix to stop logging JWTs/headers/bodies in access-control.

Procedure

• Each PR is tracked as a sub-issue of Enhancement Request: Modern Observability for Texera #4070 ([Observability] pr1 — LogSanitizer + OpenTelemetry dependency baseline #5327–[Observability] pr18 — profiles panel + flame chart #5352) and maps 1:1 to a branch.
• Track A PRs are stacked (each opens against the previous); Track B PRs target main directly and can merge in any order.
• Suggested land order: security fix + logging PRs (parallel) → the feature stack in numeric order. Shortest path to a first shippable signal (logs end-to-end): bootstrap → gateway-core → gateway-logs → ui-shell → ui-logs.
• Per-signal verification: backend via sbt <module>/compile (+ specs), frontend via ng build.

Questions for reviewers

1. Is the two-track split + per-signal gateway/UI granularity the right reviewing unit, or would you prefer coarser PRs?
2. Should the codebase-wide logging track live under Enhancement Request: Modern Observability for Texera #4070, or as its own umbrella issue?
3. Any concerns with the stacked-PR procedure for the feature track?

[Ma77Ball]

@Yicong-Huang @chenlica @zuozhiw, any opinions on this path forward for raising the PRs for the observability task?

[Ma77Ball]

@zuozhiw Sounds good, I'll work on making each PR more self-contained going forward.

To answer your other question: the goal is to build our own UI and replace the LGTMP stack components with Apache-license-compatible alternatives. I've been working on this for a couple of weeks now, and it's starting to come together nicely. I also have branches in progress that expand logging coverage across the platform.

[Ma77Ball]

Here is what the design looks like now, but it can be edited based on input.

[Ma77Ball]

Note

I plan to also move the tab under the admin panel

[zuozhiw]

OK, I'm a bit suspious on building our own FE, because searching and visualization do requires many work. But it's now AI era and we vibe code everything so it's fine.

[Yicong-Huang]

I like the idea of building our own UI, more customizable.

[Yicong-Huang]

Hi @Ma77Ball, I don't have time to verify the project details yet, just a comment on opening issues and PRs related to a big topic. It will be good to have an umbrella issue (for example #4070) and on that issue to describe your plan. Then you can open multiple sub issues, each as a todo. The sub issues can be in parallel or in sequential to be completed.

For PRs though, if you find yourself stacking PRs (meaning that PR B depends on PR A, and PR C depends on PR B), then I suggest you open them one by one to make review easier:

• reviewer's attention needs to focus on incremental diff between the stacked PRs, and because github limitation, all PRs are against main branch, then PR B will contain diffs from PR A, and PR C will contain diffs from both A and B. It is hard for reviewers to check.
• for author, you, to resolve comments from PR A, you will likely need to push code and change the branch. Then you need to rebase or merge PR B and C, that's a lot of workload for author as well.

So my point is, you can open many issues (todos), but for PRs, it's better to open one at a time to get reviewed. If you have something that has no dependency and can be done/reviewed in parallel, sure you can open multiple PRs for those kind of parallel tasks.

I see you had more than 8(?) stacked PRs opening right now, and as a potential reviewer I am a bit lost where to start. for you, you need to update the other 7 as the first PR evolves, that's a lot of waste of time.