Skip to content

Commit 6d74d99

Browse files
frankmcsherryclaude
andcommitted
compute: make per-column compression flag replica-scoped
Addresses review feedback on #37111: scope `enable_arrangement_column_compression_alpha` to the replica (`ParameterScope::Replica`) so it can carry per-replica / per-size-family overrides, like the other replica-local physical flags (lgalloc, persist pager, column-paged batcher). This also replaces the manual capture chain the flag previously borrowed from dictionary compression — InstanceConfig/ReplicaConfig fields, the controller's per-replica capture, and the `handle_create_instance` store — with the replica-scoped idiom: the per-replica-resolved value arrives in the replica's `worker_config`, and `apply_worker_config` mirrors it into the process-global `mz_row_spine::COLUMN_COMPRESSION` flag (next to the lgalloc-region store). The controller-level capture read only the environment value and would have missed per-replica overrides. Re-applying on each config tick is safe: every batch records its own codec, so a flip only affects batches sealed afterwards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent d9f7564 commit 6d74d99

80 files changed

Lines changed: 134008 additions & 26 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.agents/skills/mz-demo-data

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../misc/demo-data

FEATURE_FLAG_AUDIT.md

Lines changed: 283 additions & 0 deletions
Large diffs are not rendered by default.

misc/build-time/baseline-run.log

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
[1/3] Clean dev build with --timings: mz-environmentd
2+
(this is the long step; ~5-15 min)
3+
Compiling mz-service v0.0.0 (/Users/frankmcsherry/Projects/materialize/src/service)
4+
Compiling mz-orchestrator-tracing v0.0.0 (/Users/frankmcsherry/Projects/materialize/src/orchestrator-tracing)
5+
Compiling mz-orchestratord v26.26.0-dev.0 (/Users/frankmcsherry/Projects/materialize/src/orchestratord)
6+
Compiling mz-compute-types v0.0.0 (/Users/frankmcsherry/Projects/materialize/src/compute-types)
7+
Compiling mz-adapter-types v0.0.0 (/Users/frankmcsherry/Projects/materialize/src/adapter-types)
8+
Compiling mz-timestamp-oracle v0.0.0 (/Users/frankmcsherry/Projects/materialize/src/timestamp-oracle)
9+
Compiling mz-controller-types v0.0.0 (/Users/frankmcsherry/Projects/materialize/src/controller-types)
10+
Compiling mz-dyncfgs v0.0.0 (/Users/frankmcsherry/Projects/materialize/src/dyncfgs)
11+
Compiling mz-storage-client v0.0.0 (/Users/frankmcsherry/Projects/materialize/src/storage-client)
12+
Compiling mz-sql v0.0.0 (/Users/frankmcsherry/Projects/materialize/src/sql)
13+
Compiling mz-storage-controller v0.0.0 (/Users/frankmcsherry/Projects/materialize/src/storage-controller)
14+
Compiling mz-compute-client v0.0.0 (/Users/frankmcsherry/Projects/materialize/src/compute-client)
15+
Compiling mz-controller v0.0.0 (/Users/frankmcsherry/Projects/materialize/src/controller)
16+
Compiling mz-transform v0.0.0 (/Users/frankmcsherry/Projects/materialize/src/transform)
17+
Compiling mz-catalog v0.0.0 (/Users/frankmcsherry/Projects/materialize/src/catalog)
18+
Compiling mz-adapter v0.0.0 (/Users/frankmcsherry/Projects/materialize/src/adapter)
19+
Compiling mz-authenticator v0.1.0 (/Users/frankmcsherry/Projects/materialize/src/authenticator)
20+
Compiling mz-pgwire v0.0.0 (/Users/frankmcsherry/Projects/materialize/src/pgwire)
21+
Timing report saved to /Users/frankmcsherry/Projects/materialize/target/cargo-timings/cargo-timing-20260521T150152024Z-f2742f705a47b692.html
22+
Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 52s
23+
clean build: 173s
24+
archived: /Users/frankmcsherry/Projects/materialize/misc/build-time/cargo-timing.html
25+
extracted 1130 units to /Users/frankmcsherry/Projects/materialize/misc/build-time/units.json
26+
[2/3] Incremental rebuild probes
27+
probing: src/repr/src/scalar.rs
28+
19s, 1012 units rebuilt
29+
probing: src/adapter/src/coord.rs
30+
24s, 1012 units rebuilt
31+
probing: src/sql/src/plan/statement/ddl.rs
32+
14s, 1012 units rebuilt
33+
probing: src/compute/src/render.rs
34+
14s, 1012 units rebuilt
35+
[3/3] Rendering report
36+
Wrote /Users/frankmcsherry/Projects/materialize/misc/build-time/baseline.md
37+
Done. See /Users/frankmcsherry/Projects/materialize/misc/build-time/baseline.md

misc/build-time/baseline.md

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
# Build-time baseline
2+
3+
Target package: **mz-environmentd**. Profile: **dev**. Total clean build: **173s** wall-clock.
4+
5+
Open `cargo-timing.html` next to this file for the full Gantt chart. This report summarizes its embedded unit data.
6+
7+
## Critical path (longest dependency chain)
8+
9+
Walked backward from the latest-finishing unit. The sum of these durations is the build's lower bound regardless of parallelism. Shortening any of these units shortens wall-clock by the same amount. Other units (not on the path) only matter if they push past their parent's finish time.
10+
11+
| # | Crate (unit) | Start (s) | Frontend (s) | Codegen (s) | Total (s) |
12+
|---:|---|---:|---:|---:|---:|
13+
| 1 | mz-environmentd (todo) | 153.4 | 10.7 | 6.9 | 17.7 |
14+
| 2 | mz-environmentd (todo) | 171.1 | 0.0 | 0.0 | 1.5 |
15+
16+
**Critical-path total: 19.2s** (clean wall-clock: 173s; the difference is sink-tail / link / serialization overhead).
17+
18+
## Top 30 units by total compile time (aggregated across modes)
19+
20+
`frontend` is parse+typecheck+borrow-check+MIR (blocks downstream crates from starting, since dependents need rmeta). `codegen` is LLVM IR + machine code (parallelizes well; mostly hurts total CPU-seconds, less so wall-clock).
21+
22+
| Crate | Builds | Total (s) | Frontend (s) | Codegen (s) | Codegen-dominated? |
23+
|---|---:|---:|---:|---:|---|
24+
| protobuf-src | 3 | 56.8 | 0.0 | 0.0 | no |
25+
| openssl-sys | 3 | 56.7 | 0.1 | 0.0 | no |
26+
| mz-catalog-protos | 3 | 38.6 | 26.4 | 11.5 | no |
27+
| aws-sdk-s3 | 1 | 34.5 | 27.4 | 7.1 | no |
28+
| mz-adapter | 1 | 31.0 | 12.6 | 18.4 | mixed |
29+
| aws-lc-sys | 3 | 29.4 | 0.5 | 0.1 | no |
30+
| timely | 1 | 22.6 | 4.5 | 18.2 | yes |
31+
| k8s-openapi | 3 | 21.4 | 14.3 | 6.5 | no |
32+
| mz-catalog | 1 | 20.5 | 10.7 | 9.8 | mixed |
33+
| mz-environmentd | 4 | 20.3 | 10.7 | 6.9 | no |
34+
| iceberg | 1 | 19.3 | 12.5 | 6.8 | no |
35+
| parquet | 1 | 17.2 | 5.1 | 12.1 | yes |
36+
| mz-expr | 3 | 14.8 | 8.1 | 2.4 | no |
37+
| mz-persist-client | 3 | 14.1 | 5.3 | 4.5 | mixed |
38+
| mz-sql | 1 | 13.3 | 7.8 | 5.5 | mixed |
39+
| zstd-sys | 3 | 12.6 | 0.0 | 0.0 | no |
40+
| insta | 1 | 12.3 | 1.3 | 11.0 | yes |
41+
| mz-transform | 1 | 11.9 | 3.2 | 8.7 | yes |
42+
| h2 | 3 | 10.0 | 7.8 | 2.2 | no |
43+
| mz-txn-wal | 3 | 9.8 | 2.2 | 3.4 | yes |
44+
| mz-storage-controller | 1 | 9.6 | 3.5 | 6.1 | yes |
45+
| mz-repr | 3 | 8.4 | 2.5 | 1.6 | no |
46+
| aws-sdk-secretsmanager | 1 | 8.4 | 5.6 | 2.8 | no |
47+
| opendal | 1 | 8.3 | 4.4 | 4.0 | mixed |
48+
| mz-pgwire | 1 | 8.1 | 7.8 | 0.3 | no |
49+
| mz-storage-types | 3 | 8.0 | 2.7 | 1.0 | no |
50+
| mz-storage-client | 1 | 7.9 | 4.3 | 3.6 | mixed |
51+
| tokio | 2 | 7.6 | 5.2 | 2.4 | no |
52+
| reqwest | 2 | 7.3 | 2.9 | 4.4 | yes |
53+
| protobuf-native | 3 | 7.2 | 0.1 | 0.1 | no |
54+
55+
## Top frontend-heavy crates (block downstream the most)
56+
57+
These are the crates where investing in frontend reduction (fewer generics in public API, fewer derives, lighter macros, smaller trait surfaces) directly cuts wall-clock for everything downstream.
58+
59+
| Crate | Frontend (s) | Codegen (s) | Total (s) |
60+
|---|---:|---:|---:|
61+
| aws-sdk-s3 | 27.4 | 7.1 | 34.5 |
62+
| mz-catalog-protos | 26.4 | 11.5 | 38.6 |
63+
| k8s-openapi | 14.3 | 6.5 | 21.4 |
64+
| mz-adapter | 12.6 | 18.4 | 31.0 |
65+
| iceberg | 12.5 | 6.8 | 19.3 |
66+
| mz-environmentd | 10.7 | 6.9 | 20.3 |
67+
| mz-catalog | 10.7 | 9.8 | 20.5 |
68+
| mz-expr | 8.1 | 2.4 | 14.8 |
69+
| mz-sql | 7.8 | 5.5 | 13.3 |
70+
| h2 | 7.8 | 2.2 | 10.0 |
71+
| mz-pgwire | 7.8 | 0.3 | 8.1 |
72+
| aws-sdk-secretsmanager | 5.6 | 2.8 | 8.4 |
73+
| mz-persist-client | 5.3 | 4.5 | 14.1 |
74+
| tokio | 5.2 | 2.4 | 7.6 |
75+
| parquet | 5.1 | 12.1 | 17.2 |
76+
77+
## Top codegen-heavy crates (llvm-lines / monomorphization targets)
78+
79+
These are where llvm-lines work and generic-instantiation pruning pay off. They mostly hurt total CPU-seconds; wall-clock impact depends on whether they're on the critical path (check the path table above).
80+
81+
| Crate | Codegen (s) | Frontend (s) | Total (s) |
82+
|---|---:|---:|---:|
83+
| mz-adapter | 18.4 | 12.6 | 31.0 |
84+
| timely | 18.2 | 4.5 | 22.6 |
85+
| parquet | 12.1 | 5.1 | 17.2 |
86+
| mz-catalog-protos | 11.5 | 26.4 | 38.6 |
87+
| insta | 11.0 | 1.3 | 12.3 |
88+
| mz-catalog | 9.8 | 10.7 | 20.5 |
89+
| mz-transform | 8.7 | 3.2 | 11.9 |
90+
| aws-sdk-s3 | 7.1 | 27.4 | 34.5 |
91+
| mz-environmentd | 6.9 | 10.7 | 20.3 |
92+
| iceberg | 6.8 | 12.5 | 19.3 |
93+
| k8s-openapi | 6.5 | 14.3 | 21.4 |
94+
| arrow-ord | 6.4 | 0.6 | 7.0 |
95+
| backtrace | 6.2 | 0.2 | 6.4 |
96+
| arrow-cast | 6.1 | 0.9 | 7.1 |
97+
| mz-storage-controller | 6.1 | 3.5 | 9.6 |
98+
99+
## Incremental rebuild probes
100+
101+
Each probe: append a no-op marker to the file, run `cargo build -p environmentd`, count rebuilt units. Probes are ordered by file. `seconds` is wall-clock incremental rebuild. `units` is the number of compiler artifacts produced (proxy for downstream closure size).
102+
103+
| Edited file | Rebuild (s) | Units rebuilt |
104+
|---|---:|---:|
105+
| `src/repr/src/scalar.rs` | 21 | 39 |
106+
| `src/adapter/src/coord.rs` | 21 | 39 |
107+
| `src/sql/src/plan/statement/ddl.rs` | 17 | 9 |
108+
| `src/compute/src/render.rs` | 16 | 9 |
109+
110+
## How to read this
111+
112+
- **Codegen-dominated units (column = yes)** are candidates for llvm-lines / generic-monomorphization work. They burn CPU but mostly parallelize, so they hurt total CPU-seconds more than wall-clock.
113+
- **High-rmeta units** are critical-path. Reducing their frontend cost (less generics in public API, fewer derives, lighter macros) directly cuts wall-clock.
114+
- **High units-rebuilt on incremental** means that file is upstream of a lot of code. Decoupling work on those files / crates pays off in incremental wall-clock immediately.
115+
116+
## Limitations
117+
118+
- `dev` profile only. Release builds have very different shape (LTO, more codegen).
119+
- One sample per probe; rerun a few times if a number looks surprising. Disk cache / sccache state matters.
120+
- The JSON doesn't expose Cargo's true critical path. `rmeta` is a good proxy but not authoritative. Open the HTML for the real picture.
121+
- Incremental probes assume no in-flight uncommitted changes to the probed files (the script reverts its own marker, but a dirty file may already have non-trivial rebuild cost baked in).

0 commit comments

Comments
 (0)