Skip to content

Storage introspection mirrored in compute#35863

Draft
frankmcsherry wants to merge 1 commit intoMaterializeInc:mainfrom
frankmcsherry:storage_introspection
Draft

Storage introspection mirrored in compute#35863
frankmcsherry wants to merge 1 commit intoMaterializeInc:mainfrom
frankmcsherry:storage_introspection

Conversation

@frankmcsherry
Copy link
Copy Markdown
Contributor

Events from storage's timely infra are captured and replayed in the compute logging dataflows, with identifiers bumped up by 2^48. They are pretty large at that point, but shouldn't collide with the compute identifiers. The nudged identifiers should include Both the timely operator ids, and also the dataflow identifiers (the lead coordinate of addresses).

Not much testing, other than observing the information flowing after creating an auction leaden source.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 3, 2026

Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone.

PR title guidelines

  • Use imperative mood: "Fix X" not "Fixed X" or "Fixes X"
  • Be specific: "Fix panic in catalog sync when controller restarts" not "Fix bug" or "Update catalog code"
  • Prefix with area if helpful: compute: , storage: , adapter: , sql:

Pre-merge checklist

  • The PR title is descriptive and will make sense in the git log.
  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).

@frankmcsherry frankmcsherry force-pushed the storage_introspection branch 2 times, most recently from ae77261 to 7baca63 Compare April 3, 2026 17:52
@frankmcsherry frankmcsherry force-pushed the storage_introspection branch from 7baca63 to ec858b9 Compare April 3, 2026 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant