feat(ci): smart upstream-monitor with release intelligence

[polaz]

Summary

Current upstream-monitor.yml is a dumb sync — it counts commits behind upstream and tries to merge. It provides zero intelligence about what changed upstream. We need feature-level visibility to make informed merge decisions.

Current behavior

Feature	Status
Detect new upstream commits	yes
Auto-merge if clean	yes
Create conflict issue	yes
Detect merged feature branches	yes
Parse upstream releases/tags	no
Show what changed (features/fixes/breaking)	no
Link to upstream issues/PRs	no
Detect breaking changes	no
Categorize by conventional commits	no

Proposed behavior

1. Release detection

Check upstream tags/releases, not just commits. When a new release appears:

🏷️ New upstream release: lsm-tree v3.2.0 (was v3.1.2)

2. Commit categorization

Parse git log between our last sync point and upstream HEAD using conventional commit format:

## Upstream changes since last sync

### ⚠️ Breaking (1)
- `feat!: remove deprecated flush_sync()` — fjall-rs/lsm-tree#290

### ✨ Features (3)
- `feat: add custom merge operator support` — fjall-rs/lsm-tree#280
- `feat(bloom): partitioned bloom filters` — fjall-rs/lsm-tree#275
- `feat: io_uring backend` — fjall-rs/lsm-tree#270

### 🐛 Fixes (2)
- `fix: race condition in concurrent compaction` — fjall-rs/lsm-tree#285
- `fix(vlog): corrupted blob header on crash` — fjall-rs/lsm-tree#282

### ⚡ Performance (1)
- `perf: vectorized block decoding` — fjall-rs/lsm-tree#278

### 📝 Other (4)
- `docs: update MSRV to 1.91` — fjall-rs/lsm-tree#288
- `test: add property tests for range scan` — fjall-rs/lsm-tree#286
- `chore(deps): bump lz4_flex to 0.14` — fjall-rs/lsm-tree#284
- `ci: add aarch64 cross-compilation` — fjall-rs/lsm-tree#283

3. Issue/PR linking

Extract #NNN references from commit messages and link them:

git log origin/main..upstream/main --format="%s" | grep -oE '#[0-9]+' | sort -u

For each reference, fetch title from upstream repo API.

4. Fork overlap detection

Check if any upstream changes touch files that our fork patches have modified:

# Files we've changed vs upstream
OUR_FILES=$(git diff origin/main...upstream/main --name-only)
FORK_FILES=$(git log --all --format="" --name-only -- src/ | sort -u)
OVERLAP=$(comm -12 <(echo "$OUR_FILES" | sort) <(echo "$FORK_FILES" | sort))

If overlap exists → flag as "needs manual review" even if merge is clean (semantic conflicts possible).

5. Smart PR/issue body

Instead of generic "N new commits", generate a structured body:

## Upstream Sync: v3.1.2 → v3.2.0

**Release:** [v3.2.0](https://github.com/fjall-rs/lsm-tree/releases/tag/v3.2.0)
**Commits:** 47 new commits since last sync
**Breaking changes:** 1 ⚠️

### Changes by category
[categorized list from step 2]

### Fork overlap
These upstream changes touch files our fork has modified:
- `src/compaction/worker.rs` — our merge operator patches may need adaptation
- `src/tree/mod.rs` — our prefix bloom integration

### Review checklist
- [ ] Breaking changes evaluated for fork impact
- [ ] Overlapping files reviewed for semantic conflicts
- [ ] Fork-specific tests pass with upstream changes
- [ ] lsm-tree version in fjall Cargo.toml updated

Implementation

New workflow steps

- name: Analyze upstream changes
  id: analyze
  run: |
    # 1. Detect release
    LATEST_TAG=$(git tag -l --sort=-v:refname 'v*' | head -1)
    UPSTREAM_TAG=$(git ls-remote --tags upstream | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+$' | sort -V | tail -1)

    # 2. Categorize commits
    git log origin/main..upstream/main --format="%s" | while read msg; do
      case "$msg" in
        feat\!:*|*BREAKING*) echo "breaking: $msg" ;;
        feat:*|feat\(*) echo "feature: $msg" ;;
        fix:*|fix\(*) echo "fix: $msg" ;;
        perf:*|perf\(*) echo "perf: $msg" ;;
        *) echo "other: $msg" ;;
      esac
    done > /tmp/categorized.txt

    # 3. Extract issue references
    git log origin/main..upstream/main --format="%s %b" | grep -oE '#[0-9]+' | sort -un > /tmp/refs.txt

    # 4. Check fork overlap
    git diff --name-only origin/main..upstream/main > /tmp/upstream_files.txt
    # Compare with our fork-specific changes (commits not in upstream)
    git diff --name-only upstream/main..origin/main > /tmp/fork_files.txt
    comm -12 <(sort /tmp/upstream_files.txt) <(sort /tmp/fork_files.txt) > /tmp/overlap.txt

Backward compatible

• Keep existing auto-merge/conflict behavior
• Add intelligence as additional context in PR body / issue body
• No change to schedule (Mon/Thu 8am UTC)

Acceptance criteria

• Detects new upstream release tags (not just commits)
• Categorizes commits by conventional commit type
• Extracts and resolves upstream issue/PR references (title + URL)
• Detects fork file overlap (potential semantic conflicts)
• Generates structured PR body with all of the above
• Generates structured issue body (conflict case) with all of the above
• Breaking changes highlighted prominently
• Backward compatible with current merge/conflict behavior

Related

• fjall will get the same upgrade (separate issue)
• gitlab-mcp needs upstream-monitor from scratch (separate issue)
• strongswan already has sync-upstream.yml — evaluate if it needs the same intelligence

Time estimate

1d — shared script + workflow updates for lsm-tree (fjall is copy+adapt)

---

Additional: Add .coderabbit.yaml configuration

Currently no .coderabbit.yaml exists — CodeRabbit uses defaults. Add a proper config to:

• Disable pauses (reviews should never be paused/skipped)
• Set assertive profile (systems code needs thorough review)
• Add path-specific instructions (Rust safety, tests, benchmarks)
• Enable knowledge base with our coding guidelines
• Disable poem (noise in review walkthrough)
• Enable related issues/PRs detection

Proposed .coderabbit.yaml

# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json
language: "en-US"
early_access: true
reviews:
  profile: "assertive"
  request_changes_workflow: false
  high_level_summary: true
  high_level_summary_placeholder: "@coderabbitai summary"
  poem: false
  review_status: true
  collapse_walkthrough: true
  commit_status: true
  assess_linked_issues: true
  related_issues: true
  related_prs: true
  suggested_labels: true
  auto_review:
    enabled: true
    auto_incremental_review: true
    drafts: false
    base_branches: ["main"]
    ignore_title_keywords: ["WIP", "DO NOT MERGE"]
  path_instructions:
    - path: "src/**/*.rs"
      instructions: |
        Review for memory safety, ownership correctness, and panic-free error handling.
        No unwrap() on I/O or user-input paths. Prefer Result<T, E> everywhere.
        No Box<dyn Any> as type bypass. No global mutable state (lazy_static Mutex<HashMap>).
        Check multi-instance safety: no in-memory-only mutable state that breaks with N replicas.
    - path: "tests/**"
      instructions: |
        Verify test coverage of edge cases and error paths.
        No unwrap() on paths that test error handling.
        Tests must have descriptive names and comments explaining WHAT is being tested.
    - path: "benches/**"
      instructions: |
        Check benchmark methodology. Use criterion properly.
        Measure P99/P999 latency, not just throughput.
    - path: "src/compaction/**"
      instructions: |
        Compaction is crash-safety critical. Every state mutation must be atomic.
        Verify no partial writes can corrupt on-disk state.
    - path: "src/encryption.rs"
      instructions: |
        Security-critical code. Review for timing attacks, nonce reuse, key handling.
        RNG must be CSPRNG. No hardcoded keys or IVs.
  tools:
    shellcheck:
      enabled: true
    markdownlint:
      enabled: true
    github-checks:
      enabled: true
      timeout_ms: 120000
chat:
  auto_reply: true
knowledge_base:
  learnings:
    scope: auto
  issues:
    scope: auto
  pull_requests:
    scope: auto
  code_guidelines:
    enabled: true
    filePatterns:
      - ".github/copilot-instructions.md"
      - ".github/instructions/*.instructions.md"
issue_enrichment:
  labeling:
    auto_apply_labels: true
    labeling_instructions:
      - label: "bug"
        instructions: "Code defect, incorrect behavior, crash, data corruption, wrong results"
      - label: "enhancement"
        instructions: "New feature, new API, new capability not previously available"
      - label: "performance"
        instructions: "Optimization, reduced allocations, faster path, benchmark improvement"
      - label: "refactor"
        instructions: "Code restructuring without behavior change — renames, extractions, trait threading"
      - label: "test"
        instructions: "New tests, test infrastructure, test helpers, flaky test fixes"
      - label: "ci"
        instructions: "CI/CD workflows, GitHub Actions, benchmarks pipeline, release automation"
      - label: "comparator"
        instructions: "UserComparator threading, custom key ordering, lexicographic vs comparator-aware"
      - label: "compaction"
        instructions: "LSM compaction logic, leveled/tiered strategy, L0/L1/L2 overlap, compaction picker"
      - label: "crash-safety"
        instructions: "Crash recovery, fsync ordering, atomic writes, WAL correctness, data durability"
      - label: "encryption"
        instructions: "Block encryption, AES-GCM, key management, nonce handling"
      - label: "fs-trait"
        instructions: "Filesystem abstraction, Fs trait, io_uring, per-level routing, StdFs"
      - label: "upstream-candidate"
        instructions: "Fix or feature that could be contributed back to fjall-rs upstream"
      - label: "fork-only"
        instructions: "Feature specific to CoordiNode fork — range tombstones, merge operators, prefix bloom, V4 format"
      - label: "upstream-sync"
        instructions: "Automated upstream synchronization — merge conflicts, release tracking"

Key decisions

Setting	Value	Why
profile	assertive	Systems/storage code needs thorough review — "chill" misses too much
poem	false	Noise in walkthrough, wastes review tokens
auto_incremental_review	true	Review each push, not just first
drafts	false	Don't review drafts — they're WIP
path_filters	none needed	lsm-tree repo has no dirs to exclude (donor codebases live in coordinode workspace, not here)
path_instructions	Rust safety rules	Encode our engineering principles directly into review guidelines
early_access	true	Get new CodeRabbit features as they ship
knowledge_base.code_guidelines	copilot-instructions.md	CodeRabbit auto-reads this for review context
commit_status	true	Block merge until review completes
assess_linked_issues	true	Check if PR actually addresses linked issue

Checklist addition

• Create .coderabbit.yaml with config above
• Verify CodeRabbit picks it up on next PR (comment @coderabbitai configuration to confirm)

[polaz]

Related smart upstream-monitor issues across all forks:

• feat(ci): smart upstream-monitor with release intelligence coordinode-fjall#40 — same pattern (adapt from this)
• feat(ci): add upstream-monitor workflow with release intelligence gitlab-mcp#357 — monitor-only mode (too diverged for auto-merge)
• feat(ci): add release intelligence to sync-upstream workflow strongswan#17 — security-focused (CVE detection + patch impact)

Recommend implementing lsm-tree first, extracting shared analyze-upstream.sh script, then adapting for other repos.

[m13v]

This is a really well-thought-out proposal. We have a similar need in our CI/CD pipeline (Fazm, macOS app built with Codemagic) where we track upstream changes. A few things that helped us:

1. Conventional commit parsing is the right foundation. We parse commit messages with a simple regex (^(feat|fix|perf|docs|chore|ci|test)(!)?(\(.+\))?: (.+)) and group them. Works for 90% of well-maintained repos. For the other 10%, fall back to showing raw commit messages.

2. Fork overlap detection is critical. Your approach of comparing modified files between fork patches and upstream changes is exactly right. One addition: also check for changes to function signatures or public API types in overlapping files. A clean git merge can still break your code if upstream renamed a function parameter you depend on.

3. Release detection tip: Instead of polling tags, use GitHub's release webhook or the Atom feed at /{owner}/{repo}/releases.atom. The Atom feed is simpler if you are already running a cron-based monitor. We poll it every 4 hours in our CI pipeline.

4. Breaking change detection: Beyond the ! conventional commit marker, also scan for keywords like "BREAKING", "deprecated", "removed" in commit bodies and PR descriptions. Many projects do not use conventional commits consistently but do mention breaking changes in prose.

5. Actionable output matters: The issue template format you have is great. Consider also generating a machine-readable JSON summary alongside the markdown, so downstream automation can make merge/wait decisions without parsing markdown.

[m13v]

Our CI/CD pipeline config (Codemagic-based, handles release tagging, build, notarization, and auto-update publishing) is here if you want to see the release intelligence side in practice: https://github.com/m13v/fazm/blob/main/codemagic.yaml

It auto-triggers on version tags and publishes releases with changelogs.

[polaz]

Include #154 (dependabot) in this issue

Both are CI infrastructure tasks. #154 is trivial (enable dependabot + normalize schedule). Rolling it into the upstream-monitor work avoids a separate tiny PR.

Adding to scope:

• Enable Dependabot vulnerability alerts
• Normalize update schedule (Monday)
• Smart upstream-monitor with release intelligence (existing scope)

@polaz — confirm? If yes, close #154 with pointer here.