Unified mixs-legacy-diff tool: v4+ schema comparison with CI tests

[turbomam]

Summary

• Unified mixs-legacy-diff CLI replaces both the old diff-releases script (2408 lines, LinkML-only) and the original format-agnostic diff tool into a single entry point
• Supports v4 Excel (.xls), v5 Excel (.xlsx), and v6+ LinkML YAML — any combination
• Scoped to v4+ only; pre-v4 formats (2009–2011 Word/Excel) removed as unstable — cataloged in assets/terminology_by_version.yaml for future reimplementation
• Mapping hints (expected renames, splits, merges, deletions) unified into YAML format grouped by entity type

What changed

Area	Before	After
Diff tools	Two separate tools with incompatible mapping formats	One unified mixs-legacy-diff CLI
Mapping hints	5 TSV files (flat)	Single mapping_config.yaml (grouped by slots/classes/enums/subsets)
Format support	v4, v5, v6+, plus unstable pre-v4	v4, v5, v6+ (stable only)
CI tests	None	30 tests on every PR touching diff code
Release workflow	Hardcoded path that 404'd on pre-v6.2 tags	Full GitHub spec inputs with correct paths
Test data	Required local mixs-legacy checkout	Auto-downloads from public GitHub repo

Release workflow change

The create-release-pr.yaml workflow inputs changed from bare refs to full GitHub specs:

Before (broken for pre-v6.2 tags):

diff_old: mixs6.0.0        # → constructs src/mixs/schema/mixs.yaml → 404!
diff_new: main

After (explicit paths, works for any tag):

diff_old: GenomicsStandardsConsortium/[email protected]:src/mixs/schema/mixs.yaml
diff_new: GenomicsStandardsConsortium/mixs@main:src/mixs/schema/mixs.yaml

To compare against mixs6.0.0, use: GenomicsStandardsConsortium/[email protected]:model/schema/mixs.yaml

The key insight: the schema path changed from model/schema/mixs.yaml (tags mixs6.0.0 through mixs6.1.1) to src/mixs/schema/mixs.yaml (tags v6.2.0 onward). The old workflow hardcoded the newer path, so comparing against pre-v6.2 tags would silently fail.

Usage

# Install dependencies
poetry install --with legacy-diff

# v4 → v5
mixs-legacy-diff --old ../mixs-legacy/mixs4/MIxS_210514.xls \
  --new ../mixs-legacy/mixs5/mixs_v5.xlsx

# v5 → current LinkML
mixs-legacy-diff --old ../mixs-legacy/mixs5/mixs_v5.xlsx \
  --new src/mixs/schema/mixs.yaml

# Tagged release → tagged release (fetches from GitHub)
mixs-legacy-diff \
  --old "GenomicsStandardsConsortium/[email protected]:src/mixs/schema/mixs.yaml" \
  --new "GenomicsStandardsConsortium/[email protected]:src/mixs/schema/mixs.yaml"

# With mapping hints for known renames/deletions
mixs-legacy-diff --old ... --new ... \
  --mappings-dir assets/between_diff_mappings/6_to_pre_7

Test plan

• 30 automated tests (v4→v5, v5→v6, v6→v6, format detection, models)
• Interactive: v4 xls → v5 xlsx (343 → 600 terms, 328 shared)
• Interactive: v5 xlsx → v6 LinkML (600 → 1059 terms, 560 shared)
• Interactive: v4 xls → v6 LinkML (343 → 1059 terms, 306 shared)
• Interactive: GitHub tagged v6.2.0 → v6.2.3
• Interactive: GitHub mixs6.0.0 → main with mapping config (804 → 1099 terms)
• Release workflow simulation with new inputs

Closes #1114

🤖 Generated with Claude Code

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://GenomicsStandardsConsortium.github.io/mixs/pr-preview/pr-1115/
Built to branch gh-pages at 2026-03-10 23:02 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[Copilot]

Pull request overview

This pull request adds a new mixs-legacy-diff CLI tool for comparing MIxS schemas across different historical formats and versions. The tool enables comparisons between Excel (.xls, .xlsx), Word (.docx), XSD (.xsd), and LinkML YAML formats through a configuration-driven approach using YAML profiles.

Changes:

• Adds new src/mixs/diff/ module with readers for Excel, Word, XSD, and LinkML formats
• Implements configuration-based Excel parsing using YAML profiles in assets/legacy_format_profiles/
• Provides normalized schema representation through NormalizedTerm and NormalizedSchema classes for cross-format comparison
• Adds optional dependency group legacy-diff containing openpyxl, xlrd, and python-docx

Reviewed changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
pyproject.toml	Adds optional legacy-diff dependency group and CLI entry point
poetry.lock	Updates lock file with new dependencies for legacy format support
src/mixs/diff/__init__.py	Module initialization with clear warnings about internal-only data models
src/mixs/diff/models.py	Defines normalized data structures for cross-format schema comparison
src/mixs/diff/cli.py	CLI entry point with argument validation and error handling
src/mixs/diff/comparison.py	Schema comparison logic with support for term name mappings
src/mixs/diff/output.py	YAML and text output formatting for comparison results
src/mixs/diff/readers/base.py	Abstract base reader with format detection
src/mixs/diff/readers/excel_reader.py	Profile-driven Excel (.xls, .xlsx) reader
src/mixs/diff/readers/linkml_reader.py	LinkML YAML reader with GitHub spec support
src/mixs/diff/readers/word_reader.py	Word document (.docx) table parser
src/mixs/diff/readers/xsd_reader.py	XML Schema (.xsd) parser
assets/legacy_format_profiles/*.yaml	Configuration files for Excel format variations

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[turbomam]

Rebase & cross-reference note (2026-03-10)

This PR was rebased onto current main along with #1090, #1110, and #1130.

Merge order: this PR should merge FIRST. It changes the release workflow parameters (unified mixs-legacy-diff tool) and replaces the diff result files that both #1110 and #1090 reference. Specifically:

• Document release workflow in edit_workflow.md #1090 (release workflow docs) documents workflow parameters whose format changes in this PR — will need updating after this merges.
• Add curated schema diff documentation for v6.0.0 to main #1110 (schema diff docs) references assets/diff_results/schema_comparison_results.yaml, which this PR deletes — will need path updates after this merges.
• Test MIxS build against linkml main + TSV list normalization demo #1130 (LinkML main compat test) is independent.

The test-legacy-diff CI workflow is now running for the first time on this branch.

[Copilot]

Pull request overview

Copilot reviewed 39 out of 45 changed files in this pull request and generated 9 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Same issue in _process_packages_sheet_xls: if req_idx and row[req_idx] will skip requirement values when the requirement column is at index 0. Prefer req_idx is not None (and bounds check) to avoid dropping data.

[turbomam]

Valid bug. On line 395 (xlsx) and line 470 (xls), if req_idx and row[req_idx] will incorrectly evaluate to falsy when req_idx == 0 (i.e., when the requirement column is the first column). Should be if req_idx is not None and row[req_idx]. Will fix.

[Copilot]

The --old-format / --new-format options are computed (old_fmt/new_fmt) but not actually used to choose the reader: get_reader(old, ...) always re-detects from the path. As a result, forcing formats has no effect beyond logging. Wire the forced formats into reader selection (e.g., pass a format_override into get_reader() or select ExcelReader/LinkMLReader directly based on the option).

[turbomam]

Valid observation. The --old-format and --new-format options compute old_fmt/new_fmt (lines 196-197) but then get_reader() at lines 201-202 uses the raw path for auto-detection, ignoring the format override. The format variables are only used for the log message at line 199. Will fix — get_reader() should accept a format override parameter, or we should select the reader based on the resolved format.

[Copilot]

Unused import: os is imported but not used in this test fixture module. Remove it to avoid lint noise and keep dependencies clear.

[turbomam]

Valid — import os on line 7 of conftest.py is unused. Will remove.

[Copilot]

Unused import: os is imported but not referenced anywhere in this module. Removing it avoids lint noise and makes the module's dependencies clearer.

[turbomam]

Valid — import os on line 7 of linkml_reader.py is unused (was needed by the old temp-file approach that used os.unlink(), which was refactored to use shutil.rmtree() instead). Will remove.

[Copilot]

In the workflow triggers, pull_request.paths includes assets/legacy_format_profiles/** and assets/between_diff_mappings/**, but the push trigger only watches src/mixs/diff/** and tests. That means pushes to main that only change mapping/profile assets won't run this job. Consider mirroring the asset paths under push.paths as well for consistent coverage.

Suggested change

	- "tests/conftest.py"
	- "tests/conftest.py"
	- "assets/legacy_format_profiles/**"
	- "assets/between_diff_mappings/**"

[turbomam]

Valid — the push trigger (lines 12-17) is missing the asset paths that are present in the pull_request trigger (assets/legacy_format_profiles/** and assets/between_diff_mappings/**). Changes to format profiles or mapping configs pushed directly to main would not trigger the workflow. Will fix.

[Copilot]

In _process_packages_sheet_xlsx, requirement = ... if req_idx and row[req_idx] else "" treats column index 0 as falsy, so a valid requirement column at index 0 would be ignored. Use an explicit req_idx is not None check (and bounds check) instead of truthiness.

[turbomam]

Same bug as comment #12 — if req_idx and row[req_idx] on line 395 (xlsx path) has the same falsy-zero problem. Will fix both together using req_idx is not None.

[Copilot]

load_mapping_config() doesn't parse the merge structure used in assets/between_diff_mappings/4_to_5/mapping_config.yaml (merges: <old_name>: {target: <new_name>, ...}). The current code expects merges to be keyed by the new name with a list/dict of source names, so merges from that config will silently become empty. Update the parser to accept the {target: ...} form (or change the mapping YAML structure to match the parser) so merges are actually applied.

[turbomam]

Looking at the current code in load_mapping_config() (comparison.py lines 596-671), it does handle merges — see lines 641-648 which parse the merges key from the YAML config, supporting both list and dict formats with optional descriptions. The merge structure is parsed into config.merges (Dict[str, List[str]]) and config.merge_descriptions. If the comment is about a specific merge YAML structure that isn't being parsed correctly, could you share an example of the config that fails?

[Copilot]

The tests write to hard-coded /tmp/test_legacy_diff_* directories. This can cause cross-run conflicts (parallel test runs) and makes the suite less portable (non-POSIX environments). Prefer tempfile.TemporaryDirectory() (or pytest's tmp_path) to allocate unique per-run output dirs.

[turbomam]

Valid concern. The tests use hardcoded Path("/tmp/test_legacy_diff_*") paths (lines 166, 224, 271). Since these are unittest.TestCase classes (not pytest functions), the tmp_path fixture isn't available directly. However, the tests do clean up via tearDownClass with shutil.rmtree(). The pragmatic fix would be to switch to tempfile.mkdtemp() for cross-platform safety — /tmp doesn't exist on Windows. Will address.

[Copilot]

The test data download URLs are pinned to the mixs-legacy repo's master branch. This makes tests non-deterministic if the upstream files change or if the default branch is renamed. Consider pinning to an immutable ref (tag/commit SHA) or mirroring the test fixtures into this repo to keep CI stable.