refactor(ingestion): rename capability summary to connector registry and split into package-based files#16106
Conversation
…and split into package-based files Renames the capability summary feature to connector registry across the codebase and refactors the implementation to use package-based organization: - Renamed capability_summary.py → connector_registry.py - Renamed capabilitySummary.ts → connectorRegistry.ts - Renamed gradle task capabilitySummary → connectorRegistry - Generation script now groups plugins by top-level package name from classname - Each package generates its own JSON file (e.g., datahub.json) in connector_registry/ - Added manifest.json listing all available packages for frontend discovery - Removed generated_at timestamp field (keeping generated_by, added package field) - Documentation generator loads and merges all package files from directory - Frontend fetches manifest and loads all packages in parallel - Updated build tasks and CI workflows to work with directory structure
github-actions
bot
added
ingestion
|
Linear: ING-1518 |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
📝 [actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:11:32: Double quote to prevent globbing and word splitting [shellcheck]
There was a problem hiding this comment.
📝 [actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:17:31: Double quote to prevent globbing and word splitting [shellcheck]
There was a problem hiding this comment.
📝 [actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:20:32: Double quote to prevent globbing and word splitting [shellcheck]
There was a problem hiding this comment.
📝 [actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:2:29: Double quote to prevent globbing and word splitting [shellcheck]
There was a problem hiding this comment.
📝 [actionlint] reported by reviewdog 🐶
shellcheck reported issue in this script: SC2086:info:8:31: Double quote to prevent globbing and word splitting [shellcheck]
datahub-cyborg
bot
added
the
needs-review
Codecov Report❌ Patch coverage is
❌ Your patch status has failed because the patch coverage (35.00%) is below the target coverage (75.00%). You can increase the patch coverage or adjust the target coverage. 📢 Thoughts on this report? Let us know! |
|
✅ Meticulous spotted 0 visual differences across 969 screens tested: view results. Meticulous evaluated ~9 hours of user flows against your PR. Expected differences? Click here. Last updated for commit d51790c. This comment will update as new commits are pushed. |
Bundle ReportChanges will increase total bundle size by 576 bytes (0.0%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: datahub-react-web-esmAssets Changed:
Files in
|
datahub-cyborg
bot
added
pending-submitter-merge
and removed
needs-review
…and split into package-based files (datahub-project#16106)
…and split into package-based files (datahub-project#16106)
2 participants
Summary
This PR refactors the capability summary feature into a more scalable connector registry system with package-based organization:
Key Changes
capability_summary→connector_registryacross codebaseconnector_registry/datahub.json- connectors from datahub.* packageconnector_registry/manifest.json- lists all available packages for discoveryscripts/capability_summary.py→scripts/connector_registry.pygenerated_attimestamp (keepinggenerated_by, addedpackagefield)Files Changed
.github/workflows/metadata-ingestion.yml- Updated CI task namedatahub-web-react/build.gradle- Updated copy task to handle directorydatahub-web-react/src/app/ingestV2/shared/capabilitySummary.ts→connectorRegistry.tsdatahub-web-react/src/app/ingestV2/shared/hooks/useCapabilitySummary.ts- Parallel loading logicmetadata-ingestion/build.gradle- Updated gradle task configurationmetadata-ingestion/scripts/capability_summary.py→connector_registry.pymetadata-ingestion/scripts/docgen.py- Loads from directory instead of single filemetadata-ingestion/src/datahub/ingestion/autogenerated/connector_registry/- New directory structureBenefits
Test plan
metadata-ingestionworkflow)./gradlew :metadata-ingestion:connectorRegistryto generate new filesconnector_registry/directory containsmanifest.jsonanddatahub.json./gradlew :metadata-ingestion:docGento verify documentation generation works🤖 Generated with Claude Code