Backport 2287 to release/0.4.z

[mrrajan]

Manual backport of PR #2287 to the release/0.4.z branch.

The automated backport failed due to merge conflicts. This PR contains the manually cherry-picked and conflict-resolved commits.

Changes

This backport includes 14 commits from PR #2287 and PR #2242

Resolved Conflicts

Merge conflicts were resolved in:

• Entity and migration module imports (sbom_group, sbom_group_assignment, sbom_license_expanded)
• License query service refactoring (union queries for SPDX and non-SBOM licenses)
• Database schema updates (MigratorExt trait, migration list)
• SBOM service imports (sbom_license_expanded, expanded_license)
• Recursion limit attributes for CI coverage builds
• Import cleanup in cyclonedx.rs

Custom Fix:

• IngestResult.id is Id enum on release/0.4.z, not String. Changed Uuid::parse_str(&result.id) to result.id.try_as_uid() in modules/fundamental/src/license/service/test.rs, modules/fundamental/src/sbom/service/test.rs
• Commit 49185eb rewrote licenses() but lost NOT EXISTS optimization from 3e9348d. Query reverted to inefficient LEFT JOIN + IS NULL (scans all rows). Applied NOT EXISTS pattern. Changed from sbom_package_license to sbom_license_expanded table on modules/fundamental/src/license/service/mod.rs

Summary

Fixes compatibility issues and restores performance optimizations lost during the backport of PR #2287 to release/0.4.z.

Summary by Sourcery

Normalize and pre-expand license expressions into dedicated tables, refactor license querying and filtering across services to use the new schema and COALESCE-based expressions, and add supporting migrations, entities, and tests while increasing recursion limits for large builds.

New Features:

• Introduce expanded_license and sbom_license_expanded tables and entities to store normalized expanded license texts and their SBOM/license mappings.
• Populate expanded license data at SBOM ingestion time for both SPDX and CycloneDX documents using a dedicated ingestion helper.
• Add comprehensive tests validating license union queries, COALESCE behavior, junction-table integrity, and license-based filtering across SBOM and PURL services.

Bug Fixes:

• Fix inconsistent behavior between SPDX and CycloneDX license handling by unifying them on a normalized, pre-expanded license representation.
• Eliminate duplicate expanded license entries by deduplicating on an MD5 hash of the expanded text.

Enhancements:

• Refactor license listing, SBOM, and PURL services to use pre-expanded license data via COALESCE expressions instead of on-the-fly expansion functions and complex CTEs.
• Simplify and optimize license-based filtering for SBOMs, packages, and PURLs by replacing custom CASE/CTE logic with unioned subqueries against normalized license data.
• Deprecate the licenses_ref_mapping field in SBOM models and OpenAPI schemas now that licenses are pre-expanded and the mapping is always empty.
• Extend the migration framework with a MigratorExt abstraction and register new license-related migrations.
• Increase recursion limits across several crates and test modules to support deeper macro expansions and large code paths.

Tests:

• Add new integration tests for license services and SBOM services covering COALESCE expansion, filtering, pagination, ordering, and ingestion-time population of expanded licenses.
• Add component type parsing/serialization tests for CycloneDX component types.

Summary by Sourcery

Backport license normalization and querying improvements to the 0.4.z release branch, including new expanded license storage, ingestion-time expansion, and performance-optimized license filtering across SBOM and PURL services.

New Features:

• Introduce expanded_license and sbom_license_expanded tables and entities to store normalized expanded license texts and their SBOM/license mappings.
• Populate expanded license data during SPDX and CycloneDX SBOM ingestion using a dedicated helper.

Enhancements:

• Refactor license listing and license-based filtering in license, SBOM, and PURL services to use pre-expanded license data with COALESCE expressions and NOT EXISTS optimizations.
• Deprecate licenses_ref_mapping in SBOM models and OpenAPI schemas now that licenses are pre-expanded and mappings are always empty.
• Add database indexes and migrations to improve license query performance and normalize expanded license storage.
• Increase recursion limits for several crates to support larger builds and deeper macro expansions.

Tests:

• Add extensive tests for license union queries, COALESCE behavior, junction-table integrity, pagination/counting, and license-based filtering across services.
• Add SBOM and PURL service tests validating COALESCE-based license aggregation and filtering, and coverage for edge cases like overlapping LicenseRef identifiers and NOT NULL filters.

[sourcery-ai]

Reviewer's Guide

Backports the normalized, pre-expanded license infrastructure and query refactor to release/0.4.z, replacing runtime expansion/CTE-based logic with dictionary/junction tables, COALESCE-based querying, and new migrations plus tests, while fixing backport-specific type and performance issues.

Sequence diagram for SBOM ingestion and expanded license usage

sequenceDiagram
    actor User
    participant API as SbomIngestionAPI
    participant Ingestor as SbomIngestor
    participant DB as Database
    participant LicenseSvc as LicenseService

    User->>API: POST /api/v1/sboms (SPDX or CycloneDX)
    API->>Ingestor: parse_and_store_sbom(document)
    Ingestor->>DB: insert sbom, packages, licenses, sbom_package_license
    Ingestor->>DB: populate_expanded_license(sbom_id)
    activate DB
    DB-->>DB: INSERT expanded texts into expanded_license
    DB-->>DB: INSERT mappings into sbom_license_expanded
    deactivate DB
    Ingestor-->>API: ingestion_result(sbom_id)
    API-->>User: 201 Created (sbom_id)

    User->>API: GET /api/v1/licenses?q=license:MIT
    API->>LicenseSvc: licenses(search, paginated, connection)
    LicenseSvc->>DB: SELECT via UNION
    DB-->>LicenseSvc: rows from expanded_license and license
    LicenseSvc-->>API: PaginatedResults<LicenseText>
    API-->>User: normalized license list

Loading

ER diagram for expanded_license and sbom_license_expanded normalization

erDiagram
    LICENSE {
        uuid id
        text text
    }

    SBOM {
        uuid sbom_id
    }

    SBOM_PACKAGE_LICENSE {
        uuid sbom_id
        uuid license_id
        uuid node_id
        text license_type
    }

    EXPANDED_LICENSE {
        uuid id
        text expanded_text
        text text_hash
    }

    SBOM_LICENSE_EXPANDED {
        uuid sbom_id
        uuid license_id
        uuid expanded_license_id
    }

    SBOM ||--o{ SBOM_PACKAGE_LICENSE : has_package_license
    LICENSE ||--o{ SBOM_PACKAGE_LICENSE : used_by_package

    SBOM ||--o{ SBOM_LICENSE_EXPANDED : has_expanded_license
    LICENSE ||--o{ SBOM_LICENSE_EXPANDED : normalized_for
    EXPANDED_LICENSE ||--o{ SBOM_LICENSE_EXPANDED : referenced_by

    SBOM_PACKAGE_LICENSE ||--o{ SBOM_LICENSE_EXPANDED : maps_to_expanded_via_keys

Loading

Class diagram for license querying and expanded license ingestion

classDiagram
    class LicenseService {
        +get_license_ref_mapping(connection) Result~Option~Vec~LicenseRefMapping~~
        +licenses(search, paginated, connection) Result~PaginatedResults~LicenseText~~
    }

    class SbomService {
        +fetch_sboms(search, paginated, connection) Result~PaginatedResults~Sbom~~
        +describes_packages(sbom_id, options, db) Result~R::Output~SbomPackageRelation~~
    }

    class PurlService {
        +search_purls(search, paginated, connection) Result~PaginatedResults~Purl~~
    }

    class PurlDetails {
        +from_model(qualified_package, connection) Result~PurlDetails~~
    }

    class LicenseFilterCommon {
        <<module>>
        +LICENSE : &str
        +license_text_coalesce() SimpleExpr
    }

    class ExpandedLicenseIngestor {
        <<module>>
        +populate_expanded_license(sbom_id, db) Result~(), DbErr~
    }

    class ExpandedLicenseEntity {
        <<entity>>
        +id : Uuid
        +expanded_text : String
        +text_hash : String
    }

    class SbomLicenseExpandedEntity {
        <<entity>>
        +sbom_id : Uuid
        +license_id : Uuid
        +expanded_license_id : Uuid
    }

    class SbomPackageLicenseEntity {
        <<entity>>
        +sbom_id : Uuid
        +license_id : Uuid
        +node_id : Uuid
        +license_type : String
        +Relation::License
        +Relation::SbomLicenseExpanded
    }

    class LicenseEntity {
        <<entity>>
        +id : Uuid
        +text : String
    }

    class SbomEntity {
        <<entity>>
        +sbom_id : Uuid
    }

    LicenseService --> LicenseFilterCommon : uses_license_text_coalesce
    LicenseService --> SbomLicenseExpandedEntity : joins
    LicenseService --> ExpandedLicenseEntity : joins
    LicenseService --> LicenseEntity : joins
    LicenseService --> SbomPackageLicenseEntity : joins

    SbomService --> LicenseFilterCommon : uses_LICENSE_constant
    SbomService --> SbomLicenseExpandedEntity : filters_via_subquery
    SbomService --> ExpandedLicenseEntity : filters_via_subquery
    SbomService --> SbomPackageLicenseEntity : filters_via_subquery
    SbomService --> LicenseEntity : filters_cyclonedx

    PurlService --> LicenseFilterCommon : uses_LICENSE_constant
    PurlService --> SbomLicenseExpandedEntity : filters_via_subquery
    PurlService --> ExpandedLicenseEntity : filters_via_subquery
    PurlService --> SbomPackageLicenseEntity : filters_via_subquery
    PurlService --> LicenseEntity : filters_cyclonedx

    PurlDetails --> LicenseFilterCommon : uses_license_text_coalesce
    PurlDetails --> SbomLicenseExpandedEntity : joins
    PurlDetails --> ExpandedLicenseEntity : joins
    PurlDetails --> SbomPackageLicenseEntity : joins
    PurlDetails --> LicenseEntity : joins

    ExpandedLicenseIngestor --> ExpandedLicenseEntity : inserts
    ExpandedLicenseIngestor --> SbomLicenseExpandedEntity : inserts
    ExpandedLicenseIngestor --> LicenseEntity : reads
    ExpandedLicenseIngestor --> SbomPackageLicenseEntity : reads
    ExpandedLicenseIngestor --> SbomEntity : uses_sbom_id_keys

Loading

File-Level Changes

Change	Details	Files
Replace on-the-fly license expansion logic with COALESCE over pre-expanded license tables and simplify license filtering across services.	Remove legacy license_filtering helpers that used custom DB functions, CTEs, and unioned SPDX/CycloneDX queries. Introduce a shared license_text_coalesce() helper that prefers expanded_license.expanded_text and falls back to license.text. Refactor LicenseService::get_all_license_info() to select and order by the COALESCE expression via joins on sbom_license_expanded and expanded_license. Rewrite LicenseService::licenses() to UNION SPDX-expanded licenses from expanded_license with non-SBOM licenses from license using NOT EXISTS instead of LEFT JOIN + IS NULL, and apply filtering/sorting on the logical license text column.	modules/fundamental/src/common/license_filtering.rs modules/fundamental/src/license/service/mod.rs
Refactor SBOM and PURL license-based queries to use the new expanded license schema and subquery-based filtering.	Update SBOM listing and package queries to join via sbom_license_expanded/expanded_license and to filter by license using SPDX and CycloneDX subqueries instead of shared CTEs. Simplify PURL service license filtering to use separate SPDX and CycloneDX subqueries with in_subquery conditions, removing raw SQL/CTE plumbing. Adjust join_licenses() aggregation to use license_text_coalesce() and new LEFT JOIN chain through sbom_license_expanded and expanded_license. Remove licensing_infos-based mapping fetches and deprecate use of licenses_ref_mapping in SbomPackage construction.	modules/fundamental/src/sbom/service/sbom.rs modules/fundamental/src/purl/service/mod.rs modules/fundamental/src/sbom/model/details.rs modules/fundamental/src/sbom/model/mod.rs
Introduce expanded license dictionary and junction entities plus ingestion-time population and supporting migrations.	Add expanded_license and sbom_license_expanded entities and relations, including relation from sbom_package_license to sbom_license_expanded. Create migrations for license query performance (indexes + optimized expand_license_expression_with_mappings) and for normalizing expanded licenses into dictionary/junction tables with backfill and cleanup of old functions. Implement populate_expanded_license() using raw SQL/CTEs to fill expanded_license and sbom_license_expanded for a given SBOM at ingestion time. Wire populate_expanded_license() into both SPDX and CycloneDX SBOM ingestion flows after package/license creation.	entity/src/expanded_license.rs entity/src/sbom_license_expanded.rs entity/src/sbom_package_license.rs migration/src/m0002110_license_query_performance.rs migration/src/m0002110_license_query_performance/up.sql migration/src/m0002110_license_query_performance/down.sql migration/src/m0002120_normalize_expanded_license.rs migration/src/m0002120_normalize_expanded_license/up.sql migration/src/m0002120_normalize_expanded_license/down.sql modules/ingestor/src/graph/sbom/common/expanded_license.rs modules/ingestor/src/graph/sbom/common/mod.rs modules/ingestor/src/graph/sbom/cyclonedx.rs modules/ingestor/src/graph/sbom/spdx.rs migration/src/lib.rs entity/src/lib.rs
Deprecate legacy PL/pgSQL license expansion entry points and adjust custom DB function plumbing.	Remove ExpandLicenseExpression and CaseLicenseTextSbomId idents from the Rust db func module as they are no longer invoked from queries. Document that expand_license_expression_with_mappings remains as a raw-SQL-only function due to complexity. Drop old case_license_text_sbom_id and expand_license_expression functions in the normalize_expanded_license migration down/up scripts and restore them in down.sql for rollback.	common/src/db/func.rs migration/src/m0002120_normalize_expanded_license/up.sql migration/src/m0002120_normalize_expanded_license/down.sql
Update downstream models, endpoints, and OpenAPI to reflect pre-expanded licenses and deprecate LicenseRef mappings.	Switch PurlDetails license lookup to use license_text_coalesce() and the new LEFT JOIN chain through sbom_license_expanded and expanded_license. Mark SbomPackage.licenses_ref_mapping as deprecated, always returning an empty vector, and remove runtime construction of mappings from SPDX expressions. Mark licenses_ref_mapping fields as deprecated in OpenAPI schemas with explanatory documentation.	modules/fundamental/src/purl/model/details/purl.rs modules/fundamental/src/sbom/model/mod.rs openapi.yaml
Extend test suites to cover new license expansion, filtering, and COALESCE behavior across services and endpoints.	Add a dedicated license service test module validating UNION behavior, COALESCE correctness, MD5-based deduplication, pagination, and filtering semantics. Extend SBOM service tests to assert COALESCE-based license exposure for SPDX and CycloneDX SBOMs and license-based package filtering and JSON aggregation behavior. Add endpoint tests to ensure no partial LicenseRef matches and correct expansion of overlapping LicenseRef identifiers. Introduce SPDX test data with overlapping LicenseRef identifiers used by the new tests.	modules/fundamental/src/license/service/test.rs modules/fundamental/src/sbom/service/test.rs modules/fundamental/src/license/endpoints/test.rs etc/test-data/spdx/license-ref-overlap.json
Miscellaneous backport-specific adjustments (type compatibility and recursion limits).	Fix IngestResult.id handling on release/0.4.z by replacing Uuid::parse_str(&result.id) with result.id.try_as_uid() in tests. Increase recursion_limit to 512 in several crates to keep CI builds passing with the larger code paths. Apply NOT EXISTS optimization for non-SBOM license selection to restore performance lost in an intermediate upstream commit.	modules/fundamental/src/sbom/service/test.rs modules/fundamental/src/license/service/test.rs modules/importer/src/lib.rs modules/user/src/lib.rs server/src/lib.rs modules/fundamental/src/license/service/mod.rs

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've left some high level feedback:

• There are unresolved merge conflict markers in modules/ingestor/src/graph/sbom/cyclonedx.rs (e.g. <<<<<<< HEAD / >>>>>>> 49185eb8 around ComponentCreator and ComponentType), which must be resolved before this can be merged.
• The normalization/backfill logic for expanded licenses is currently duplicated between the migration SQL (m0002120_normalize_expanded_license/up.sql) and the ingestion helper (populate_expanded_license); consider centralizing this into a single SQL statement or helper to avoid future drift between the two code paths.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- There are unresolved merge conflict markers in `modules/ingestor/src/graph/sbom/cyclonedx.rs` (e.g. `<<<<<<< HEAD` / `>>>>>>> 49185eb8` around `ComponentCreator` and `ComponentType`), which must be resolved before this can be merged.
- The normalization/backfill logic for expanded licenses is currently duplicated between the migration SQL (`m0002120_normalize_expanded_license/up.sql`) and the ingestion helper (`populate_expanded_license`); consider centralizing this into a single SQL statement or helper to avoid future drift between the two code paths.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[ctron]

That seems to pull in all kinds of stuff from main which should not be in the PR.

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• The raw SQL used to populate expanded_license/sbom_license_expanded appears in both the migration backfill and populate_expanded_license; consider consolidating this logic or at least clearly documenting the intended differences to avoid subtle drift between ingestion-time and migration-time behavior.
• In LicenseService::licenses, the comment describing non_sbom_query still mentions CycloneDX licenses "that were never expanded", but CycloneDX ingestion now calls populate_expanded_license; updating this comment to match the actual behavior will make the intent clearer for future maintainers.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The raw SQL used to populate `expanded_license`/`sbom_license_expanded` appears in both the migration backfill and `populate_expanded_license`; consider consolidating this logic or at least clearly documenting the intended differences to avoid subtle drift between ingestion-time and migration-time behavior.
- In `LicenseService::licenses`, the comment describing `non_sbom_query` still mentions CycloneDX licenses "that were never expanded", but CycloneDX ingestion now calls `populate_expanded_license`; updating this comment to match the actual behavior will make the intent clearer for future maintainers.

## Individual Comments

### Comment 1
<location path="modules/ingestor/src/graph/sbom/common/expanded_license.rs" line_range="19-28" />
<code_context>
+pub async fn populate_expanded_license(
</code_context>
<issue_to_address>
**suggestion (bug_risk):** populate_expanded_license should run in a transaction to avoid partial state between dictionary and junction tables

Because this runs two separate SQL statements (one for `expanded_license`, one for `sbom_license_expanded`), a failure in the second after the first succeeds can leave inconsistent data that ingestion won’t fix. Please wrap both in a single transaction (or take a `&DatabaseTransaction` instead of a generic `ConnectionTrait`) so they either both commit or both roll back.

Suggested implementation:

```rust
pub async fn populate_expanded_license(
    sbom_id: Uuid,
    db: &impl ConnectionTrait,
) -> Result<(), DbErr> {
    // Run all dictionary and junction-table operations in a single transaction
    let txn = db.begin().await?;

    // Step 1: Insert into expanded_license dictionary
    txn.execute(Statement::from_sql_and_values(
        txn.get_database_backend(),
        r#"
INSERT INTO expanded_license (expanded_text)
SELECT DISTINCT expand_license_expression_with_mappings(
    l.text,

```

1. Replace all other uses of `db.execute(...)` within `populate_expanded_license` with `txn.execute(...)`, and use `txn.get_database_backend()` instead of `db.get_database_backend()` where needed.
2. Before the final `Ok(())` in `populate_expanded_license`, add `txn.commit().await?;` so the transaction is committed only after both the `expanded_license` and `sbom_license_expanded` insert statements succeed.
3. Ensure there are no early `return` statements in the middle of the function that would bypass the `txn.commit().await?;` (errors will naturally roll back the transaction).
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[mrrajan]

That seems to pull in all kinds of stuff from main which should not be in the PR.

@ctron I have aligned the changes and it requires backport from PR #2242 as well. Along with the couple of changes included for the resolve conflicts as well. PR is ready for review - Please let me know your thoughts.

• IngestResult.id is Id enum on release/0.4.z, not String. Changed Uuid::parse_str(&result.id) to result.id.try_as_uid() in modules/fundamental/src/license/service/test.rs, modules/fundamental/src/sbom/service/test.rs
• Commit 49185eb rewrote licenses() but lost NOT EXISTS optimization from 3e9348d. Query reverted to inefficient LEFT JOIN + IS NULL (scans all rows). Applied NOT EXISTS pattern. Changed from sbom_package_license to sbom_license_expanded table on modules/fundamental/src/license/service/mod.rs

[ctron]

Minor nit with the ignores.

However, I'm just saying that that PR has quite an impact in how we handle query as well as ingestion. And we don't have any performance measurements, as far as I know.

And the goal here is to merge this into a patch release.

I think it would make sense to, at least manually, give it a try and see how it impacts performance with a larger set of data.

[ctron]

Couldn't we just remove this line and the next one?

[mrrajan]

Removed the lines.

[ctron]

Thanks ❤️ Technically that PR is fine. I think someone needs to make a decision if we want to this merged without performance checks. Unless we already have some somewhere.

[PhilipCattanach]

@ctron @mrrajan Before we merge this PR into the Release branch let's meet. I want to discuss our Release options.