Skip to content

Support workflow modules (kind: Workflow) in the module system - #7363

Merged
bentsherman merged 2 commits into
adr-workflow-modulesfrom
260714-workflow-modules
Sep 2, 2026
Merged

Support workflow modules (kind: Workflow) in the module system#7363
bentsherman merged 2 commits into
adr-workflow-modulesfrom
260714-workflow-modules

Conversation

@jorgee

@jorgee jorgee commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Draft — stacked on #7342 (the workflow-modules ADR). Base is adr-workflow-modules so the diff shows only the implementation; retarget to master once #7342 lands.

Implements the workflow modules feature from the ADR (#7342): the module system is extended to support standalone workflow modules (kind: Workflow) alongside process modules — published, installed, resolved, included, and (via a follow-up PR) executed through the existing registry / nextflow module CLI infrastructure.

What's implemented

Spec model & schema

  • ModuleSpec/ModuleSpecFactory parse a top-level kind (Process|Workflow, default Process) and a nested requires.modules list of direct dependencies.
  • adr/module-spec-schema.json allows kind, requires.modules (exact-pin references), nested scope/name names, and adds channel / custom-record to the parameter type enum as documentation tags for statically-typed declarations.

Install / include

  • Nested vendoringModuleResolver installs a workflow module together with its transitive dependencies, each vendored under its own nested modules/ directory (modules/<scope>/<name>/modules/...). No cross-module flattening — duplication accepted per the ADR's diamond-dependency decision — plus dependency-cycle detection.
  • Context-relative include resolution — the runtime path (IncludeDef) and both compile-time paths (nf-lang ResolveIncludeVisitor and the source-discovery ModuleResolver) resolve a module's remote includes relative to the including file's directory, so a workflow module's own includes find its nested dependencies.
  • Integrity checksum covers the module's full installed subtree (own files + vendored deps), computed after vendoring — so a freshly installed workflow-with-deps reads VALID, and edits to either the module or a vendored dependency are detected as MODIFIED.
  • module install -update-deps — for an already-installed module, re-vendors its dependencies to match the (possibly locally-edited) meta.yml without reinstalling the module itself: installs new deps, updates changed versions, prunes removed ones (a locally-modified dependency is never silently overwritten or pruned). Ignored when the module is not installed; mutually exclusive with -force.

Publish / validate

  • kind-aware validation: a workflow module must define exactly one workflow; when its meta.yml declares input/output, the counts must match the workflow's take:/emit: arity.
  • requires.modules are validated against the registry at publish time (they exist at their pinned version), since dependencies are re-resolved from the registry by consumers at install time.
  • Nested vendored dependencies are excluded from the publish bundle (consumers re-resolve them from the registry).
  • Schema-location override (-schema flag, NXF_MODULE_SPEC_SCHEMA env, or meta.yml $schema).

nextflow module CLI — kind awareness

  • list — shows a Kind column (and kind JSON field).
  • create-kind Process|Workflow and -typed. Typed scaffolds set nextflow.enable.types and use typed input/output (process) or take/emit (workflow); the generated meta.yml follows suit, and typed process/workflow declare the minimum Nextflow version that introduced them (25.10.0 / 26.04.0). An untyped workflow scaffold documents its channel take/emit.
  • spec — supports workflow modules, deriving input/output from the workflow's take:/emit: (types inferred when statically typed; TODO placeholder otherwise).
  • search / view — show the module kind (formatted + JSON), defaulting to Process.

Registry dependency (npr-api 0.24.10 → seqeralabs/nextflow-registry#366)

The kind shown by module search / module view comes from the registry API, which added kind to ModuleSearchResult / ModuleMetadata in seqeralabs/nextflow-registry#366 (npr-api 0.24.10). This branch bumps io.seqera:npr-api / npr-client to 0.24.10.

CI will not pass until npr-api 0.24.10 is published to the Seqera Maven repository. To build and test locally in the meantime:

# 1. Build the registry API client from PR #366 (VERSION 0.24.10) and publish it to local Maven
cd <nextflow-registry checkout on seqeralabs/nextflow-registry#366>
./gradlew :npr-api:publishToMavenLocal :npr-client:publishToMavenLocal

# 2. In this repo, add mavenLocal() to the repositories block in build.gradle (do NOT commit this):
#    repositories {
#        mavenLocal()
#        mavenCentral()
#        ...
#    }

# 3. Build / run nextflow as usual
make compile

Not in this PR (follow-ups)

  • Direct typed execution nextflow module run scope/name — entry-workflow synthesis from take:/emit:, samplesheet input loading, and work-dir-path outputs — is in a separate PR.
  • module search --kind filter (registry-side support needed).
  • Typed-record samplesheet loading and the ADR's no-output-directory / index-file outputs (both follow-ups of the module-run PR).

Testing

  • Unit: ModuleSpecFactoryTest (process + workflow spec, typed types, channel/custom-record), ModuleValidatorTest, ModuleSchemaValidatorTest, ModuleResolverDependencyTest (nested install layout, per-consumer duplication, cycle detection, auto-install, checksum/integrity, -update-deps), ModuleStorageTest (bundle excludes nested modules/), CmdModuleCreateTest, CmdModuleListTest, CmdModulePublishTest, CmdModuleSearchTest, CmdModuleViewTest.
  • Manual e2e: published a kind: Workflow module (adapted from nf-core mafft_align + its mafft/align dependency) to a dev registry, ran module install, and included/composed it in a pipeline — verified working.

Notes

  • The registry-side changes (persisting/serving kind + requires.modules, exposing kind on search/view) are in seqeralabs/nextflow-registry#366.

Add support for workflow modules (`kind: Workflow`) in the module system:
a module can now publish a named workflow, declare its own module
dependencies, and be installed with those dependencies vendored under its
own nested `modules/` directory.

- Module spec: `kind` (Process/Workflow) and `requires.modules`, validated
  against the registry rather than local vendoring.
- Install: nested per-module vendoring of transitive dependencies, with
  `-update-deps` to re-vendor an installed module's dependencies, and a
  checksum refresh after vendoring.
- Publish: validate `requires.modules` and the workflow take/emit interface,
  and exclude nested vendored modules from the bundle.
- Create: typed and workflow scaffolding (`-kind`, `-typed`).
- Spec/validate: derive a workflow module's input/output from its take/emit.
- Show the module kind in `module list`, `module search`, and `module view`
  output, defaulting to Process when the registry does not report one.

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
The `kind` field on ModuleSearchResult / ModuleMetadata is not in a released
npr-api yet (registry side: seqeralabs/nextflow-registry#366), so referencing
it does not compile. Comment out the kind display in both commands and the
corresponding test fixtures and assertions.

Revert this commit once npr-api ships the field.

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@bentsherman
bentsherman force-pushed the 260714-workflow-modules branch from f442fb9 to 9209bb9 Compare September 2, 2026 21:08
@bentsherman
bentsherman marked this pull request as ready for review September 2, 2026 21:09
@bentsherman
bentsherman requested review from a team as code owners September 2, 2026 21:09
@bentsherman
bentsherman merged commit 5648445 into adr-workflow-modules Sep 2, 2026
21 checks passed
@bentsherman
bentsherman deleted the 260714-workflow-modules branch September 2, 2026 21:47
@bentsherman

Copy link
Copy Markdown
Member

Continuing review and testing on #7342

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants