Support workflow modules (kind: Workflow) in the module system - #7363
Merged
Conversation
jorgee
force-pushed
the
260714-workflow-modules
branch
from
July 24, 2026 10:50
e1f20cc to
39715ef
Compare
bentsherman
force-pushed
the
adr-workflow-modules
branch
from
September 2, 2026 20:50
d691aa8 to
30f7dfb
Compare
Add support for workflow modules (`kind: Workflow`) in the module system: a module can now publish a named workflow, declare its own module dependencies, and be installed with those dependencies vendored under its own nested `modules/` directory. - Module spec: `kind` (Process/Workflow) and `requires.modules`, validated against the registry rather than local vendoring. - Install: nested per-module vendoring of transitive dependencies, with `-update-deps` to re-vendor an installed module's dependencies, and a checksum refresh after vendoring. - Publish: validate `requires.modules` and the workflow take/emit interface, and exclude nested vendored modules from the bundle. - Create: typed and workflow scaffolding (`-kind`, `-typed`). - Spec/validate: derive a workflow module's input/output from its take/emit. - Show the module kind in `module list`, `module search`, and `module view` output, defaulting to Process when the registry does not report one. Signed-off-by: Ben Sherman <bentshermann@gmail.com>
The `kind` field on ModuleSearchResult / ModuleMetadata is not in a released npr-api yet (registry side: seqeralabs/nextflow-registry#366), so referencing it does not compile. Comment out the kind display in both commands and the corresponding test fixtures and assertions. Revert this commit once npr-api ships the field. Signed-off-by: Ben Sherman <bentshermann@gmail.com>
bentsherman
force-pushed
the
260714-workflow-modules
branch
from
September 2, 2026 21:08
f442fb9 to
9209bb9
Compare
bentsherman
marked this pull request as ready for review
September 2, 2026 21:09
Member
|
Continuing review and testing on #7342 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the workflow modules feature from the ADR (#7342): the module system is extended to support standalone workflow modules (
kind: Workflow) alongside process modules — published, installed, resolved, included, and (via a follow-up PR) executed through the existing registry /nextflow moduleCLI infrastructure.What's implemented
Spec model & schema
ModuleSpec/ModuleSpecFactoryparse a top-levelkind(Process|Workflow, defaultProcess) and a nestedrequires.moduleslist of direct dependencies.adr/module-spec-schema.jsonallowskind,requires.modules(exact-pin references), nestedscope/namenames, and addschannel/custom-recordto the parametertypeenum as documentation tags for statically-typed declarations.Install / include
ModuleResolverinstalls a workflow module together with its transitive dependencies, each vendored under its own nestedmodules/directory (modules/<scope>/<name>/modules/...). No cross-module flattening — duplication accepted per the ADR's diamond-dependency decision — plus dependency-cycle detection.IncludeDef) and both compile-time paths (nf-langResolveIncludeVisitorand the source-discoveryModuleResolver) resolve a module's remote includes relative to the including file's directory, so a workflow module's own includes find its nested dependencies.VALID, and edits to either the module or a vendored dependency are detected asMODIFIED.module install -update-deps— for an already-installed module, re-vendors its dependencies to match the (possibly locally-edited)meta.ymlwithout reinstalling the module itself: installs new deps, updates changed versions, prunes removed ones (a locally-modified dependency is never silently overwritten or pruned). Ignored when the module is not installed; mutually exclusive with-force.Publish / validate
kind-aware validation: a workflow module must define exactly oneworkflow; when itsmeta.ymldeclaresinput/output, the counts must match the workflow'stake:/emit:arity.requires.modulesare validated against the registry at publish time (they exist at their pinned version), since dependencies are re-resolved from the registry by consumers at install time.-schemaflag,NXF_MODULE_SPEC_SCHEMAenv, or meta.yml$schema).nextflow moduleCLI — kind awarenesslist— shows aKindcolumn (andkindJSON field).create—-kind Process|Workflowand-typed. Typed scaffolds setnextflow.enable.typesand use typedinput/output(process) ortake/emit(workflow); the generatedmeta.ymlfollows suit, and typed process/workflow declare the minimum Nextflow version that introduced them (25.10.0 / 26.04.0). An untyped workflow scaffold documents its channel take/emit.spec— supports workflow modules, derivinginput/outputfrom the workflow'stake:/emit:(types inferred when statically typed;TODOplaceholder otherwise).search/view— show the modulekind(formatted + JSON), defaulting toProcess.Registry dependency (npr-api 0.24.10 → seqeralabs/nextflow-registry#366)
The
kindshown bymodule search/module viewcomes from the registry API, which addedkindtoModuleSearchResult/ModuleMetadatain seqeralabs/nextflow-registry#366 (npr-api 0.24.10). This branch bumpsio.seqera:npr-api/npr-clientto0.24.10.CI will not pass until npr-api 0.24.10 is published to the Seqera Maven repository. To build and test locally in the meantime:
Not in this PR (follow-ups)
nextflow module run scope/name— entry-workflow synthesis fromtake:/emit:, samplesheet input loading, and work-dir-path outputs — is in a separate PR.module search --kindfilter (registry-side support needed).Testing
ModuleSpecFactoryTest(process + workflow spec, typed types,channel/custom-record),ModuleValidatorTest,ModuleSchemaValidatorTest,ModuleResolverDependencyTest(nested install layout, per-consumer duplication, cycle detection, auto-install, checksum/integrity,-update-deps),ModuleStorageTest(bundle excludes nestedmodules/),CmdModuleCreateTest,CmdModuleListTest,CmdModulePublishTest,CmdModuleSearchTest,CmdModuleViewTest.kind: Workflowmodule (adapted from nf-coremafft_align+ itsmafft/aligndependency) to a dev registry, ranmodule install, and included/composed it in a pipeline — verified working.Notes
kind+requires.modules, exposingkindon search/view) are in seqeralabs/nextflow-registry#366.