Documentation/721 add workflow generate (#724)

ArBridgeman · web-flow · commit 5626f330e7d9 · 2026-02-25T15:45:38.000+01:00
* Split up API into sub-files &amp; fix so BaseConfig can be viewed
* Update to workflow:generate in docs
* Add workflow_patcher page
* Add models to API
diff --git a/doc/api/base_config.rst b/doc/api/base_config.rst
@@ -0,0 +1,13 @@
+.. _base_config:
+
+BaseConfig
+==========
+
+.. currentmodule:: exasol.toolbox.config
+
+.. autoclass:: BaseConfig
+   :members:
+   :undoc-members:
+   :show-inheritance:
+   :member-order: bysource
+   :special-members: __init__
diff --git a/doc/api/index.rst b/doc/api/index.rst
@@ -0,0 +1,11 @@
+.. _api:
+
+:octicon:`cpu` API Reference
+=============================
+
+.. toctree::
+    :maxdepth: 2
+
+    base_config
+    workflow_exceptions
+    workflow_patcher_config
diff --git a/doc/api/workflow_exceptions.rst b/doc/api/workflow_exceptions.rst
@@ -1,8 +1,3 @@
-.. _api:
-
-:octicon:`cpu` API Reference
-=============================
-
 .. _workflow_exceptions:
 
 Workflow Exceptions
diff --git a/doc/api/workflow_patcher_config.rst b/doc/api/workflow_patcher_config.rst
@@ -0,0 +1,36 @@
+.. _workflow_patcher_config:
+
+WorkflowPatcherConfig
+=====================
+
+.. currentmodule:: exasol.toolbox.util.workflows.patch_workflow
+
+.. autoclass:: WorkflowPatcherConfig
+   :members:
+   :undoc-members:
+   :show-inheritance:
+   :member-order: bysource
+   :special-members: __init__
+
+.. autoclass:: Workflow
+   :members:
+   :undoc-members:
+   :show-inheritance:
+   :member-order: bysource
+   :special-members: __init__
+
+.. autoclass:: StepCustomization
+   :members:
+   :undoc-members:
+   :show-inheritance:
+   :member-order: bysource
+   :special-members: __init__
+
+.. autoclass:: StepContent
+   :members:
+   :undoc-members:
+   :show-inheritance:
+   :member-order: bysource
+   :special-members: __init__
+
+.. autofunction:: validate_workflow_name
diff --git a/doc/changes/unreleased.md b/doc/changes/unreleased.md
@@ -2,19 +2,29 @@
 
 ## Summary
 
+In this major version:
+* the Nox session `workflow:generate` has been added to replace the deprecated
+`tbx workflow install` and `tbx workflow update`. It has the additional feature
+that users may customize the PTB provided workflows with a `.workflow-patcher.yml`
+file.
+* the GitHub workflow templates have been modified to include step_ids and to follow
+an AP-format naming convention, as such it is anticipated that updating the workflows
+results in several small changes.
+
 ## Feature
 
 * #691: Started customization of PTB workflows by defining the YML schema
 * #712: Added basic logging to workflow processing
-* #714: Added logic to modify a workflow using the .workflow-patcher.yml
-* #717: Restricted workflow names in .workflow-patcher.yml to template workflow names
-* #719: Added nox session `workflow:generate` to generate/update workflows using the `.workflow-patcher.yml` (if desired)
+* #714: Added logic to modify a workflow using the` .workflow-patcher.yml`
+* #717: Restricted workflow names in `.workflow-patcher.yml` to template workflow names
+* #719: Added Nox session `workflow:generate` to generate/update workflows using the `.workflow-patcher.yml` (if desired)
 * #725: Added newline after headlines for dependency changes
 
 ## Documentation
 
 * #705: Described how the versions of poetry and python are retrieved
 * #706: Added description how to ignore findings to the User Guide
+* #721: Added documentation for Nox session `workflow:generate`
 
 ## Refactoring
 
diff --git a/doc/index.rst b/doc/index.rst
@@ -52,5 +52,5 @@ Documentation of the Exasol-Toolbox
    developer_guide/developer_guide
    tools
    github_actions/github_actions
-   api
+   api/index
    changes/changelog
diff --git a/doc/user_guide/configuration.rst b/doc/user_guide/configuration.rst
@@ -11,3 +11,4 @@ especially when starting a new project.
 
    features/metrics/sonar
    Formatting <features/formatting_code/configuration>
+   features/github_workflows/github_project_configuration
diff --git a/doc/user_guide/features/github_workflows/create_and_update.rst b/doc/user_guide/features/github_workflows/create_and_update.rst
@@ -3,12 +3,12 @@
 Creating and Updating the GitHub Workflows in Your Project
 ==========================================================
 
-PTB can initially generate the GitHub workflows in your project and also
+The PTB can initially generate the GitHub workflows in your project and also
 update existing workflows.
 
 The workflows are based on Jinja templates with variables populated by the
-PTB. The PTB reads the values from various places within your project, see
-:ref:`template_variables`.
+PTB. The PTB reads the values from various attributes and properties of your
+project's config, see :ref:`template_variables`.
 
 Please note that the PTB only updates the values in the GitHub workflows when
 *updating* the workflows. So, after updating the :ref:`list of Python versions
@@ -36,8 +36,8 @@ Many workflows are using a Build-matrix to iterate over multiple versions of
 Python and/or the Exasol Docker DB. This is to make sure your code is valid,
 free of bugs and working correctly for each combination of these items.
 
-PTB has a default for these versions, but you can override it in file
-``noxconfig.py``, e.g.
+The PTB has a default for these versions, but you can override it in the
+``noxconfig.py`` file, e.g.
 
 .. code-block:: shell
 
@@ -49,18 +49,62 @@ PTB has a default for these versions, but you can override it in file
 Some workflows are expected to not depend on a specific Python version and
 will use only the lowest Python version in the list specified above.
 
+.. _customize_workflows:
+
+Customize Workflows for Your Project
+------------------------------------
+
+The PTB allows you to customise workflows by targeting specific jobs or steps:
+
+* Remove a job by its job_name.
+* Replace a step (referenced by step_id) with one or more new steps.
+* Insert steps after a specific step_id.
+
+.. note::
+
+   These operations do not currently cascade. For example, removing a job
+   without accounting for its downstream dependencies may result in errors.
+   You must manually adjust any subsequent steps that rely on the removed
+   job's or step's output.
+
+To utilize this feature, create a ``.workflow-patcher.yml`` file in your project's
+root directory, as specified further in :ref:`workflow_patcher`. This file will be
+automatically detected, validated by a pydantic model, and
+used when you :ref:`install or update your workflows <update_workflows>`.
+
+.. note::
+  The pydantic validation checks that the yml file is in the expected format
+  and that the specified workflow names exist. However, when a workflow is being
+  generated, each removed job or modified step_id is checked to see if it exists.
+  If it does not exist, an exception will be raised (:ref:`workflow_exceptions`).
+
 .. _update_workflows:
 
 Add all Workflows to Your Project
 ---------------------------------
 
 .. code-block:: shell
 
-    tbx workflow install all
+    poetry run -- nox -s workflow:generate -- all
 
 .. warning::
-    #. If you already have various workflows, you may want to run the
-       :code:`update` command instead of the :code:`install` command.
+    Some workflows depend on other workflows. Please ensure you have all
+    the required workflows if you do not install all of them.
+
+.. note::
+
+   The commands:
+
+   * ``tbx workflow install all`` - used to install workflows
+   * ``tbx workflow update all`` - used to update workflows
+
+   are considered historic variants of this command.
+
+   **Deprecation Notice:**
+   These ``tbx`` endpoints are marked as **deprecated** and are scheduled for removal
+   by **April 22nd, 2026**.
 
-    #. Some workflows depend on other workflows. Please ensure you have all
-       the required workflows if you do not install all of them.
+   Please note that these legacy commands do not allow users to use their specified
+   ``.workflow-patcher.yml`` file to further customise or patch workflows. Users
+   should transition to the ``nox``-based command to leverage full customisation
+   features.
diff --git a/doc/user_guide/features/github_workflows/github_project_configuration.rst b/doc/user_guide/features/github_workflows/github_project_configuration.rst
@@ -7,8 +7,9 @@ Branch Protection
 -----------------
 
 The best and most maintainable way to have solid branch protection
-(:code:`Settings/Branches/main`) is to require the workflow :code:`CI / Allow
-Merge` to pass successfully.
+(:code:`Settings/Branches/main`) is to require the workflow :code:`merge-gate / Allow
+Merge` to pass successfully. Additionally, if it makes sense for your project,
+you can further require that ``SonarCloud Code Analysis`` passes.
 
 .. note::
    Setting the required status checks to pass before merging is only possible
@@ -17,7 +18,7 @@ Merge` to pass successfully.
 Manual Approval
 ---------------
 
-If your CI workflow involves slow or expensive steps you can guard these to be
+If your CI workflow involves slow or expensive steps, you can guard these to be
 executed only after manual approval. The CI workflow will automatically create
 a GitHub environment named :code:`manual-approval`. You only need to add
 reviewers in (:code:`Settings/Environments/manual-approval`) and move the
@@ -27,11 +28,11 @@ file :code:`.github/workflows/merge-gate.yml`.
 Secrets
 -------
 
-For accessing specific services in the Internet, your project often needs a
+For accessing specific services in the internet, your project often needs a
 related *token* or other credentials. These credentials can be acquired by
 registering on the service's website.
 
-In many cases your company or organization may manage the credentials
+In many cases, your company or organization may manage the credentials
 centrally and enable the use in multiple projects. The credentials can be
 managed as Secrets in GitHub and can be made accessible to GitHub projects and
 used by their workflows.
diff --git a/doc/user_guide/features/github_workflows/index.rst b/doc/user_guide/features/github_workflows/index.rst
@@ -10,6 +10,7 @@ GitHub Workflow Templates
     github_project_configuration
     create_and_update
     template_variables
+    workflow_patcher
 
 The PTB ships with configurable GitHub workflow templates covering the most common
 CI/CD setup variants for Python projects. The templates are defined in:
@@ -20,13 +21,15 @@ workflows from the templates.
 
 .. code-block:: bash
 
-    poetry run -- tbx workflow --help
+    poetry run -- nox -s workflow:generate --help
 
 .. attention::
 
    In most cases, we recommend using _all_ workflows without change to ensure
    consistent interdependencies. For more details, see :ref:`ci_actions`.
 
+   The deprecated alternate is to use the CLI provided by
+   ``poetry run -- tbx workflow --help``. This will be removed by April 22nd, 2026.
 
 Workflows
 ---------
diff --git a/doc/user_guide/features/github_workflows/template_variables.rst b/doc/user_guide/features/github_workflows/template_variables.rst
@@ -6,9 +6,10 @@ Template Variables
 Underlying the CLI, the PTB uses Jinja to dynamically generate project-specific
 workflows. The rendering process is supported by the ``github_template_dict`` found in
 your ``noxconfig.py::PROJECT_CONFIG``. This dictionary is inherited by default from
-``BaseConfig.py``, ensuring a standardized baseline that can be easily overridden, if
-necessary.
+:py:attr:`exasol.toolbox.config.BaseConfig.github_template_dict`, ensuring a
+standardized baseline that can be easily overridden, if necessary.
 
 .. literalinclude:: ../../../../exasol/toolbox/config.py
   :language: python
   :start-at: github_template_dict
+  :end-before: @computed_field
diff --git a/doc/user_guide/features/github_workflows/workflow_patcher.rst b/doc/user_guide/features/github_workflows/workflow_patcher.rst
@@ -0,0 +1,57 @@
+.. _workflow_patcher:
+
+Workflow Patcher
+================
+
+Underlying the CLI, the PTB uses, if defined, a ``.workflow_patcher.yml`` file to
+customize generated project-specific workflows. The rendering process is supported by
+the ``github_workflow_patcher_yaml`` found in your ``noxconfig.py::PROJECT_CONFIG``.
+This filepath is inherited by default from
+:py:attr:`exasol.toolbox.config.BaseConfig.github_workflow_patcher_yaml`
+ensuring a standardized baseline that can be easily overridden, if necessary.
+
+.. literalinclude:: ../../../../exasol/toolbox/config.py
+  :language: python
+  :start-at: github_workflow_patcher_yaml
+
+Model
+-------
+
+.. code-block:: yaml
+
+   workflows:
+     - name: pr-merge
+       remove_jobs:
+         - publish-docs
+       step_customizations:
+         - action: REPLACE | INSERT_AFTER
+           job: run-unit-tests
+           step_id: check-out-repository
+           content:
+             - name: Check out Repository
+               id: check-out-repository
+               uses: actions/checkout@v6
+               with:
+                 fetch-depth: 0
+
+
+* ``name``: Name of the GitHub workflow to customize. The PTB supports these workflows:
+  `exasol/toolbox/templates/github/workflows <https://github.com/exasol/python-toolbox/tree/main/exasol/toolbox/templates/github/workflows>`__.
+* ``remove_jobs``: List of job names to remove from the workflow.
+  This is useful when a job like ``publish-docs`` is not applicable for a project.
+* ``step_customizations``: List of customizations:
+
+  * ``action``: Type of customization
+
+    * ``REPLACE``: Replace an existing step with the new content
+    * ``INSERT_AFTER``: Insert the content **after** the specified step
+
+  * ``job``: Name of the job inside the workflow that should be modified, e.g. ``run-unit-tests``.
+  * ``step_id``: ID of the step to replace or after which to insert the new step
+  * ``content``: Content of the new step. The PTB does not validate that this will work on
+    GitHub, but it does validate that it is valid YAML content.
+
+.. note::
+
+    For more information, see the API documentation for
+    :class:`exasol.toolbox.util.workflows.patch_workflow.WorkflowPatcherConfig`.
diff --git a/exasol/toolbox/config.py b/exasol/toolbox/config.py
diff --git a/exasol/toolbox/util/workflows/patch_workflow.py b/exasol/toolbox/util/workflows/patch_workflow.py
diff --git a/test/integration/project-template/nox_test.py b/test/integration/project-template/nox_test.py

Original file line number	Diff line number	Diff line change
`@@ -11,3 +11,4 @@ especially when starting a new project.`
`11`	`11`
`12`	`12`	`features/metrics/sonar`
`13`	`13`	`Formatting <features/formatting_code/configuration>`
	`14`	`+ features/github_workflows/github_project_configuration`