feat: add support for preserving and labeling intermediate stage images

[ezopezo]

This adds support for preserving and labeling intermediate stage images in multi-stage builds. In contrast to the --layers flag, --cache-stages preserves only the final image from each named stage (FROM ... AS name), not every instruction layer. This also keeps the final image's layer count unchanged compared to a regular build.

New flags:

• --cache-stages: preserve intermediate stage images instead of removing them
• --stage-labels: add metadata labels to intermediate stage images (stage name, base image, build ID, parent stage name). Requires --cache-stages.
• --build-id-file: write unique build ID (UUID) to file for easier identification and grouping of intermediate images from a single build. Requires --stage-labels.

The implementation also includes:

• Detection of transitive alias patterns (stage using another intermediate stage as base)
• Validation that --stage-labels requires --cache-stages
• Validation that --build-id-file requires --stage-labels
• Test coverage (15 tests) and documentation updates

What type of PR is this?

/kind feature

What this PR does / why we need it:

General use: This functionality is useful for identification, debugging, and reusing intermediate stage images in multi-stage builds.

Specific need: Identifying the content copied from intermediate stages in multi-stage builds into the final image is a hard requirement for supporting Contextual SBOM - an SBOM that understands the origin of each component.
While intermediate images can be extracted using the --layers option, this approach has several issues for our use case:

• Intermediate stage images are unlabeled, making it difficult to determine which image corresponds to which build stage - especially when the Containerfile reuses the same pullspec across multiple stages.
• All instructions from all intermediate stages appear in the cache (visible via buildah images --all), which introduces unnecessary noise for our purposes.
• rootfs.diff_ids are not squashed in final stage: the final-stage image ends up containing diff IDs for every instruction in the final stage. However, we need the final build image to resemble a regular build (without --layers), meaning:
  • it should contain the diff IDs inherited from the base image, and
  • exactly one diff ID representing the squashed final-stage instructions.

Related repositories:
konflux (uses mobster for SBOM generation),
mobster (implements contextual SBOM functionality requiring this change),
capo (wraps builder content identification functionality for mobster),
Contact person: emravec (RedHat) / @ezopezo (Github)

How to verify it

Run any multistage build with intermediate stage specified with implemented arguments. Resulting intermediate images should be correctly labeled. Example:
buildah build --cache-stages --stage-labels --build-id-file ./file.txt -t test:0.1 .

Which issue(s) this PR fixes:

Fixes: #6257
Internal Jira: https://issues.redhat.com/browse/ISV-6122

Does this PR introduce a user-facing change?

Add `--cache-stages`, `--stage-labels`, and `--build-id-file` flags for preserving and labeling intermediate stage images in multi-stage builds.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ezopezo
Once this PR has been reviewed and has the lgtm label, please assign luap99 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[packit-as-a-service]

Ephemeral COPR build failed. @containers/packit-build please check.

[ezopezo]

/retest

[openshift-ci]

@ezopezo: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ezopezo]

@nalind can you please take a look and put ok-to-test label? It seems to me that tests are failing most likely with some timeouts and thus I would like to try to re-run them (or please tell me what I just broke :) ).

[nalind]

/ok-to-test

[ezopezo]

/test

[openshift-ci]

@ezopezo: No presubmit jobs available for containers/buildah@main

Details

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ezopezo]

@nalind @mtrmac @TomSweeneyRedHat can you please take a look on this? (or pick up some appropriate reviewers?) Thanks in advance!

[mtrmac]

@flouthoc if you have time

[TomSweeneyRedHat]

@Luap99 @nalind PTAL

[TomSweeneyRedHat]

just to show the bool values in play, perhaps

Suggested change

	buildah build --cache-stages --stage-labels -t imageName .
	buildah build --cache-stages false --stage-labels false -t imageName .

[ezopezo]

I skipped that since this is a relatively uncommon use case for boolean flags, because the false behavior is implicit whenever the flag is absent. Other flags such as --no-cache or --layers follow the same pattern (present when true, absent when not, without presence of the --layers false) so I wanted to stay consistent with your docs. If you feel strongly about it, I certainly can add the examples :)

[nalind]

There's a long-standing bit of confusion in the man pages where, even though the boolean argument values are optional, we don't suggest that they always have to be supplied in the --flag=value form, with an equal sign, to prevent them from being treated as unrelated arguments.

[ezopezo]

@nalind Does it make a sense to you to add args with equal sign and boolean or leave it as it is? What do you think?

[nalind]

If we want an example where the flag is set to false, the equal sign is going to be necessary.

[TomSweeneyRedHat]

This is great! TY!

[TomSweeneyRedHat]

Please add these three options in alphabetically. I.e. CacheStages at line 396.5 and so on.

[TomSweeneyRedHat]

To keep making my OCD happy, please move this to line 63.5

[TomSweeneyRedHat]

More OCD alpha changes, please put this at line 28.5

[TomSweeneyRedHat]

can you update and/or add tests to use "true" and "false" for the boolean flags? Leave at least one using the default true for each too.

[ezopezo]

For false I added two new tests for --cache-stages and --stage-labels and for true I updated one existing test 👍

[TomSweeneyRedHat]

First pass LGTM, just a couple of nitty things here and there.

@nalind or @Luap99 PTAL

Also, have you tried vendoring Buildah into Podman with these changes yet?

@ddarrah should we have a feature/epic for this in RHEL 9.8/10.2?

[ezopezo]

Overall I think I understand the goals here. The new flags interact with --layers in confusing ways, though - the build ID recorded for --build-id-file won't match the image whose ID we return if the build is a cache hit, but we may quietly commit new versions of the final product of intermediate stages. I'm not really sure what the intended result is there.

Does it make a sense to add an mutual exclusion condition that --layers and --cache-stages cannot be used together (as serve fundamentally different purposes)? I want to keep this functionality as isolated as possible from others to avoid unnecessary complexities like this.

[ezopezo]

@nalind Would you be able to take an another look please? Thanks in advance!

[nalind]

--layers=true is the default used by podman build, so I'd be hesitant about making a new feature mutually exclusive with it.
If we weren't worried about reusing cache images with previously-generated build UUIDs, I would generally expect the images we commit for each stage, including the final one, to use multiple layers when it's appropriate.
Would forcing checkForLayers to be false at the start of stageExecutor.execute() work as expected for the use case? I could see an argument for "fresh build, every time" when the new flag is used, and that would still allow subsequent builds that don't use the new flag to use the results as cache hits. If we did that, would the cache-related labels be left alone or altered/cleared for the products of that build?

[ezopezo]

--layers=true is the default used by podman build, so I'd be hesitant about making a new feature mutually exclusive with it. If we weren't worried about reusing cache images with previously-generated build UUIDs, I would generally expect the images we commit for each stage, including the final one, to use multiple layers when it's appropriate. Would forcing checkForLayers to be false at the start of stageExecutor.execute() work as expected for the use case? I could see an argument for "fresh build, every time" when the new flag is used, and that would still allow subsequent builds that don't use the new flag to use the results as cache hits. If we did that, would the cache-related labels be left alone or altered/cleared for the products of that build?

Yes, it works well buildah build --layers --cache-stages --stage-labels stores all the layers of the Containerfile, correctly labels them and subsequent plain buildah build --layers reuses those by hitting cache - those layers remain intact, labels are preserved. Are we okay with such behavior?

[ezopezo]

@nalind Also I almost forgot to ask - how I can effectively test if current changes are compatible with podman?
(I believe this is what @TomSweeneyRedHat asked before as well):
 

Also, have you tried vendoring Buildah into Podman with these changes yet?

[nalind]

@nalind Also I almost forgot to ask - how I can effectively test if current changes are compatible with podman? (I believe this is what @TomSweeneyRedHat asked before as well):

Since we're on main, check out podman's main branch, and use go mod edit -replace github.com/containers/buildah=github.com/ezopezo/buildah@478bf2159f6ffe878c5162ed1d83c69ae3e6a6d6 && go mod tidy && go mod vendor && make to build. (If your branch name was a valid version string, you could use its name directly instead of having to look up the commit ID.) At that point you can either attempt to test locally with make .install.ginkgo localintegration and related targets, or commit everything to a branch, push it to a podman fork, and open a PR to have podman's CI do it, taking care to mark it as not intended for merging.

[ezopezo]

@nalind Also I almost forgot to ask - how I can effectively test if current changes are compatible with podman? (I believe this is what @TomSweeneyRedHat asked before as well):

Since we're on main, check out podman's main branch, and use go mod edit -replace github.com/containers/buildah=github.com/ezopezo/buildah@478bf2159f6ffe878c5162ed1d83c69ae3e6a6d6 && go mod tidy && go mod vendor && make to build. (If your branch name was a valid version string, you could use its name directly instead of having to look up the commit ID.) At that point you can either attempt to test locally with make .install.ginkgo localintegration and related targets, or commit everything to a branch, push it to a podman fork, and open a PR to have podman's CI do it, taking care to mark it as not intended for merging.

@nalind Thank you I did it according your guide and created this testing PR. I think it is failing because of the docs mismatch and exceeding some limit of the binary size (which could be overridden by label I think) and also I think I need from maintainers approval for running all of the jobs, could you help me with this please?

[nalind]

I've given it a kick.

[ezopezo]

@nalind I am re-pushing changes, because /retest flag just does not work for me (maybe I need some privileges?) and checks are often sporadically failing.
Also I don't know how to proceed with podman vendoring test I left a comment in PR. Can you please help me?

[nalind]

@nalind I am re-pushing changes, because /retest flag just does not work for me (maybe I need some privileges?) and checks are often sporadically failing. Also I don't know how to proceed with podman vendoring test I left a comment in PR. Can you please help me?

The configuration for the bot which handled those things was updated to not drop most containers-org repositories by openshift/release#71397; anything that was in-progress before it was merged would still include a record of previous interactions with the bot.

[nalind]

--layers=true is the default used by podman build, so I'd be hesitant about making a new feature mutually exclusive with it. If we weren't worried about reusing cache images with previously-generated build UUIDs, I would generally expect the images we commit for each stage, including the final one, to use multiple layers when it's appropriate. Would forcing checkForLayers to be false at the start of stageExecutor.execute() work as expected for the use case? I could see an argument for "fresh build, every time" when the new flag is used, and that would still allow subsequent builds that don't use the new flag to use the results as cache hits. If we did that, would the cache-related labels be left alone or altered/cleared for the products of that build?

Yes, it works well buildah build --layers --cache-stages --stage-labels stores all the layers of the Containerfile, correctly labels them and subsequent plain buildah build --layers reuses those by hitting cache - those layers remain intact, labels are preserved. Are we okay with such behavior?

I had to think about this for a bit, and I'm not sure it's what we want. Much like earlier versions of build --label, adding the new labels when we commit, without having their presence show up in the history, prevents the cache evaluation logic from "seeing" them.

For the --label CLI flag, we eventually reworked the build so that it internally appended a LABEL instruction with its arguments to the set of build steps in executor.buildStage() before processing the steps for the stage, so that labels from the command line could be accounted for.

Would this use case be better served by doing something similar? The buildStage() method potentially appends a LABEL instruction to the end of the final stage, and potentially prepends an ENV instruction (for --env values) at the beginning of every stage, so we have options. Adding the stage label instruction at the start of the stage's instructions would cause cache misses and naturally provide the "fresh build, every time" behavior which I think we're aiming for. Adding it as an instruction at the end of the stage would allow every instruction before it to be a cache hit, if we want it the other way.

[ezopezo]

--layers=true is the default used by podman build, so I'd be hesitant about making a new feature mutually exclusive with it. If we weren't worried about reusing cache images with previously-generated build UUIDs, I would generally expect the images we commit for each stage, including the final one, to use multiple layers when it's appropriate. Would forcing checkForLayers to be false at the start of stageExecutor.execute() work as expected for the use case? I could see an argument for "fresh build, every time" when the new flag is used, and that would still allow subsequent builds that don't use the new flag to use the results as cache hits. If we did that, would the cache-related labels be left alone or altered/cleared for the products of that build?

Yes, it works well buildah build --layers --cache-stages --stage-labels stores all the layers of the Containerfile, correctly labels them and subsequent plain buildah build --layers reuses those by hitting cache - those layers remain intact, labels are preserved. Are we okay with such behavior?

I had to think about this for a bit, and I'm not sure it's what we want. Much like earlier versions of build --label, adding the new labels when we commit, without having their presence show up in the history, prevents the cache evaluation logic from "seeing" them.

For the --label CLI flag, we eventually reworked the build so that it internally appended a LABEL instruction with its arguments to the set of build steps in executor.buildStage() before processing the steps for the stage, so that labels from the command line could be accounted for.

Would this use case be better served by doing something similar? The buildStage() method potentially appends a LABEL instruction to the end of the final stage, and potentially prepends an ENV instruction (for --env values) at the beginning of every stage, so we have options. Adding the stage label instruction at the start of the stage's instructions would cause cache misses and naturally provide the "fresh build, every time" behavior which I think we're aiming for. Adding it as an instruction at the end of the stage would allow every instruction before it to be a cache hit, if we want it the other way.

prevents the cache evaluation logic from "seeing" them.

I may be misunderstanding this concern. Our implementation explicitly disables cache lookup when --cache-stages is used (stage_executor.go:1260-1262):

if s.executor.cacheStages {
    checkForLayers = false
}

This ensures every build with --cache-stages is a fresh build, and caches new intermediate images with a new build UUID. Cache evaluation logic (search) for stages doesn't run at all when --cache-stages is used.

Regarding cache reuse scenarios:
Scenario 1:

• Build 1: --cache-stages (without --layers) → creates non-layered intermediate images
• Build 2: --cache-stages (without --layers) → CANNOT reuse (explicitly disabled cache lookup)

Scenario 2:

• Build 1: --cache-stages (without --layers) → creates non-layered intermediate images
• Build 2: (without --cache-stages and --layers) → CANNOT reuse (insufficient history for cache hit, if I understand logic correctly), but that's fine for our use case

Scenario 3 (considering that podman has always --layers enabled):

• Build 1: --cache-stages --layers → creates layered intermediate images
• Build 2: --layers (without --cache-stages) → CAN reuse those layers (you're right - even when labeled, because of their absence in history - I'll expand this later*)

However, our actual use case (Konflux SBOM generation) will use --cache-stages without --layers, so the layered cache scenario isn't relevant for us.
The --layers compatibility was added based on your earlier suggestion that it's podman's default, but our production workflow won't use it.
The intermediate images from --cache-stages builds are intended for inspection (SBOM generation), not for cache reuse. We don't need subsequent builds to reuse them.

For the --label CLI flag...

*I see the architectural similarity, but there's a practical difference IMHO:

• --label: user-defined, cache evaluation is active (user applies different labels across builds = cache miss)
• --stage-labels: system metadata, cache evaluation is explicitly disabled (checkForLayers = false by mandatory preceeding --cache-stages)

Given that cache is disabled when --cache-stages is used, is there still a concern about labels (added by --stage labels, that cannot be used without --cache-stages) not being in build history?

Could you clarify what specific scenario you're concerned about where the current approach would fail?

[ezopezo]

@nalind I am re-pushing changes, because /retest flag just does not work for me (maybe I need some privileges?) and checks are often sporadically failing. Also I don't know how to proceed with podman vendoring test I left a comment in PR. Can you please help me?

The configuration for the bot which handled those things was updated to not drop most containers-org repositories by openshift/release#71397; anything that was in-progress before it was merged would still include a record of previous interactions with the bot.

@nalind I understand, thank you for explanation! I am just repushing commit with chnages to trigger pipeline - is it common that I am getting so much failures from timeouts? It is very hard for me to get all-green (I wittnessed it few times) :|

[nalind]

--layers=true is the default used by podman build, so I'd be hesitant about making a new feature mutually exclusive with it. If we weren't worried about reusing cache images with previously-generated build UUIDs, I would generally expect the images we commit for each stage, including the final one, to use multiple layers when it's appropriate. Would forcing checkForLayers to be false at the start of stageExecutor.execute() work as expected for the use case? I could see an argument for "fresh build, every time" when the new flag is used, and that would still allow subsequent builds that don't use the new flag to use the results as cache hits. If we did that, would the cache-related labels be left alone or altered/cleared for the products of that build?

Yes, it works well buildah build --layers --cache-stages --stage-labels stores all the layers of the Containerfile, correctly labels them and subsequent plain buildah build --layers reuses those by hitting cache - those layers remain intact, labels are preserved. Are we okay with such behavior?

I had to think about this for a bit, and I'm not sure it's what we want. Much like earlier versions of build --label, adding the new labels when we commit, without having their presence show up in the history, prevents the cache evaluation logic from "seeing" them.
For the --label CLI flag, we eventually reworked the build so that it internally appended a LABEL instruction with its arguments to the set of build steps in executor.buildStage() before processing the steps for the stage, so that labels from the command line could be accounted for.
Would this use case be better served by doing something similar? The buildStage() method potentially appends a LABEL instruction to the end of the final stage, and potentially prepends an ENV instruction (for --env values) at the beginning of every stage, so we have options. Adding the stage label instruction at the start of the stage's instructions would cause cache misses and naturally provide the "fresh build, every time" behavior which I think we're aiming for. Adding it as an instruction at the end of the stage would allow every instruction before it to be a cache hit, if we want it the other way.

prevents the cache evaluation logic from "seeing" them.

I may be misunderstanding this concern. Our implementation explicitly disables cache lookup when --cache-stages is used (stage_executor.go:1260-1262):

if s.executor.cacheStages {
    checkForLayers = false
}

This ensures every build with --cache-stages is a fresh build, and caches new intermediate images with a new build UUID. Cache evaluation logic (search) for stages doesn't run at all when --cache-stages is used.

Yes, I was hoping to be able to remove this special-case check.

Regarding cache reuse scenarios: Scenario 1:

* Build 1: `--cache-stages` (without `--layers`) → creates non-layered intermediate images

* Build 2:  `--cache-stages` (without `--layers`) → CANNOT reuse (explicitly disabled cache lookup)

Scenario 2:

* Build 1: `--cache-stages` (without `--layers`) → creates non-layered intermediate images

* Build 2: (without --cache-stages and `--layers`) → CANNOT reuse (insufficient history for cache hit, if I understand logic correctly), but that's fine for our use case

Your understand matches mine, yes.

Scenario 3 (considering that podman has always --layers enabled):

* Build 1: `--cache-stages --layers` → creates layered intermediate images

* Build 2: `--layers` (without `--cache-stages`) → CAN reuse those layers (you're right - even when labeled, because of their absence in history - I'll expand this later*)

However, our actual use case (Konflux SBOM generation) will use --cache-stages without --layers, so the layered cache scenario isn't relevant for us. The --layers compatibility was added based on your earlier suggestion that it's podman's default, but our production workflow won't use it. The intermediate images from --cache-stages builds are intended for inspection (SBOM generation), not for cache reuse. We don't need subsequent builds to reuse them.

For the --label CLI flag...

*I see the architectural similarity, but there's a practical difference IMHO:

* --label: user-defined, cache evaluation is active (user applies different labels across builds = cache miss)

* --stage-labels: system metadata, cache evaluation is explicitly disabled (checkForLayers = false by mandatory preceeding --cache-stages)

There's really nothing enforcing that distinction, though, and I would advise against assuming otherwise when using the produced images.

Given that cache is disabled when --cache-stages is used, is there still a concern about labels (added by --stage labels, that cannot be used without --cache-stages) not being in build history?

Could you clarify what specific scenario you're concerned about where the current approach would fail?

Throwing the stage labels into the history makes the new feature's interaction with caching much easier to reason about - based on where in the parsed tree the new labels are added, we can predict how the produced images will interact with the cache evaluation logic in future builds, and I value that.