Improve non-interactive install where multiple packages match

[Thahun]

So far we used PECL and are now trying to replace it with Pie, but having some issues with automating it.

Just as a short background, due to our container structure we're at first building a base image and afterwards using that as a base to build more granular images for different purposes (e.g. app and cron). But they share the same PHP setup and to avoid redundant work and time, PHP is already installed in the base image. For now that was quite simple and we could simply use pecl install + docker-php-ext-enable to install and enable our PHP extensions like e.g. mongodb and redis. They were part of the base image and by that the setup was equal for all subsequent images. Next to that, build dependencies are deleted at the end of the build to keep the images smaller and the final containers are of course running as www-data so only the build has root permissions.
Of course we already use composer to install dependencies as well, so we already have them defined in the composer.json like

    "require": {
        "php": "~8.4.0",
        "ext-mongodb": "~2.1",
        "ext-redis": "~6.1",
        [...]
    }

Having the setup given, we tried to use Pie in a similar way and hit an issue with this non interactive approach.
Of course we could use Pie with hardcoded packages in the Dockerfile like:

pie install mongodb/mongodb-extension:~2.1
pie install phpredis/phpredis:~6.1
[...]

(Installing multiple extensions at once is not yet possible, but there's already a feature request for it: #466)

But we're more in favor of relying on the composer.json integration to avoid defining the extensions twice, cause for Composer they are anyways already defined. So just calling:

pie install [--allow-non-interactive-project-install]

And here's our issue, the resolution of ext-* to a package name is always triggering a prompt.
Even with --allow-non-interactive-project-install or --no-interaction the prompt remains (which is quite understandable, cause how should Pie be sure about which package is correct, except just using the suggested one automatically)
The only location we could determine where this mapping is persisted is the pie.lock in /root/.pie/php8.4_d2b6f37be78b1b629f281b53efb060b9/pie.lock but of course it's not designed to be modified.

So, the final question, is there any way to use Pie in a fully non interactive way, using the composer.json as a source?
At least I couldn't find anything in the documentation nor the command help.

The only working but unsafe way to achieve it, by automatically accepting option 1 of the prompts, seems to be currently

printf "%s\n" 1 | pie install --allow-non-interactive-project-install

[asgrim]

First a question for you:

For ext-mongodb and ext-redis, there should only be one option for either of those; so those two specific extensions shouldn't prompt when using --allow-non-interactive-project-install - could you check which specific extensions are being asked for in that case? To try to reproduce your issue, I built a Dockerfile with the following:

FROM ubuntu AS test_non_interactive
RUN apt-get update && apt-get install -y php unzip libbz2-dev gcc make autoconf bison re2c pkg-config libtool php-dev
COPY --from=build_pie_phar /app/pie.phar /usr/local/bin/pie
RUN mkdir /foo
WORKDIR /foo
RUN echo '{"require":{"php":"~8.3.0","ext-mongodb":"~2.1","ext-redis":"~6.1"}}' >> /foo/composer.json
RUN echo '{"_readme":["This file locks the dependencies of your project to a known state","Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies","This file is @generated automatically"],"content-hash":"201d95d4413cb1f999fba94555181bb0","packages":[],"packages-dev":[],"aliases":[],"minimum-stability":"stable","stability-flags":{},"prefer-stable":false,"prefer-lowest":false,"platform":{"php":"~8.3.0","ext-mongodb":"~2.1","ext-redis":"~6.1"},"platform-dev":{},"plugin-api-version":"2.6.0"}' >> /foo/composer.lock
RUN pie install --allow-non-interactive-project-install
RUN pie show

And this successfully installs the two missing extensions:

#19 [test_non_interactive 9/9] RUN pie show
#19 0.208 🥧 PHP Installer for Extensions (PIE) 1.3.5@a13873e, from The PHP Foundation
#19 0.322 You are running PHP 8.3.6
#19 0.340 Target PHP installation: 8.3.6 nts, on Linux/OSX/etc x86_64 (from /usr/bin/php8.3)
#19 0.391 Tip: to include extensions in this list that PIE does not manage, use the --all flag.
#19 0.574 
#19 0.574 Loaded PIE extensions:
#19 1.146   redis:6.3.0 (from 🥧 phpredis/phpredis:6.3.0 ✅)
#19 1.533   mongodb:2.1.4 (from 🥧 mongodb/mongodb-extension:2.1.4 ✅)
#19 DONE 1.5s

In this particular instance, it --allow-non-interactive-project-install should work just fine for you, I think?

However, yes; if there is more than one extension (maybe there is an extension not given in your example) that provides the ext-<extname> it will fail. In this circumstance, the issue is "how do we define that map" as you rightly point out. Using the pie.lock isn't going to work; as those instructions wouldn't exist in your "project" as such.

This is definitely something I'd like to improve on, but we need to find an effective way for PIE users to say "when I want ext-blah I mean extension foo/blah (as opposed to bar/blah or baz/blah-improved, etc.).

[Thahun]

Sorry for not providing the reproduceable example and wasting your time.

Cause now I discovered the issue that let us think we have an issue.

FROM ubuntu AS test_non_interactive
RUN apt-get update && apt-get install -y php unzip libbz2-dev gcc make autoconf bison re2c pkg-config libtool php-dev
COPY --from=ghcr.io/php/pie:bin /pie /usr/bin/pie
RUN mkdir /foo
WORKDIR /foo
RUN echo '{"require":{"php":"~8.3.0","ext-mongodb":"~2.1","ext-redis":"~6.1"}}' >> /foo/composer.json
RUN echo '{"_readme":["This file locks the dependencies of your project to a known state","Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies","This file is @generated automatically"],"content-hash":"201d95d4413cb1f999fba94555181bb0","packages":[],"packages-dev":[],"aliases":[],"minimum-stability":"stable","stability-flags":{},"prefer-stable":false,"prefer-lowest":false,"platform":{"php":"~8.3.0","ext-mongodb":"~2.1","ext-redis":"~6.1"},"platform-dev":{},"plugin-api-version":"2.6.0"}' >> /foo/composer.lock
WORKDIR /foo

docker build -t pie_test .
docker run -it pie_test

root@a220a1923045:/foo# pie install  --allow-non-interactive-project-install
🥧 PHP Installer for Extensions (PIE) 1.3.5, from The PHP Foundation
You are running PHP 8.3.6
Target PHP installation: 8.3.6 nts, on Linux/OSX/etc arm64 (from /usr/bin/php8.3)
Checking extensions for your project __root__ (path: /foo)
requires: ext-mongodb:~2.1 🚫 Missing

The following packages may be suitable, which would you like to install:
  [0] None
  [1] mongodb/mongodb-extension: MongoDB driver extension
 > 0
Okay I won't install anything for mongodb
requires: ext-redis:~6.1 🚫 Missing

The following packages may be suitable, which would you like to install:
  [0] None
  [1] phpredis/phpredis: A PHP extension for Redis
 > 0
Okay I won't install anything for redis

Finished checking extensions.

The whole issue is that the --allow-non-interactive-project-install option doesn't have any effect on an interactive shell, and by that the install command can't be manually tested. Or at least it behaves different and we didn't expect that. So we just thought it's not working as expected while infact it works on a real non-interactive shell, just not on an interactive shell.

Independet of that, yes it would be good to have some option to define a mapping of ext to package, cause pretty sure at some point there'll be extensions matching multiple packages. In best case this could be done directly in composer.json to avoid a second config, but even an additional config yaml and/or option for the install command would be fine from my point of view.

[asgrim]

No worries at all :) FWIW, you could use COMPOSER_NO_INTERACTION=1 pie install --allow-non-interactive-project-install to simulate a non-interactive shell (I think)

But yes, I would like to have some way of defining what extensions are picked, but I was hoping to shelve that problem for now 😆

[Thahun]

Yes, that works, I tried it. We just didn't have that in mind. Of course it's obvious when having the relation between Composer and Pie in mind that Composer environment variables should work as well, but maybe it's worth to add a hint to the documentation of --allow-non-interactive-project-install or even a general one if all Composer environment variables are useable in Pie as well.

Anyways, thanks for the help :)

[Ninos]

Are there already any ideas how to finally solve this? I created an issue for that with a possible solution:
#506

In my opinion there must be some mappings to the specific extensions (not "just* for security reasons...), if you want to implement such non-interactive mode (required for building images)

[asgrim]

For continuity/ease of discussion, the linked issue suggestion was:

Currently it's not possible to autoinstall pie just using composer.json/composer.lock file.

Current behavior

Create a composer.json file, e.g. with following content:

{
    "name": "1m9/php-build",
    "type": "example",
    "require": {
        "php": ">=8.0",
        "ext-apcu": "*",
        "ext-redis": "*"
    }
}

After that run composer install --ignore-platform-reqs to generate composer.lock file.

To install pie extensions, you need to run pie install in same folder. Pie is now (happily for security reasons) asking, which extension we want to install, e.g. apcu/apcu for ext-apcu.

Because of those user input requirements it's not possible to autoinstall pie extensions, e.g. in container environment.

Solution

Add an additional section, e.g. extensions, which is loading all relevant meta data for required extensions (not just the package & version, but also the additional pie flags (--with-xyz, ...).

Example

{
    "name": "1m9/php-build",
    "type": "example",
    "extensions": {
        "apcu/apcu": "*",
        "phpredis/phpredis": {
            "version": "*",
            "flags": ["--with-xyz", "--without-xyz", "--enable-xyz", "--whatever..."]
        }
    },
    "require": {
        "php": ">=8.0",
        "ext-apcu": "*",
        "ext-redis": "*"
    }
}

[asgrim]

@Ninos Yep, something like that could work in theory, but I need to discuss with the Composer team if it is feasible, or if we might need to take a different approach. I'm well aware of this issue; at the moment at least for now it shouldn't be a major issue, but certainly as forks or alternate approaches for extensions become available, this will certainly become a problem.

FWIW, in the interim a workaround if you need a specific extension where more than one match causes issues, or you need to provide specific flags would be something like:

RUN pie install phpredis/phpredis --with-xyz=foo  --enable-xyz \
 && pie install --allow-non-interactive-project-install # + other flags if necessary...

(the intention being: install phpredis/phpredis with specific configure flags, then install everything else automatically)

[Ninos]

I see, thx for clarification! Let me know, if there's something I can help with, but I think at least you have better connections to the composer team... :D

Also thx for the workaround, solved it with a custom script by looping through all extensions, think it's a bit easier to handle :-) For interested folks:
https://ego.codes/1m9/library/php-build/-/blob/main/bin/php-build?ref_type=heads#L146-157

Would just be awesome to see such thing in future, makes lots of things easier (no need to define extensions in Containerfile) 👼