Large memory use during extraction of pixi-pack executable archive

[matthewfeickert]

This is a follow up issue from a discussion with @pavelzw on the Prefix.dev Discord server 👍 where I observe the extraction of a pixi-pack executable archive taking 8 GB of memory.

Reproducer

I'm executing this on the following Ubuntu machine:

$ uname -srm
Linux 6.8.0-60-generic x86_64
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

The pixi-pack archive is created from https://github.com/UW-Madison-DSI/pixi-docker-chtc at 3cca38260c830f5a2b991bd430686e2fc8ba092c from the examples/hello_pytorch/ directory.

$ git clone git@github.com:UW-Madison-DSI/pixi-docker-chtc.git
$ cd pixi-docker-chtc/
$ git reset --hard 3cca38260c830f5a2b991bd430686e2fc8ba092c
$ cd examples/hello_pytorch/
$ ls -1ap pixi.*
pixi.lock
pixi.toml

I first install the conda-forge distribution of time which has additional functionality compared to some Linux distributions packed versions (or at least my Bash shell's time, which is why I'll be using ~/.pixi/bin/time -v for the rest of this)

$ pixi global install time
└── time: 1.9 (installed)
    └─ exposes: time

and then I use pixi-pack

$ pixi global install pixi-pack
└── pixi-pack: 0.7.1 (already installed)
    └─ exposes: pixi-pack
$ pixi pack --version
pixi-pack 0.7.1

to create an executable archive of the prod environment

$ ~/.pixi/bin/time -v pixi pack --environment prod --platform linux-64 --create-executable pixi.toml
⏳ Downloading 119 packages...
📦 Creating self-extracting executable
📥 Fetching pixi-unpack executable...
[00:00:02] ######################################## Download complete                                                                                                                                       ✅ pixi-unpack executable downloaded successfully
📦 Created pack at /home/feickert/Code/GitHub/uw-madison/pixi-docker-chtc/examples/hello_pytorch/environment.sh with size 4.11 GiB.
	Command being timed: "pixi pack --environment prod --platform linux-64 --create-executable pixi.toml"
	User time (seconds): 24.54
	System time (seconds): 34.92
	Percent of CPU this job got: 43%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 2:15.30
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 7593140
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 123
	Minor (reclaiming a frame) page faults: 1897408
	Voluntary context switches: 1477374
	Involuntary context switches: 23342
	Swaps: 0
	File system inputs: 20016
	File system outputs: 15080096
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
$ ls -lh environment.sh
-rwxr-xr-x 1 feickert feickert 4.2G Jul  3 21:44 environment.sh

So now that we have the archive let's drop into a Docker container and check the memory use of installing the prod environment with Pixi

$ docker pull ghcr.io/prefix-dev/pixi:latest
$ docker run --rm -ti -v $PWD:/read ghcr.io/prefix-dev/pixi:latest
root@f93f17e673ca:/#
root@f93f17e673ca:/# pixi --version
pixi 0.49.0
root@f93f17e673ca:/# mkdir pixi-install
root@f93f17e673ca:/# cp /read/pixi.* ./pixi-install/
root@f93f17e673ca:/# cd pixi-install/
root@f93f17e673ca:/pixi-install# export CONDA_OVERRIDE_CUDA=12.0  # force __cuda virtual package
root@f93f17e673ca:/pixi-install# /root/.pixi/bin/time -v pixi install --environment prod --locked
The prod environment has been installed.
        Command being timed: "pixi install --environment prod --locked"
	User time (seconds): 96.68
	System time (seconds): 29.60
	Percent of CPU this job got: 110%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:53.93
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 590192
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 128
	Minor (reclaiming a frame) page faults: 189794
	Voluntary context switches: 506880
	Involuntary context switches: 77207
	Swaps: 0
	File system inputs: 7008
	File system outputs: 19548056
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

and then compare that to extracting the pixi-pack archive

root@f93f17e673ca:/pixi-install# cd ..
root@f93f17e673ca:/# cp /read/environment.sh .
root@f93f17e673ca:/# /root/.pixi/bin/time -v ./environment.sh
⏳ Extracting and installing 119 packages to /tmp/.tmphnkotX/cache...
💫 Finished unpacking to /.
	Command being timed: "./environment.sh"
	User time (seconds): 35.17
	System time (seconds): 35.11
	Percent of CPU this job got: 104%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:07.02
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 8602240
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 84
	Minor (reclaiming a frame) page faults: 8386077
	Voluntary context switches: 2525036
	Involuntary context switches: 51531
	Swaps: 0
	File system inputs: 28888208
	File system outputs: 32469984
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
root@f93f17e673ca:/#

So looking at the maximum resident size as my metric of memory use, extracting the pixi-pack archive uses a significant amount of memory (comparatively).

pixi install	pixi-pack archive creation	pixi-pack archive extraction
0.6 GB	7.6 GB	8.6 GB

Please let me know if I'm missing something obvious!

[pavelzw]

the archive creation is because in the current implementation, the whole tarball is temporarily in RAM. we should write the tar directly via base64 to the executable file instead

without --create-executable this doesn't happen

[pavelzw]

in #210 we fixed the issue while packing. didn't investigate the extraction yet.

[e8035669]

According to this

without --create-executable this doesn't happen

I think the problem is not inside pixi-unpack, maybe is the header script?
The header script uising two external tool grep and sed to search and extract the payload, so I try to investigate the script line-by-line.

[jeff@jeff-inspiron5593 temp]$ ~/.pixi/bin/time -v grep -anm 1 "^@@END_HEADER@@" ../environment.sh
51:@@END_HEADER@@
        Command being timed: "grep -anm 1 ^@@END_HEADER@@ ../environment.sh"
        User time (seconds): 0.00
        System time (seconds): 0.00
        Percent of CPU this job got: 86%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 4084
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 137
        Voluntary context switches: 1
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

[jeff@jeff-inspiron5593 temp]$ ~/.pixi/bin/time -v grep -anm 1 "^@@END_ARCHIVE@@" ../environment.sh
54:@@END_ARCHIVE@@
        Command being timed: "grep -anm 1 ^@@END_ARCHIVE@@ ../environment.sh"
        User time (seconds): 1.52
        System time (seconds): 1.88
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.41
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 6436808
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 16988
        Voluntary context switches: 1
        Involuntary context switches: 21
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

[jeff@jeff-inspiron5593 temp]$ ~/.pixi/bin/time -v sed -n "53,53p" ../environment.sh > /dev/null
        Command being timed: "sed -n 53,53p ../environment.sh"
        User time (seconds): 0.64
        System time (seconds): 1.56
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.21
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 8606688
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 10765
        Voluntary context switches: 1
        Involuntary context switches: 64
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

[jeff@jeff-inspiron5593 temp]$ ~/.pixi/bin/time -v sed -n '56,$p' ../environment.sh > /dev/null
        Command being timed: "sed -n 56,$p ../environment.sh"
        User time (seconds): 0.55
        System time (seconds): 1.67
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.24
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 8606692
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 10765
        Voluntary context switches: 1
        Involuntary context switches: 89
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

According the result above, I think these lines will use lots of memory during the extraction when its is handing a large file.

pixi-pack/src/header.sh

Lines 29 to 41 in d6907de

	archive_end=$(grep -anm 1 "^@@END_ARCHIVE@@" "$0" | awk -F: '{print $1}')
	if [ -z "$archive_begin" ] || [ -z "$archive_end" ]; then
	echo "ERROR: Markers @@END_HEADER@@ or @@END_ARCHIVE@@ not found." >&2
	exit 1
	fi
	archive_begin=$((archive_begin + 2))
	archive_end=$((archive_end - 1))
	pixi_pack_start=$((archive_end + 2))
	sed -n "$archive_begin,${archive_end}p" "$0" | base64 -d > "$TEMPDIR/archive.tar"
	sed -n "$pixi_pack_start,\$p" "$0" | base64 -d > "$TEMPDIR/pixi-unpack"

[matthewfeickert]

(Sorry for not responding for months)

According the result above, I think these lines will use lots of memory during the extraction when its is handing a large file.

pixi-pack/src/header.sh

Lines 29 to 41 in d6907de

	archive_end=$(grep -anm 1 "^@@END_ARCHIVE@@" "$0" | awk -F: '{print $1}')
	if [ -z "$archive_begin" ] || [ -z "$archive_end" ]; then
	echo "ERROR: Markers @@END_HEADER@@ or @@END_ARCHIVE@@ not found." >&2
	exit 1
	fi
	archive_begin=$((archive_begin + 2))
	archive_end=$((archive_end - 1))
	pixi_pack_start=$((archive_end + 2))
	sed -n "$archive_begin,${archive_end}p" "$0" | base64 -d > "$TEMPDIR/archive.tar"
	sed -n "$pixi_pack_start,\$p" "$0" | base64 -d > "$TEMPDIR/pixi-unpack"

Is there any realistic solution or workaround for this then? Or will this fundamentally always use a large amount of memory with this kind of approach?