Reduce RAM consumption when downdload/upload file to file_size*2#28
Reduce RAM consumption when downdload/upload file to file_size*2#28vetal4444 wants to merge 1 commit into
Conversation
… more then file_size*3
|
Just in case somebody is interested... I have created another fork here and included this PR respectively. |
Multi-GB GET/PUT held plaintext and ciphertext simultaneously and left the process RSS near peak long after each request, which tripped OOM kills on hosts with tight memory budgets. Drop the slice references between phases (post-decrypt, post-write, post-encrypt, post-upload) and call runtime/debug.FreeOSMemory past a 100 MiB threshold so freed pages actually return to the OS. Also stop forwarding an empty x-amz-server-side-encryption header on zero-byte PutObject responses; some upstreams (e.g. Hetzner Object Storage) return an empty algorithm there and strict SDK clients choke on the empty value. Refs #28
|
Thanks @vetal4444 — the RAM-reduction work landed in #41 (released as part of v1.5.0+ / current
Future RAM work. Even with these patches the proxy still buffers entire bodies in RAM — peak is roughly Closing this PR as integrated. The signal in your benchmark is what motivated the rewrite — really appreciated. |
The current implementation consumes about 3× the file size when uploading to S3 (tested with a 1 GB file, memory usage was ~3.2 GB). Downloading is even worse: s3proxy is consistently killed by the OS OOM killer when trying to download a 1 GB file from S3. At that moment the server had 5–6 GB of free RAM, and s3proxy consumed all of it.
With the changes from this PR, s3proxy consumes about 2× the file size for both uploads and downloads.