Skip to content

feat(runtime): add configurable rlimits (nofile/nproc) for jobs#73

Open
luthermonson wants to merge 1 commit into
mainfrom
feat/runtime-rlimits
Open

feat(runtime): add configurable rlimits (nofile/nproc) for jobs#73
luthermonson wants to merge 1 commit into
mainfrom
feat/runtime-rlimits

Conversation

@luthermonson
Copy link
Copy Markdown
Contributor

Summary

Adds a [runtime.rlimits] block to config.toml that lets operators raise the per-job container's RLIMIT_NOFILE and RLIMIT_NPROC above containerd's default 1024.

  • Defaults: nofile = 1024, nproc = 1024 — exactly what containerd's WithDefaultSpecForPlatform already produces, so an empty config is a no-behavior-change.
  • Why this matters: jobs (notably build tools doing ulimit -n 4096) were failing with Operation not permitted because raising the hard limit from inside the container needs CAP_SYS_RESOURCE, which containerCapabilities deliberately omits. Setting the hard limit higher in the OCI spec lets ulimit succeed without granting the cap.
  • Both soft and hard are set equal. Lowering is always allowed; raising still requires the cap we don't grant.

Implementation

  • pkg/config/config.go — new RuntimeConfig{Rlimits RuntimeRlimits{Nofile, Nproc int64}} with Resolved() filling defaults for zero/negative values.
  • pkg/runtime/runtime.go — new Rlimits config.RuntimeRlimits field on runtime.Config, wired into the OCI opts slice next to seccompOpts().
  • pkg/runtime/rlimits_linux.go — replaces (not appends) spec.Process.Rlimits so the containerd-default RLIMIT_NOFILE=1024 entry from WithDefaultSpecForPlatform doesn't end up duplicated.
  • pkg/runtime/rlimits_other.go — no-op for Windows / macOS. HCS-isolated Windows containers and Vz Linux/macOS VMs use different resource-limit models that are configured at the VM level, not in the OCI process spec.
  • cmd/ephemerd/main.go — passes Rlimits: cfg.Runtime.Rlimits.Resolved() at both runtime.New sites.

Usage

[runtime.rlimits]
nofile = 4096
nproc = 2048

Test plan

  • go test ./pkg/config/... — 6 new tests for Resolved() defaults, explicit values, negative fallback, mixed zero/explicit, TOML round-trip, omitted block.
  • go test ./pkg/runtime/... — 4 new linux-only tests covering applied values, defaults-when-zero, duplicate-rlimit avoidance, and nil-Process safety.
  • GOOS=linux ./bin/golangci-lint run ./... → 0 issues.
  • mage build:windows succeeds end-to-end (700 MB ephemerd.exe + embedded linux binary).
  • Verify on the active Hyper-V Linux VM host: redeploy ephemerd.exe with nofile = 4096 and confirm ulimit -n inside a job returns 4096 (no Operation not permitted from ulimit -n 2048 calls).

Local CI notes

mage lint and the build-side of mage test trip the documented miekg/pkcs11 cgo preprocessing failure on Windows hosts (AGENTS.md); lint and test compile clean under GOOS=linux, which is what the GitHub Actions CI runs.

Adds [runtime.rlimits] config block with `nofile` and `nproc` keys.
Defaults to 1024/1024 — same as containerd's built-in OCI spec — so an
empty config is a no-behavior-change. Higher values let build tools call
`ulimit -n N` up to the configured ceiling without needing
CAP_SYS_RESOURCE, which we deliberately don't grant.

The Linux spec helper replaces (not appends) `spec.Process.Rlimits` so
the containerd-default RLIMIT_NOFILE entry from
`oci.WithDefaultSpecForPlatform` doesn't end up duplicated alongside
our own. Both soft and hard are set equal; raising the hard limit from
inside the container requires CAP_SYS_RESOURCE which is intentionally
not in containerCapabilities.

On Windows and macOS the helper is a no-op — HCS and Vz use different
resource-limit models that are configured at the VM/utility-VM level.

Local CI notes: lint and tests pass cross-compiled for linux. `mage
test` on this Windows host hits the documented pkcs11/ocicrypt cgo
preprocessing failure for packages that transitively import it
(cmd/ephemerd, pkg/containerd, pkg/dind, pkg/workflow); GOOS=linux
go test -run xxx compiles them clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant