Skip to content

SECRETS-ENV-VAR-RISK.md

Latest commit

 

History

History
221 lines (182 loc) · 9.9 KB

SECRETS-ENV-VAR-RISK.md

File metadata and controls

221 lines (182 loc) · 9.9 KB
Raw

Container Env-Var Introspection Risk (Audit C-MED-4)

This is a design note about how Containarium currently delivers tenant secrets into containers, the risk that comes with it, and the tmpfs-mount alternative we plan to offer as opt-in.

Today: env-var stamping

internal/server/secrets_server.go:stampSecretsOnLXC calls incus config set environment.<NAME>=<value> for every secret a tenant owns. The values land in the LXC's Incus config; on the next container start (or RefreshSecrets call) they become environment variables of the container's init process (PID 1) and inherit into every child process via execve(2).

The app code reads them as os.Getenv("OPENAI_API_KEY") or whatever's idiomatic in its language — convenient, language-agnostic, no SDK.

What "in env" means in practice

Once a value is in the container's environment, every process that can escalate to the same UID — and many that can't — can read it:

Reader Has access?
Any process running as the app UID Yes (/proc/self/environ)
Root inside the container Yes (/proc/<any-pid>/environ)
ps eww from a peer process Yes if same UID, no otherwise
Container init's child after fork Yes (inherited)
New exec inside the container Yes
Docker / podman-in-LXC sub-container Only if the operator explicitly forwards it
Operator on the daemon host Yes (incus config show)
Another tenant's container No (separate LXC, separate kernel namespaces)

The interesting columns for tenants are the first three. If an attacker gets RCE inside the container, even as a non-root user, they can read every secret the app process can read. The env vars are also visible in /proc/<pid>/environ to root — so a privileged daemon inside the same container (logging agent, supervisor) sees them too.

That's a wider surface than many operators assume. It does NOT cross tenant boundaries (one container can't read another's env), but it does NOT confine secrets within a single container either.

Threat model — what this DOES and DOES NOT protect

What's already protected (today):

  • Cross-tenant exposure: secrets are scoped per username in the secrets table, encrypted with AES-256-GCM at rest, and stamped into exactly one LXC. Another tenant's container can't read them.
  • Operator audit trail: every set_secret / get_secret is audit-logged.
  • At-rest encryption: the master key lives at /etc/containarium/secrets.key mode 0400, off the wire and not in any backup unless the operator explicitly arranges it.

What's NOT protected (audit C-MED-4):

  • Same-container introspection: any process inside the container that can read /proc/<pid>/environ sees every secret. That's the Linux env-var contract.
  • Docker-in-LXC sub-containers: the LXC has the env, but docker run / docker compose require explicit -e passthrough. Operators who forget the passthrough get a "missing env var" runtime error — visible failure mode, not silent.

Mitigations available today

For an operator who needs to harden this further without waiting for the tmpfs alternative:

  1. Don't put high-risk secrets in env vars. Use the secret-manager pattern: store a short-lived bootstrap token in env, have the app fetch the real secret from a vault on startup. The bootstrap token stays in env; the real secret never does.
  2. Drop privileges inside the container. A non-root app user with setuid semantics can read its own /proc/self/environ but not that of root daemons. (The reverse is also true — root can read everyone.)
  3. Audit incus config show access. Operators who can run that command see every container's secrets. Limit who has the incus-admin group on the daemon host.
  4. Rotate aggressively. The combination of Phase 1.2 (jti + revocation), Phase 1.6 (short-lived access + refresh rotation), and containarium secret set (which versions in place) means an exposed secret has a bounded blast radius.

Future: tmpfs-mount alternative (planned, not yet implemented)

The cleanest mitigation is to stop putting secrets in env entirely:

  • Mount a tmpfs into the container at e.g. /run/secrets/.
  • Write each decrypted secret to its own file, mode 0400, owned by the app user (not root).
  • App reads from the file instead of os.Getenv.

Properties:

  • Per-process access control: file permissions, not the env-var free-for-all. The app user can read; root can read; nobody else.
  • No execve inheritance: values aren't passed to child processes unless the app explicitly hands them off.
  • No incus config show leak: tmpfs contents aren't in the Incus config — only the mount point is.
  • Crash-safe disposal: tmpfs exists in kernel memory, evicted when the container stops. No risk of secrets persisting in container snapshots.

Costs:

  • App-side change: read files instead of env. Most apps that already support <SECRET>_FILE env-var conventions (Postgres password, Vault sidecars, Docker Compose secrets) work with no code change — set PGPASSWORD_FILE=/run/secrets/PGPASSWORD instead of PGPASSWORD=….
  • Operator UX: a second toggle on containarium secret set (--delivery=env|file, opt-in to file mode).
  • One more code path on the daemon (mount management, file rotation on RefreshSecrets).

The plan is to offer this as opt-in alongside the env-var path, not as a replacement. Operators choose per-tenant or per-secret. The default stays env-stamping for backwards compatibility; new deployments are encouraged to use file mode for high-risk values.

Status: shipped through Phase B-2

  • Phase A (PR #274): the delivery field is plumbed through proto / Store / CLI. Operators can set it via containarium secrets set <user> <NAME> <value> --delivery file.
  • Phase B-1 (PR #275): stampSecretsOnLXC dispatches per-secret. delivery="file" rows are written to /run/secrets/<NAME> mode 0400. Since /run is tmpfs on every systemd distro, file-mode secrets inherit the in-memory ephemeral-disposal property for free — the tmpfs evaporates when the container stops.
  • Phase B-2 (this PR): tenant ownership. The /run/secrets directory is now 0750 root:<username> and each file is 0440 root:<username>. The app process — running as the tenant user inside the container, the convention since CreateContainer — can read its own secrets without sudo. Fallback: if the chown fails (early-boot race, user not in /etc/passwd), the file stays mode 0400 root and the operator sees a WARNING log line.

Wire shape inside the container:

$ ls -ld /run/secrets
drwxr-x---. 2 root alice 60 /run/secrets

$ ls -l /run/secrets
-r--r-----. 1 root alice  43 OPENAI_API_KEY
-r--r-----. 1 root alice 128 DATABASE_URL

The app running as alice can cat /run/secrets/OPENAI_API_KEY; no other in-container user can.

  • Phase B-3 (this PR): periodic reconciler. The daemon ticks every 60s, asks the Store which tenants have file-mode secrets, and re-stamps each one whose container is Running. A bare incus restart not routed through the daemon now self-heals within the tick interval. Skipped on every tick:
    • tenants with no file-mode secrets (env-mode rows survive restart natively via incus config);
    • containers that are Stopped (next StartContainer will re-stamp).

The reconciler is owned by the daemon alongside the autosleep manager; same shape, same lifetime. Stamp is idempotent (writing the same content over the same path is a no-op at the bytes level), so periodic re-stamps cost nothing when state is already correct.

compose delivery — reaching nested docker / docker-compose

env and file delivery both miss one common app shape: a docker-compose stack running inside the LXC. A docker container does not inherit the LXC's Incus-config environment, and it doesn't see /run/secrets/<NAME> unless that path is explicitly mounted in. This is the same gap OTel monitoring hit (#370) and solved with a dotenv file consumed via env_file:.

--delivery compose closes it for secrets the same way:

  • The daemon renders the tenant's compose-mode secrets into a single dotenv file at /run/containarium/secrets.env, mode 0400 root, on /run (tmpfs — evicted on stop, never on the writable rootfs).
  • A compose service references it: env_file: /run/containarium/secrets.env. The compose runtime (root) reads the file at container-create time and injects the values as env into the app container — the app process never reads the file itself, so 0400 root is both safe and sufficient.

Exposure profile vs the other modes:

  • vs env: does NOT leak via incus config show (the dotenv file isn't in the Incus config). Same in-container exposure once injected.
  • vs file: one shared file instead of per-secret files; meant for compose ingestion, not per-process file ACLs.

Constraint: values must be single-line. A dotenv KEY=value line can't represent a newline, and the env_file: parser would mangle it — so a multi-line value is rejected at containarium secrets set --delivery compose time. Use --delivery file (/run/secrets/<NAME>) for multi-line blobs.

Built on the shared EnvFile delivery helper (#492); see pkg/core/container/{envfile,secrets_envfile}.go and the compose branch in stampSecretsOnLXC.

References

  • Audit finding C-MED-4 in ZERO-TRUST-AUDIT.md.
  • Implementation in internal/server/secrets_server.go.
  • Related: PR #248 (jti + revocation), PRs #254 / #255 (refresh rotation) — both shrink the blast radius of any exposed credential derived from an env-var leak.