Skip to content

Fix TOCTOU races and symlink following in opal_os_dirpath - #14257

Open
jsquyres wants to merge 4 commits into
open-mpi:mainfrom
jsquyres:codeql-toctou-dirpath
Open

Fix TOCTOU races and symlink following in opal_os_dirpath#14257
jsquyres wants to merge 4 commits into
open-mpi:mainfrom
jsquyres:codeql-toctou-dirpath

Conversation

@jsquyres

@jsquyres jsquyres commented Aug 2, 2026

Copy link
Copy Markdown
Member

This resolves the last three open CodeQL code scanning alerts — the cpp/toctou-race-condition findings in opal/util/os_dirpath.c (104, 105, 106) — plus adjacent bugs found while auditing the callers. Context that shaped the fix: the only production caller of opal_os_dirpath_create()/destroy() is the singleton/direct-launch session-directory setup in ompi_rte.c (under mpirun, PMIx supplies the directories), always mode 0700, with a fully predictable name (ompi.<node>.<pid>.<uid>) under a world-writable default root (/tmp) — so the check-then-use races there are not just theoretical. (Updated after an AI-assisted local review pass, whose findings are folded in.)

Commit 1 — rework opal_os_dirpath_create() (alerts 105, 106): mkdir() first with EEXIST as the existence probe, and all inspection/mode adjustment of an existing directory done through a descriptor (open(O_DIRECTORY | O_NOFOLLOW)/fstat/fchmod) so the object checked is the object modified. Three deliberate behavior changes, all hardening: a pre-existing file at the requested path is now an error (previously it was chmod'ed and reported as success); a symlink at the requested path is refused rather than followed (following one would make the ownership check validate the link's target — which an attacker can choose to be something the victim owns); and an existing directory owned by another user is refused with a new help message (previously any pre-created directory whose mode was a superset of the request — e.g. an attacker-owned 0777 directory at the predictable session path — was silently adopted). Intermediate path components keep a plain stat() existence probe, so traverse-only (0711) intermediate directories continue to work.

Commit 2 — rework opal_os_dirpath_destroy() with descriptor-relative traversal (alert 104): the old traversal was path-based end to end — the stat() classifying each entry followed symlinks (so a symlink inside a session directory pointing outside the tree would send the recursive destroy through the link), and any entry could be swapped between classification and the subsequent opendir()/unlink() on the rebuilt path. The traversal is now descriptor-relative throughout: the base is opened once (O_DIRECTORY | O_NOFOLLOW — a symlinked base path is refused outright), entries are classified with fstatat(AT_SYMLINK_NOFOLLOW) and removed with unlinkat(), and subdirectories are entered via openat(O_NOFOLLOW) on the held descriptor and removed with unlinkat(AT_REMOVEDIR). The object classified is always the object acted on; symlinks are unlinked, never followed. All functions used are POSIX.1-2008. Callback-veto, non-recursive, and empty-base-removal semantics are preserved.

Commit 3 — session-dir cleanup callback fix (found during the caller audit): check_file() in ompi_rte.c passed &fullpath — an uninitialized char ** — as a path component to the variadic opal_os_path(), producing a garbage path, and never checked the stat() result, so the st_size test deciding whether an output- file is preserved read uninitialized memory. The keep/delete decision during session-directory cleanup was effectively random.

Commit 4 — regression tests, wired into make check, covering every new behavior: create-on-file refusal, create-on-symlink refusal (target untouched), destroy not following an in-tree symlink (target and contents survive), destroy refusing a symlinked base, non-recursive destroy with a subdirectory (files removed, subdir preserved, error returned), and a callback veto inside a subdirectory (protected file and its parent survive, overall success).

Verified with clean VPATH builds on both macOS (clang) and Linux (AlmaLinux 10, gcc 14): no new warnings, and the test/util/opal_os_dirpath suite (38 assertions) passes on both platforms. Backports: all four commits cherry-pick cleanly onto v6.0.x. On v5.0.x the three code commits apply cleanly; the test commit needs manual adaptation (the modernized test file does not exist on that branch). (Note: test/util/opal_os_create_dirpath.c is a dead source file — it still includes orte_config.h and is not referenced by Makefile.am; left alone here, but it could be deleted in a follow-up. The same TOCTOU patterns exist upstream in OpenPMIx's pmix_os_dirpath.c, which PRRTE also uses — and that is what creates session dirs under mpirun — so an upstream OpenPMIx PR should follow once this settles.)

CodeQL (GitHub code scanning alerts 105 and 106) flagged the two
stat-then-chmod sequences in opal_os_dirpath_create(): between the
stat() that inspects an existing directory and the chmod() that
adjusts its mode, the path can be swapped (e.g., for a symlink to
another file), redirecting the chmod to an object that was never
inspected.  The only production paths that reach this code are the
session directories, whose names are fully predictable and whose
default root (/tmp) is world-writable, so the race is not merely
theoretical.

Rework the function:

  * Try mkdir() first and treat EEXIST as the probe for an existing
    directory, instead of stat-then-mkdir.

  * When the directory already exists, open it with
    O_DIRECTORY | O_NOFOLLOW and perform all inspection and mode
    adjustment through the descriptor (fstat/fchmod), so the object
    that is checked is the object that is modified.  O_DIRECTORY
    means a pre-existing *file* with the requested name is now an
    error (previously it was silently chmod'ed and reported as
    success); O_NOFOLLOW means a symlink planted at the final
    component is refused rather than followed (following it would
    make every subsequent check inspect the link's target -- which an
    attacker can choose to be something the victim owns).

  * Refuse to adopt an existing directory owned by another user
    (with a new help message).  The old code accepted any existing
    path whose mode bits happened to be a superset of the request --
    for a predictable session-directory name under /tmp, that meant
    a pre-created, attacker-owned 0777 directory was accepted
    silently.  No production caller depends on adopting foreign
    directories; the session-directory paths are per-user by
    construction.

The per-component tree-walk loop gets the same treatment for its
final component.  Intermediate components keep a plain stat()
existence probe: they need only exist, there is no check/modify pair
to protect, and (unlike open()) stat() does not require read
permission on the component, so traverse-only (e.g. 0711)
intermediate directories continue to work.

Signed-off-by: Jeff Squyres <jeff@squyres.com>
CodeQL (GitHub code scanning alert 104) flagged the stat-then-unlink
sequence in opal_os_dirpath_destroy().  Beyond that flagged pair, the
whole function was path-based: the stat() classifying each entry
followed symlinks (so a symlink inside a session directory pointing
at a directory outside the tree would send the recursive destroy
through the link), and even with lstat() an entry could be swapped
between classification and the subsequent opendir()/unlink() on the
rebuilt path.

Rework the traversal to be descriptor-relative: the directory is
opened once (O_DIRECTORY | O_NOFOLLOW -- a symlinked base path is now
refused outright), iterated with fdopendir()/readdir(), and every
entry is classified with fstatat(AT_SYMLINK_NOFOLLOW) and removed
with unlinkat() -- or, for subdirectories, opened with
openat(O_NOFOLLOW) and recursed into via the descriptor, then removed
with unlinkat(AT_REMOVEDIR).  The object that is classified is
therefore always the object that is acted on; symlinks are unlinked
(removing the link, never its target), and an entry swapped for a
symlink between classification and descent is refused by O_NOFOLLOW
instead of being followed.

All functions used (fdopendir, fstatat, openat, unlinkat) are
POSIX.1-2008.  Semantics are otherwise preserved: the per-entry
callback still vetoes removal, a directory found in non-recursive
mode still produces OPAL_ERROR while files are removed, a vanished
entry is still skipped silently, and the base directory is removed
at the end only if it is empty.

Signed-off-by: Jeff Squyres <jeff@squyres.com>
check_file() -- the callback that decides which files
opal_os_dirpath_destroy() may remove from the session directories --
built its path with:

    fullpath = opal_os_path(false, &fullpath, root, path, NULL);

opal_os_path() is variadic over string components, so &fullpath (a
char **, uninitialized at that point) was dereferenced as the first
path component, producing a garbage path.  The subsequent stat()
result was also never checked, so the st.st_size test that decides
whether an "output-" file is kept read an uninitialized structure:
the keep/delete decision was effectively random.

Build the path from root and path only, and treat a failed stat()
like an empty file (allow removal).

Signed-off-by: Jeff Squyres <jeff@squyres.com>
Cover the behavior changes from the TOCTOU rework in make check:

  * opal_os_dirpath_create() on a pre-existing regular file is an
    error (previously chmod'ed and reported as success)

  * opal_os_dirpath_create() on a symlink -- even one pointing at a
    directory the caller owns -- is refused, and the link target is
    untouched

  * opal_os_dirpath_destroy() removes a symlink found inside the
    tree without following it: the link is unlinked, and the target
    directory and its contents survive

  * opal_os_dirpath_destroy() refuses a symlink as its base path,
    leaving the link target and its contents untouched

  * non-recursive destroy of a directory containing a subdirectory
    removes the files, preserves the subdirectory, and returns an
    error

  * a callback veto on a file inside a subdirectory preserves the
    file and its parent directory while the rest of the tree is
    removed, and the overall destroy still succeeds

Signed-off-by: Jeff Squyres <jeff@squyres.com>
@jsquyres
jsquyres force-pushed the codeql-toctou-dirpath branch from bbfe433 to d69841e Compare August 2, 2026 15:03
@jsquyres
jsquyres requested a review from bwbarrett August 2, 2026 15:12
@jsquyres

jsquyres commented Aug 2, 2026

Copy link
Copy Markdown
Member Author

@rhc54 I did not check, but these might apply to PMIx and/or PRTE. Might want to wait for these to get reviewed before applying them.

@hppritcha

Copy link
Copy Markdown
Member

@jsquyres merge conflict

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants