Fix TOCTOU races and symlink following in opal_os_dirpath - #14257
Open
jsquyres wants to merge 4 commits into
Open
Fix TOCTOU races and symlink following in opal_os_dirpath#14257jsquyres wants to merge 4 commits into
jsquyres wants to merge 4 commits into
Conversation
CodeQL (GitHub code scanning alerts 105 and 106) flagged the two
stat-then-chmod sequences in opal_os_dirpath_create(): between the
stat() that inspects an existing directory and the chmod() that
adjusts its mode, the path can be swapped (e.g., for a symlink to
another file), redirecting the chmod to an object that was never
inspected. The only production paths that reach this code are the
session directories, whose names are fully predictable and whose
default root (/tmp) is world-writable, so the race is not merely
theoretical.
Rework the function:
* Try mkdir() first and treat EEXIST as the probe for an existing
directory, instead of stat-then-mkdir.
* When the directory already exists, open it with
O_DIRECTORY | O_NOFOLLOW and perform all inspection and mode
adjustment through the descriptor (fstat/fchmod), so the object
that is checked is the object that is modified. O_DIRECTORY
means a pre-existing *file* with the requested name is now an
error (previously it was silently chmod'ed and reported as
success); O_NOFOLLOW means a symlink planted at the final
component is refused rather than followed (following it would
make every subsequent check inspect the link's target -- which an
attacker can choose to be something the victim owns).
* Refuse to adopt an existing directory owned by another user
(with a new help message). The old code accepted any existing
path whose mode bits happened to be a superset of the request --
for a predictable session-directory name under /tmp, that meant
a pre-created, attacker-owned 0777 directory was accepted
silently. No production caller depends on adopting foreign
directories; the session-directory paths are per-user by
construction.
The per-component tree-walk loop gets the same treatment for its
final component. Intermediate components keep a plain stat()
existence probe: they need only exist, there is no check/modify pair
to protect, and (unlike open()) stat() does not require read
permission on the component, so traverse-only (e.g. 0711)
intermediate directories continue to work.
Signed-off-by: Jeff Squyres <jeff@squyres.com>
CodeQL (GitHub code scanning alert 104) flagged the stat-then-unlink sequence in opal_os_dirpath_destroy(). Beyond that flagged pair, the whole function was path-based: the stat() classifying each entry followed symlinks (so a symlink inside a session directory pointing at a directory outside the tree would send the recursive destroy through the link), and even with lstat() an entry could be swapped between classification and the subsequent opendir()/unlink() on the rebuilt path. Rework the traversal to be descriptor-relative: the directory is opened once (O_DIRECTORY | O_NOFOLLOW -- a symlinked base path is now refused outright), iterated with fdopendir()/readdir(), and every entry is classified with fstatat(AT_SYMLINK_NOFOLLOW) and removed with unlinkat() -- or, for subdirectories, opened with openat(O_NOFOLLOW) and recursed into via the descriptor, then removed with unlinkat(AT_REMOVEDIR). The object that is classified is therefore always the object that is acted on; symlinks are unlinked (removing the link, never its target), and an entry swapped for a symlink between classification and descent is refused by O_NOFOLLOW instead of being followed. All functions used (fdopendir, fstatat, openat, unlinkat) are POSIX.1-2008. Semantics are otherwise preserved: the per-entry callback still vetoes removal, a directory found in non-recursive mode still produces OPAL_ERROR while files are removed, a vanished entry is still skipped silently, and the base directory is removed at the end only if it is empty. Signed-off-by: Jeff Squyres <jeff@squyres.com>
check_file() -- the callback that decides which files
opal_os_dirpath_destroy() may remove from the session directories --
built its path with:
fullpath = opal_os_path(false, &fullpath, root, path, NULL);
opal_os_path() is variadic over string components, so &fullpath (a
char **, uninitialized at that point) was dereferenced as the first
path component, producing a garbage path. The subsequent stat()
result was also never checked, so the st.st_size test that decides
whether an "output-" file is kept read an uninitialized structure:
the keep/delete decision was effectively random.
Build the path from root and path only, and treat a failed stat()
like an empty file (allow removal).
Signed-off-by: Jeff Squyres <jeff@squyres.com>
Cover the behavior changes from the TOCTOU rework in make check:
* opal_os_dirpath_create() on a pre-existing regular file is an
error (previously chmod'ed and reported as success)
* opal_os_dirpath_create() on a symlink -- even one pointing at a
directory the caller owns -- is refused, and the link target is
untouched
* opal_os_dirpath_destroy() removes a symlink found inside the
tree without following it: the link is unlinked, and the target
directory and its contents survive
* opal_os_dirpath_destroy() refuses a symlink as its base path,
leaving the link target and its contents untouched
* non-recursive destroy of a directory containing a subdirectory
removes the files, preserves the subdirectory, and returns an
error
* a callback veto on a file inside a subdirectory preserves the
file and its parent directory while the rest of the tree is
removed, and the overall destroy still succeeds
Signed-off-by: Jeff Squyres <jeff@squyres.com>
jsquyres
force-pushed
the
codeql-toctou-dirpath
branch
from
August 2, 2026 15:03
bbfe433 to
d69841e
Compare
Member
Author
|
@rhc54 I did not check, but these might apply to PMIx and/or PRTE. Might want to wait for these to get reviewed before applying them. |
Member
|
@jsquyres merge conflict |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This resolves the last three open CodeQL code scanning alerts — the
cpp/toctou-race-conditionfindings inopal/util/os_dirpath.c(104, 105, 106) — plus adjacent bugs found while auditing the callers. Context that shaped the fix: the only production caller ofopal_os_dirpath_create()/destroy()is the singleton/direct-launch session-directory setup inompi_rte.c(undermpirun, PMIx supplies the directories), always mode 0700, with a fully predictable name (ompi.<node>.<pid>.<uid>) under a world-writable default root (/tmp) — so the check-then-use races there are not just theoretical. (Updated after an AI-assisted local review pass, whose findings are folded in.)Commit 1 — rework
opal_os_dirpath_create()(alerts 105, 106):mkdir()first withEEXISTas the existence probe, and all inspection/mode adjustment of an existing directory done through a descriptor (open(O_DIRECTORY | O_NOFOLLOW)/fstat/fchmod) so the object checked is the object modified. Three deliberate behavior changes, all hardening: a pre-existing file at the requested path is now an error (previously it was chmod'ed and reported as success); a symlink at the requested path is refused rather than followed (following one would make the ownership check validate the link's target — which an attacker can choose to be something the victim owns); and an existing directory owned by another user is refused with a new help message (previously any pre-created directory whose mode was a superset of the request — e.g. an attacker-owned 0777 directory at the predictable session path — was silently adopted). Intermediate path components keep a plainstat()existence probe, so traverse-only (0711) intermediate directories continue to work.Commit 2 — rework
opal_os_dirpath_destroy()with descriptor-relative traversal (alert 104): the old traversal was path-based end to end — thestat()classifying each entry followed symlinks (so a symlink inside a session directory pointing outside the tree would send the recursive destroy through the link), and any entry could be swapped between classification and the subsequentopendir()/unlink()on the rebuilt path. The traversal is now descriptor-relative throughout: the base is opened once (O_DIRECTORY | O_NOFOLLOW— a symlinked base path is refused outright), entries are classified withfstatat(AT_SYMLINK_NOFOLLOW)and removed withunlinkat(), and subdirectories are entered viaopenat(O_NOFOLLOW)on the held descriptor and removed withunlinkat(AT_REMOVEDIR). The object classified is always the object acted on; symlinks are unlinked, never followed. All functions used are POSIX.1-2008. Callback-veto, non-recursive, and empty-base-removal semantics are preserved.Commit 3 — session-dir cleanup callback fix (found during the caller audit):
check_file()inompi_rte.cpassed&fullpath— an uninitializedchar **— as a path component to the variadicopal_os_path(), producing a garbage path, and never checked thestat()result, so thest_sizetest deciding whether anoutput-file is preserved read uninitialized memory. The keep/delete decision during session-directory cleanup was effectively random.Commit 4 — regression tests, wired into
make check, covering every new behavior: create-on-file refusal, create-on-symlink refusal (target untouched), destroy not following an in-tree symlink (target and contents survive), destroy refusing a symlinked base, non-recursive destroy with a subdirectory (files removed, subdir preserved, error returned), and a callback veto inside a subdirectory (protected file and its parent survive, overall success).Verified with clean VPATH builds on both macOS (clang) and Linux (AlmaLinux 10, gcc 14): no new warnings, and the
test/util/opal_os_dirpathsuite (38 assertions) passes on both platforms. Backports: all four commits cherry-pick cleanly ontov6.0.x. Onv5.0.xthe three code commits apply cleanly; the test commit needs manual adaptation (the modernized test file does not exist on that branch). (Note:test/util/opal_os_create_dirpath.cis a dead source file — it still includesorte_config.hand is not referenced byMakefile.am; left alone here, but it could be deleted in a follow-up. The same TOCTOU patterns exist upstream in OpenPMIx'spmix_os_dirpath.c, which PRRTE also uses — and that is what creates session dirs undermpirun— so an upstream OpenPMIx PR should follow once this settles.)