Skip to content

Add $(python,...) built-in#47

Merged
jserv merged 1 commit intomainfrom
portable-command
Mar 1, 2026
Merged

Add $(python,...) built-in#47
jserv merged 1 commit intomainfrom
portable-command

Conversation

@jserv
Copy link
Contributor

@jserv jserv commented Feb 21, 2026

This introduces a portable $(python,...) preprocessor function that evaluates a Python code string in-process via exec(), returning "y" on success and "n" on exception. 'exec' namespace exposes os, sys, shutil, platform, and a shell-free run(*argv) helper for subprocess checks. Each call receives a fresh globals copy for namespace isolation.

Rewrite all six toolchain functions (cc-option, ld-option, as-instr, as-option, cc-option-bit, rustc-option) to use subprocess.Popen with argument lists (shell=False) and os.devnull, eliminating /dev/null, 2>/dev/null, and printf-pipe constructs. This closes the shell injection vector from environment variables and makes the functions work on Windows.

Add _run_argv() helper with timeout support and proper process cleanup. Remove dead _run_cmd_in_tmpdir(); replaced by TemporaryDirectory context managers in individual functions.

Add quote tracking to the macro expander so that commas and parentheses inside single, double, and triple-quoted strings are not treated as argument separators (required for $(python,assert "a,b" == "a,b")).

Close #41


Summary by cubic

Adds a portable $(python,...) built-in for in-process boolean checks and makes all Kbuild toolchain functions shell-free and Windows-compatible. Improves macro expansion with robust quote handling and adds subprocess timeouts. Closes #41.

  • New Features

    • Added $(python,code): execs in-process, returns y on success and n on exception; exposes os, sys, shutil, platform, and run(*argv) in a fresh namespace per call.
    • SystemExit(0/None/"") → y; non-zero/truthy → n. AssertionError is silent; other exceptions emit a warning with type and message.
    • Macro expander now tracks single/double/triple quotes and escapes; nested macros inside quotes still expand, and commas/parentheses inside quotes are not treated as separators.
    • Updated KBUILD.md with usage examples, portability guidance, and quoting behavior.
  • Refactors

    • Rewrote cc-option, ld-option, as-instr, as-option, cc-option-bit, and rustc-option to use subprocess.Popen with argv (shell=False) and os.devnull; removes shell injection risk and works on Windows.
    • Added _run_argv() with a 30s timeout and proper cleanup; _run_cmd now uses DEVNULL and a timeout. Exposed shell-free run() to $(python,...) code.
    • Removed _run_cmd_in_tmpdir(); callers use TemporaryDirectory. Expanded tests to cover $(python,...) and quote tracking.

Written for commit 4c399d9. Summary will update on new commits.

cubic-dev-ai[bot]

This comment was marked as resolved.

This introduces a portable $(python,...) preprocessor function that
evaluates a Python code string in-process via exec(), returning "y" on
success and "n" on exception. 'exec' namespace exposes os, sys, shutil,
platform, and a shell-free run(*argv) helper for subprocess checks. Each
call receives a fresh globals copy for namespace isolation.

Rewrite all six toolchain functions (cc-option, ld-option, as-instr,
as-option, cc-option-bit, rustc-option) to use subprocess.Popen with
argument lists (shell=False) and os.devnull, eliminating /dev/null,
2>/dev/null, and printf-pipe constructs.  This closes the shell
injection vector from environment variables and makes the functions
work on Windows.

Add _run_argv() helper with timeout support and proper process cleanup.
Remove dead _run_cmd_in_tmpdir(); replaced by TemporaryDirectory
context managers in individual functions.

Add quote tracking to the macro expander so that commas and parentheses
inside single, double, and triple-quoted strings are not treated as
argument separators (required for $(python,assert "a,b" == "a,b")).

Close #41
@jserv
Copy link
Contributor Author

jserv commented Feb 21, 2026

@eastWillow , You can apply changes below against Mado:

-- Kconfig-old	2026-02-21 20:12:01.979993489 +0800
+++ Kconfig	2026-02-22 01:18:58.406339526 +0800
@@ -7,22 +7,26 @@
 menu "Toolchain Configuration"
 
 # Compiler detection using scripts/detect-compiler.py
-config COMPILER_TYPE
-    string
-    default "$(shell,scripts/detect-compiler.py 2>/dev/null || echo Unknown)"
-
 config CC_IS_EMCC
-    def_bool $(shell,scripts/detect-compiler.py --is Emscripten 2>/dev/null && echo y || echo n)
+    def_bool $(python,assert run(sys.executable, 'scripts/detect-compiler.py', '--is', 'Emscripten'))
 
 config CC_IS_CLANG
-    def_bool $(shell,scripts/detect-compiler.py --is Clang 2>/dev/null && echo y || echo n)
+    def_bool $(python,assert run(sys.executable, 'scripts/detect-compiler.py', '--is', 'Clang'))
 
 config CC_IS_GCC
-    def_bool $(shell,scripts/detect-compiler.py --is GCC 2>/dev/null && echo y || echo n)
+    def_bool $(python,assert run(sys.executable, 'scripts/detect-compiler.py', '--is', 'GCC'))
+
+# Derived from the boolean probes above -- no shell needed.
+config COMPILER_TYPE
+    string
+    default "Emscripten" if CC_IS_EMCC
+    default "Clang" if CC_IS_CLANG
+    default "GCC" if CC_IS_GCC
+    default "Unknown"
 
 # Cross-compilation support detection
 config CROSS_COMPILE_ENABLED
-    def_bool $(shell,test -n "$(CROSS_COMPILE)" && echo y || echo n)
+    def_bool $(python,assert os.environ.get('CROSS_COMPILE'))
 
 config CROSS_COMPILE_PREFIX
     string
@@ -41,32 +45,32 @@
 
 endmenu
 
-# Dependency detection using Kconfiglib shell function
+# Dependency detection via pkg-config (shell-free subprocess).
 # For Emscripten builds, libraries are provided via ports system (-sUSE_*)
-# and do not require host pkg-config detection
+# and do not require host pkg-config detection.
 
 config HAVE_SDL2
     bool
     default n if CC_IS_EMCC
-    default $(shell,pkg-config --exists sdl2 && echo y || echo n) if !CC_IS_EMCC
+    default $(python,assert run('pkg-config', '--exists', 'sdl2')) if !CC_IS_EMCC
 
 config HAVE_PIXMAN
     default n if CC_IS_EMCC
-    def_bool $(shell,pkg-config --exists pixman-1 && echo y || echo n) if !CC_IS_EMCC
+    def_bool $(python,assert run('pkg-config', '--exists', 'pixman-1')) if !CC_IS_EMCC
 
 config HAVE_LIBPNG
     bool
     default y if CC_IS_EMCC
-    default $(shell,pkg-config --exists libpng && echo y || echo n) if !CC_IS_EMCC
+    default $(python,assert run('pkg-config', '--exists', 'libpng')) if !CC_IS_EMCC
 
 config HAVE_LIBJPEG
     bool
     default y if CC_IS_EMCC
-    default $(shell,pkg-config --exists libjpeg && echo y || echo n) if !CC_IS_EMCC
+    default $(python,assert run('pkg-config', '--exists', 'libjpeg')) if !CC_IS_EMCC
 
 config HAVE_CAIRO
     default n if CC_IS_EMCC
-    def_bool $(shell,pkg-config --exists cairo && echo y || echo n) if !CC_IS_EMCC
+    def_bool $(python,assert run('pkg-config', '--exists', 'cairo')) if !CC_IS_EMCC
 
 choice
     prompt "Backend Selection"

Determine if the above resolves Windows regressions.

@eastWillow
Copy link

eastWillow commented Mar 1, 2026

The integration of these PR changes and the provided mado configs can fix the current issues on Windows.

Windows Version

Edition	Windows 11 Home
Version	24H2
Installed on	‎3/‎1/‎2026
OS build	26100.7840
Experience	Windows Feature Experience Pack 1000.26100.291.0

Before

(venv3.13) PS C:\Users\bobga\Documents\mado\configs> menuconfig
Kconfig:12: warning: 'scripts/detect-compiler.py 2>/dev/null || echo Unknown' wrote to stderr: The system cannot find the path specified.
Kconfig:15: warning: 'scripts/detect-compiler.py --is Emscripten 2>/dev/null && echo y || echo n' wrote to stderr: The system cannot find the path specified.
Kconfig:18: warning: 'scripts/detect-compiler.py --is Clang 2>/dev/null && echo y || echo n' wrote to stderr: The system cannot find the path specified.
Kconfig:21: warning: 'scripts/detect-compiler.py --is GCC 2>/dev/null && echo y || echo n' wrote to stderr: The system cannot find the path specified.
Kconfig:25: warning: 'test -n "" && echo y || echo n' wrote to stderr: 'test' is not recognized as an internal or external command,
operable program or batch file.
Kconfig:51: warning: 'pkg-config --exists sdl2 && echo y || echo n' wrote to stderr: 'pkg-config' is not recognized as an internal or external command,
operable program or batch file.
Kconfig:55: warning: 'pkg-config --exists pixman-1 && echo y || echo n' wrote to stderr: 'pkg-config' is not recognized as an internal or external command,
operable program or batch file.
Kconfig:60: warning: 'pkg-config --exists libpng && echo y || echo n' wrote to stderr: 'pkg-config' is not recognized as an internal or external command,
operable program or batch file.
Kconfig:65: warning: 'pkg-config --exists libjpeg && echo y || echo n' wrote to stderr: 'pkg-config' is not recognized as an internal or external command,
operable program or batch file.
Kconfig:69: warning: 'pkg-config --exists cairo && echo y || echo n' wrote to stderr: 'pkg-config' is not recognized as an internal or external command,
operable program or batch file.
Loaded configuration '.config'
No changes to save (for '.config')

After

(venv3.13) PS C:\Users\bobga\Documents\mado\configs> menuconfig
Loaded configuration '.config'
No changes to save (for '.config')

@jserv jserv merged commit 3755c7c into main Mar 1, 2026
9 checks passed
@jserv jserv deleted the portable-command branch March 1, 2026 03:15
jserv added a commit to sysprog21/rv32emu that referenced this pull request Mar 1, 2026
Replace all $(shell,...) calls with the portable $(python,...) preprocessor
function from Kconfiglib.  The $(python,...) built-in evaluates Python code
in-process and returns "y"/"n" without spawning a shell, eliminating POSIX
shell dependencies (2>/dev/null, ||, echo) for cross-platform portability.

Simple checks (SDL2, SDL2_mixer) are inlined directly using the run() and
shutil.which() helpers available in the $(python,...) namespace.  Complex
checks (compiler type, LLVM 18, RISC-V toolchain) delegate to detect-env.py
via run(sys.executable, ...) with exit-code signaling.  COMPILER_TYPE is
derived from the CC_IS_CLANG/CC_IS_GCC/CC_IS_EMCC booleans using pure
Kconfig conditionals, eliminating the last $(shell,...) call.

detect-env.py changes:
- Boolean flags now use exit codes (0/1) via bool_exit() instead of
  printing "y"/"n", matching the run() convention in $(python,...).
- Compiler probing is lazy: --have-sdl2, --have-llvm18, etc. no longer
  pay the cost of running CC --version when they don't need it.
- get_compiler_version() captures both stdout and stderr to handle
  emcc variants that emit version info on stderr.

Reference: sysprog21/Kconfiglib#47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Windows] ValueError: too many values to unpack (expected 3) when loading configs

2 participants