feat: Add optional hermetic Python toolchain support

[jlojosnegros]

Description

This PR adds an optional workaround for projects using hermetic Python toolchains that encounter rules_python placeholder expansion errors.

Problem

Projects using hermetic Python toolchains (e.g., via @python3_12_host//:python or similar custom hermetic interpreters) encounter errors when running bazel run :refresh_compile_commands:

python3: can't open file '%interpreter_args%': [Errno 2] No such file or directory

Root Cause: The default py_binary implementation relies on rules_python's template placeholder expansion mechanism. When projects specify custom hermetic Python interpreters, placeholders like %interpreter_args% and %stage2_bootstrap% may fail to expand correctly, preventing the tool from running.

Solution

This PR introduces an optional use_hermetic_python_workaround parameter that enables an alternative implementation using sh_binary with a bash wrapper template instead of py_binary. This bypasses rules_python's templating entirely while maintaining full functionality.

The traditional py_binary approach remains the default for 100% backward compatibility.

Usage

Projects experiencing hermetic Python issues can opt-in to the workaround:

refresh_compile_commands(
    name = "refresh_compile_commands",
    targets = {"//...": ""},
    use_hermetic_python_workaround = True,  # Enable workaround
)

Projects without hermetic Python issues continue working without any changes.

Implementation Details

New files:

• refresh_wrapper.sh.template: Bash wrapper that locates and executes the Python script from Bazel's runfiles directory, bypassing rules_python templating

Modified files:

• BUILD: Export the new template file
• README.md: Document the new parameter, problem description, and solution
• refresh.template.py: Add if __name__ == "__main__": main() for direct execution
• refresh_compile_commands.bzl:
  • Add use_hermetic_python_workaround boolean parameter (default False)
  • Implement _expand_wrapper_template rule for template expansion
  • Add conditional logic to use sh_binary when workaround is enabled

How the workaround works:

1. Instead of py_binary, creates an sh_binary target
2. Expands refresh_wrapper.sh.template with the Python script name
3. Wrapper script searches for the Python script in multiple Bazel runfiles locations
4. Validates Python version (3.6+) before execution
5. Executes Python script directly without rules_python involvement

Testing

Successfully tested with Envoy proxy (~1200 C++ files, hermetic Python toolchain @python3_12_host//:python):

• Generated compile_commands.json successfully (16MB, 1233 entries)
• No placeholder expansion errors
• Full clangd/LSP functionality working

Test branch: jlojosnegros/envoy:test/hedron-hermetic-python-integration

This branch demonstrates:

• Complete WORKSPACE configuration
• BUILD target with use_hermetic_python_workaround = True
• CI integration script
• Successful generation workflow

Backward Compatibility

100% backward compatible

• Default behavior unchanged (use_hermetic_python_workaround = False)
• Existing users continue using py_binary without modifications
• No breaking changes to API or functionality
• Opt-in only for affected projects

Trade-offs

Pros:

• Solves hermetic Python compatibility issues
• Fully optional and backward compatible
• No changes required for unaffected users
• Maintains all existing functionality
• Clean separation of concerns

Cons:

• Adds alternative code path to maintain
• Additional template file to manage
• Slightly more complex implementation

Related Issues

This PR addresses the root cause of:

• ModuleNotFoundError: No module named 'orjson.orjson' #165 (Hermetic Python support discussion)
• Out of memory #245 (rules_python compatibility issues)
• [Self-filed] Switch back to rules_python when their issues are fixed #168 (Previous hermetic Python revert)

Future Considerations

If rules_python improves hermetic toolchain support in the future, this workaround could be deprecated. However, it provides immediate value for projects blocked by the current limitation.

Checklist

• Implementation follows existing code patterns
• Backward compatible (default behavior unchanged)
• Documentation added to README
• Tested with real-world project (Envoy)
• No breaking changes
• Optional feature (opt-in via parameter)

---

Additional Notes

This PR takes a conservative approach by:

1. Making the workaround optional (not forcing all users to switch)
2. Keeping the default behavior unchanged
3. Adding clear documentation of when and why to use it
4. Providing a real-world test case for validation

I'm happy to make any adjustments based on maintainer feedback. Thank you for considering this contribution!

[phlax]

is this repo still supported? this looks like a fairly obvious win