Skip to content

feat: BackendCpp: C++ source-generation backend for OSL shader groups#2130

Open
lgritz wants to merge 1 commit into
AcademySoftwareFoundation:mainfrom
lgritz:lg-backend-cpp
Open

feat: BackendCpp: C++ source-generation backend for OSL shader groups#2130
lgritz wants to merge 1 commit into
AcademySoftwareFoundation:mainfrom
lgritz:lg-backend-cpp

Conversation

@lgritz

@lgritz lgritz commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

Add a BackendCpp execution path that generates human-readable, compilable C++ from the post-optimized shader graph, as an alternative to the LLVM JIT.

The direct benefit is for debugging and understanding post-optimized shader by looking at its C++ equivalent rather than huge a pile of oso.

Possible secondary/future benefits (with a little more work):

  • Speed, if DSO loading is faster than optimize/JIT.
  • Pre-compiled shader networks for use in a renderer that needs a fixed set of shaders and no runtime JIT.
  • Distributing proprietary shaders as binaries.

But the REAL purpose is as a stepping stone to translating to other languages like Metal or GLSL. I envision the mechanism to simply be to subclass BackendCpp, inheriting all the parts that are common to all C++-like languages, and using virtual methods as the customization points for language-specific differences. (I don't claim that these customization points are correct now; I expect significant refactoring needs to be apparent when we add the next language.)

Anyway, so as for what's here:

Pipeline, controlled by the escalating debug_output_cpp ShadingSystem attribute (and the OSL_DEBUG_OUTPUT_CPP env var):
1 - generate a self-contained group-cpp-<name>.cpp
2 - also shell out to compile it to a DSO
3 - also load the DSO, verify its ABI, and execute it instead of the JIT
(running BackendLLVM's layout pass only, skipping full JIT codegen)

Generated output: a typed GroupData struct mirroring the BackendLLVM layout (layer-run flags, userdata, connected/output params), one void layer_N(ShaderGlobals*, GroupData*, ...) function per active layer, and a RunLLVMGroupFunc-compatible group entry. Every file includes the new internal osl_cpp_runtime.h (ABI version + extern "C" osl_* decls) and exports osl_cpp_abi_version(), checked at load. The lang_* virtual interface keeps the language tokens overridable for future backends.

Op coverage is complete: arithmetic/comparison/math emit direct C++; runtime-service ops (noise, texture, closures, transforms, pointcloud, getattribute/getmatrix/gettextureinfo, splines, regex, messages, ...) call osl_* functions. Control flow, arrays/components, structs (incl. color2/4, vector2/4), and Dual2 derivatives (scalar and triple) are all generated and verified against the JIT.

Testing is opt-out and automatic: the OSL_TEST_CPP_BACKEND CMake option (ON in the linux-vfx2026 and macos26-arm CI variants) builds a per-group DSO and runs every testshade/testrender test through the C++ path at both opt levels. The full eligible suite passes, matching JIT output (with the documented opt-outs: --entry/layers-entry, the backend-cpp fixture itself, OptiX/GPU, and batched-regression harnesses).

Cross-platform notes:

  • osl_* shadeops are exported from liboslexec for generated DSOs to resolve. On Linux this required listing osl_* under global: in hidesymbols.map (the version script otherwise localized them out of the dynamic symbol table); they are marked INTERNAL/UNSTABLE.
  • OSL_CPP_ABI_VERSION folds in the OSL major/minor version so minor releases are link-incompatible automatically (the DSOs are ephemeral; the check only guards against misuse).

Spec, plan, research, data-model, and tasks under docs/dev/specs/002-backend-cpp/.

Assisted-by: Claude Code / claude-sonnet-4-6 and claude-opus-4.8

Add a BackendCpp execution path that generates human-readable, compilable
C++ from the post-optimized shader graph, as an alternative to the LLVM
JIT.

The direct benefit is for debugging and understanding post-optimized
shader by looking at its C++ equivalent rather than huge a pile of oso.

Possible secondary/future benefits (with a little more work):
- Speed, if DSO loading is faster than optimize/JIT.
- Pre-compiled shader networks for use in a renderer that needs a
  fixed set of shaders and no runtime JIT.
- Distributing proprietary shaders as binaries.

But the REAL purpose is as a stepping stone to translating to other
languages like Metal or GLSL. I envision the mechanism to simply be to
subclass BackendCpp, inheriting all the parts that are common to all
C++-like languages, and using virtual methods as the customization
points for language-specific differences. (I don't claim that these
customization points are correct now; I expect significant refactoring
needs to be apparent when we add the next language.)

Anyway, so as for what's here:

Pipeline, controlled by the escalating `debug_output_cpp` ShadingSystem
attribute (and the `OSL_DEBUG_OUTPUT_CPP` env var):
  1 - generate a self-contained `group-cpp-<name>.cpp`
  2 - also shell out to compile it to a DSO
  3 - also load the DSO, verify its ABI, and execute it instead of the JIT
      (running BackendLLVM's layout pass only, skipping full JIT codegen)

Generated output: a typed `GroupData` struct mirroring the BackendLLVM
layout (layer-run flags, userdata, connected/output params), one
`void layer_N(ShaderGlobals*, GroupData*, ...)` function per active layer,
and a `RunLLVMGroupFunc`-compatible group entry. Every file includes the
new internal `osl_cpp_runtime.h` (ABI version + `extern "C"` osl_* decls)
and exports `osl_cpp_abi_version()`, checked at load. The `lang_*` virtual
interface keeps the language tokens overridable for future backends.

Op coverage is complete: arithmetic/comparison/math emit direct C++;
runtime-service ops (noise, texture, closures, transforms, pointcloud,
getattribute/getmatrix/gettextureinfo, splines, regex, messages, ...) call
osl_* functions. Control flow, arrays/components, structs (incl.
color2/4, vector2/4), and Dual2 derivatives (scalar and triple) are all
generated and verified against the JIT.

Testing is opt-out and automatic: the `OSL_TEST_CPP_BACKEND` CMake option
(ON in the linux-vfx2026 and macos26-arm CI variants) builds a per-group
DSO and runs every testshade/testrender test through the C++ path at both
opt levels. The full eligible suite passes, matching JIT output (with the
documented opt-outs: --entry/layers-entry, the backend-cpp fixture itself,
OptiX/GPU, and batched-regression harnesses).

Cross-platform notes:
  - osl_* shadeops are exported from liboslexec for generated DSOs to
    resolve. On Linux this required listing osl_* under `global:` in
    hidesymbols.map (the version script otherwise localized them out of
    the dynamic symbol table); they are marked INTERNAL/UNSTABLE.
  - OSL_CPP_ABI_VERSION folds in the OSL major/minor version so minor
    releases are link-incompatible automatically (the DSOs are ephemeral;
    the check only guards against misuse).
  - MSVC-clean (OSL::popcount instead of a GCC/Clang builtin).
  - Array copy of mismatched lengths copies min(dst,src) elements,
    matching BackendLLVM and avoiding an out-of-bounds source read.

Spec, plan, research, data-model, and tasks under
docs/dev/specs/002-backend-cpp/.

Assisted-by: Claude Code / claude-sonnet-4-6 and claude-opus-4.8

Signed-off-by: Larry Gritz <lg@larrygritz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant