Summary
Gemma3 models (including FunctionGemma-270M) produce incorrect outputs when run in FP16 precision through the Cactus inference engine. The root cause is FP16 overflow (values exceeding the 65504 max) at multiple points in the computation graph that destroy model outputs.
This issue is specific to FP16 and does not occur with BF16 or FP32, because BF16 supports the same exponent range as FP32 (max ~3.4e38) while FP16 is limited to 65504.
References:
Why Gemma3 architecture is uniquely affected
- Very large RMSnorm weights -> activations after norm can easily overflow FP16 range
- Post norm (attention, pre/post ffn) -> multiple possible overflow points in each transformer block
- Gate projection can overflow before activation functions compress
- Large vocab
Analysis Chain
I first checked that the prompt/tool formatting were in agreement with the official FunctionGemma docs (see #501). Then I made sure, that the model conversion and LoRA merging worked as intended and that the correct tokenizer with the special Gemma tokens was used. Next, I scanned the implementation code to ensure that the model was implemented correctly.
When everything was in place (after some fixes), I verified the correctness of the model (with LoRA merged weights) by using the standard Hugging Face transformers library in Python. When loading the exact same model weights and prompt via AutoModelForCausalLM using torch.float16, the model instantly collapses into a loop. When loaded to torch.float32, the exact same script produces perfectly formatted tool-calling output.
Since Cactus mostly supports FP16 operations, I performed an activation analysis on all sensible operations (really useful debug feature on graph level btw). I observed the following (potential, not for all inputs) overflow locations:
- Residual stream after layer 5+
- RMSnorm outputs: overflows when weight values are large
- Gate and up projections produce values that, when multiplied together after GELU, exceed FP16 range
- Post-attention / post-MLP norms
- LM head
Why this is even worse in fine-tuned models (LoRA)
While base Gemma 3 models occasionally trip the 65k limit, this issue is more severe in fine-tuned models.
During fine-tuning, LoRA adapters specifically alter the q_proj, v_proj, and MLP matrices to forcefully steer the model toward JSON schemas and strict instruction following. This requires sharper, more extreme activation peaks to override the base model's tendencies. Because the fine-tuning process happens in standard PyTorch (which defaults to bfloat16 or float32 accumulation), the LoRA matrices learn to rely on variance spikes that are mathematically impossible to represent once exported and quantized down to a (mostly) float16 inference engine.
Summary
Gemma3 models (including FunctionGemma-270M) produce incorrect outputs when run in FP16 precision through the Cactus inference engine. The root cause is FP16 overflow (values exceeding the 65504 max) at multiple points in the computation graph that destroy model outputs.
This issue is specific to FP16 and does not occur with BF16 or FP32, because BF16 supports the same exponent range as FP32 (max ~3.4e38) while FP16 is limited to 65504.
References:
Why Gemma3 architecture is uniquely affected
Analysis Chain
I first checked that the prompt/tool formatting were in agreement with the official FunctionGemma docs (see #501). Then I made sure, that the model conversion and LoRA merging worked as intended and that the correct tokenizer with the special Gemma tokens was used. Next, I scanned the implementation code to ensure that the model was implemented correctly.
When everything was in place (after some fixes), I verified the correctness of the model (with LoRA merged weights) by using the standard Hugging Face transformers library in Python. When loading the exact same model weights and prompt via AutoModelForCausalLM using torch.float16, the model instantly collapses into a loop. When loaded to torch.float32, the exact same script produces perfectly formatted tool-calling output.
Since Cactus mostly supports FP16 operations, I performed an activation analysis on all sensible operations (really useful debug feature on graph level btw). I observed the following (potential, not for all inputs) overflow locations:
Why this is even worse in fine-tuned models (LoRA)
While base Gemma 3 models occasionally trip the 65k limit, this issue is more severe in fine-tuned models.
During fine-tuning, LoRA adapters specifically alter the q_proj, v_proj, and MLP matrices to forcefully steer the model toward JSON schemas and strict instruction following. This requires sharper, more extreme activation peaks to override the base model's tendencies. Because the fine-tuning process happens in standard PyTorch (which defaults to bfloat16 or float32 accumulation), the LoRA matrices learn to rely on variance spikes that are mathematically impossible to represent once exported and quantized down to a (mostly) float16 inference engine.