fix: coerce unsloth target_modules for composite VLMs

[rohanprichard]

Summary

• patch_target_modules expands short LoRA target names (e.g. out_proj) into full module paths for composite VLMs, so standard PEFT can skip frozen submodules like the vision tower.
• Unsloth's get_peft_regex treats every target_modules entry as a leaf name and matches modules ending in it, so a full path never matches and no adapters get applied when training a VLM with use_unsloth: true.
• Added _coerce_unsloth_target_modules in src/llamafactory/model/model_utils/unsloth.py to reduce expanded paths back to unique leaf names before handing target_modules to unsloth, only for composite VLM model types.

Fixes #10662

Test plan

• Added unit tests in tests/model/model_utils/test_unsloth.py covering leaf-name passthrough, path coercion for composite VLMs, and non-composite models being left untouched.
• pytest -vv --import-mode=importlib tests/model/model_utils/test_unsloth.py

[gemini-code-assist]

Code Review

This pull request introduces a helper function _coerce_unsloth_target_modules to convert expanded VLM target modules back to leaf names for Unsloth, ensuring adapters are correctly applied to composite VLMs. It also adds corresponding unit tests. The review feedback highlights a potential issue where target_modules might be passed as a string or a set instead of a list, which would cause unexpected behavior during iteration, and provides a code suggestion to handle these types safely.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Thanks for the update, @rohanprichard. The addition of _coerce_unsloth_target_modules and the corresponding unit tests look correct and address the issue where expanded module paths prevented Unsloth from correctly identifying target layers in composite VLMs. The logic properly handles str, set, and list inputs, and correctly reduces the paths to leaf names only when necessary for composite models.

[rohanprichard]

Good catch @gemini-code-assist — added isinstance checks to coerce str and set inputs to list before the iteration. Thanks for the review!

[gemini-code-assist]

Glad to hear that, @rohanprichard! The isinstance checks are a great addition for robustness. Thanks for implementing those improvements.