feat: support oxygenvlm model on mlu device.#1131
feat: support oxygenvlm model on mlu device.#1131phantomlei3 merged 2 commits intojd-opensource:mainfrom
Conversation
phantomlei3
requested review from
DongheJin,
JimHsiung,
RobbieLeung,
XuZhang99,
liutongxuan,
walsonyang and
yq33victor
as code owners
phantomlei3
force-pushed
the
support-oxygen-2
branch
from
bae8a08 to
d1e81e3
Compare
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the Oxygen VLM model, implementing its vision transformer, attention mechanisms, and causal language model components. It adds the OxygenVisionLayer, OxygenVisionAttention, and the OxygenvlmForConditionalGeneration class, while updating ModelArgs and enhancing model loading logic for better weight prefix compatibility. Feedback highlights a critical bug in OxygenVisionAttention where rotary embeddings are applied twice, and multiple compilation errors in oxygen_vlm.h caused by referencing an undefined mm_image_merge_size property. Furthermore, improvements are suggested regarding the use of const_cast on input parameters and the access modifiers for the apply_mrope method.
phantomlei3
force-pushed
the
support-oxygen-2
branch
from
9786ca6 to
e201729
Compare
Co-authored-by: chenxb002 <[email protected]>
4 participants
No description provided.