Tracker: Improve HF Integration

[aflah02]

Issue to track support for reference config files and scripts to load in and export models from the following series (both with and without Transformer Engine). For now, I'll focus only on the dense offerings from each generation and only those models which can be constructed with existing components and don't require separate PRs to bring in.

Models Under Consideration:

Model Support Tracker

Qwen

Series	Config	Import Script	Export Script	Tests	Works with Transformer Engine	Remarks
Qwen 1	[ ]	[ ]	[ ]	[ ]	[ ]
Qwen 1.5	#1396	[ ]	[ ]	[ ]	[ ]
Qwen 2	#1397	[ ]	[ ]	[ ]	[ ]
Qwen 2.5	#1406	[ ]	[ ]	[ ]	[ ]
Qwen 3	#1399	[ ]	[ ]	[ ]	[ ]
Qwen 3 Next	[ ]	[ ]	[ ]	[ ]	[ ]	No dense offering. Only MoE
Qwen 3.5	[ ]	[ ]	[ ]	[ ]	[ ]	Uses linear attention in some layers. I don't think we have that implemented yet
Qwen 3.6	[ ]	[ ]	[ ]	[ ]	[ ]	Uses linear attention in some layers. I don't think we have that implemented yet

Llama

Series	Config	Import Script	Export Script	Tests	Works with Transformer Engine	Remarks
Llama 1	[x]	[x]	[x]	[ ]	[ ]
Llama 2	[x]	[x]	[x]	[ ]	[ ]
Llama 3	[ ]	[ ]	[ ]	[ ]	[ ]

OLMo

Series	Config	Import Script	Export Script	Tests	Works with Transformer Engine	Remarks
OLMo	#1407	[ ]	[ ]	[ ]	[ ]
OLMo2	#1408	[ ]	[ ]	[ ]	[ ]
OLMo3	[ ]	[ ]	[ ]	[ ]	[ ]

Gemma

Series	Config	Import Script	Export Script	Tests	Works with Transformer Engine	Remarks
Gemma	[ ]	[ ]	[ ]	[ ]	[ ]
Gemma 2	[ ]	[ ]	[ ]	[ ]	[ ]
Gemma 3	[ ]	[ ]	[ ]	[ ]	[ ]
Gemma 4	[ ]	[ ]	[ ]	[ ]	[ ]

Things to Include per Model:

• Scripts to load and export these models both with and without TE.
  • TE will not be a priority for now and will be added in a separate PR for each model as I need to research whether it changes save formats with FA4 now out.
• Tests to import a small test model with the import script and also a way to test export?
• Configs for different model sizes from each

Priority Order for Things -

• For each model there will be first a PR to add reference configs for users interested in just training the models
• Via separate PR after the configs I'll land support for export and import without TE
• Depending on whether TE makes sense I can then later add support with TE

Open to any feedback on the model choices, process of addition and tips on how to test export scripts.