QAT Paper checklist

[DeepCowProductions]

list not complete!

Open TODOs:

• wandb data export and analysis scirpt #32

Possible Results:

• qat compare: fine-tune vs pure qat vs qat for layer by layer retrain #37
• qat layer exchange (fine tune) #38

[DeepCowProductions]

Paper outline:
motivate hardware
motivate architecture changes for the hardware
compare qat and quant techniques for our model
Experiments =>

1. qat train from scratch vs retrain with qat adapters
2. do the same fo norm exchange with models
3. validate behavior for at least one "real" large transformer