Regarding multi-scale training VAE

[BOB-ZX]

During the training process of VQVAE, when training with VQ-4096cofig, there will be a state_dic mismatch problem. The pre-trained model is vit_base_patch14_dinov2 (using 14x14 patches), but our model definition might be based on 16x16 patches. Could you tell me how to handle it

[lxa9867]

Hi, sorry for the late reply. I think you may need to interpolate the positional encoding (which should already been implemented).

[BOB-ZX]

Hi, sorry for the late reply. I think you may need to interpolate the positional encoding (which should already been implemented).

Yes, I have solved the problem that the weights I downloaded were incorrect. But I can ask again if the training results of this xqgan can be used to replace the vqvae input of the VAR model. This is very important to me. I hope to get a reply. Thank you

[lxa9867]

Yes, if you want to directly use xqgan to train var I would recommend you to use the pq=1 setting so that you don’t need to modify var code to make it work.

[BOB-ZX]

Yes, if you want to directly use xqgan to train var I would recommend you to use the pq=1 setting so that you don’t need to modify var code to make it work.

Thank you very much for your reply. As a new graduate student, your reply has greatly boosted my confidence. However, when I put the training results of VQ-4096.yaml into the VAR model, it seems that the results do not run as expected. Is there something wrong with my approach? I hope to get an answer. Thank you!

[lxa9867]

Unfortunately, we didn't provide suitable Multiscale(MS)-Vector(V)-Residual(R10) model that is under the same setting of VAR. The only similar setting we provided pre-trained model is MSVR10P2 that used a PQ=2 which means VAR+parallel prediction token=2. The ImageFolder code in this repo uses the VAR generator (GPT2) and MSVR10P2 tokenizer which maybe useful to you.

[lxa9867]

In addition, please note that you can train your own MSVR10 tokenizer by changing the config. And then use that model for VAR training.

[BOB-ZX]

In addition, please note that you can train your own MSVR10 tokenizer by changing the config. And then use that model for VAR training.

Thank you for your answer. I will try to reproduce the VAR model using MSBR10P2-4096.yaml. Thank you sincerely!

[BOB-ZX]

Your work has greatly inspired me. But I'm sorry to disturb you so early. After training and debugging, I have gained a certain understanding of this job. However, this morning I found that in my training results from last night, the image patch effect of xqgan was very strong and didn't turn into a reasonable picture feeling. Why is that? Could you please explain to me where I did it wrong?