Where is "factorized code" process?

[amazingspidy]

Hello,
Thanks for your great works!

I'm just wondering where is "Factorized code" process.
You answered about it in issue #27 but, I cannot find "Factorized code" in your code.

If I understand correctly, your vqvae code should employ a method similar to the following technique, but it appears you've simply set nb_code = 8192 and code_dim=32 in a standard VQ-VAE model.

[exitudio]

Hi, thank you for your interest in our work.

Yes. The idea is to increase the number of codes from 512 to 8,192 while reducing the embedding dimension from 512 to 32. This keeps the overall information bottleneck exactly the same, but encourages the model to store information in the code indices rather than in the embeddings themselves. Since the second stage (the Transformer) predicts code indices, this setup provides a much larger and more diverse prediction space.