What is the new ViT-L init checkpoint?

[RohanAsokan-Perceptyne]

In the paper, you talk about how DINOv2 ViT-L is the ideal model and that is what you end up using for all your checkpoints, but then what does the newly released checkpoint change? Is it a frozen untrained DINOv2 ViT-L Checkpoint? And, why does it have 0.1B params less?

[infinity1096]

Hi thanks for the question,

This new checkpoint is based on the DINO initialization trick in the DepthAnything V3 paper. Before, we initialized the encoder with DINOv2-L and trained the global attention from scratch. The DepthAnything V3 paper says not only you can initialize the encoder weights with DINO weights, you can also initialize the global attention transformer as well. So what you do is you take DINO weights and split it in half - use the first half for your encoder and the latter half for the decoder.

The way to understand is if you think what transformer layers are doing, they are learning how to effectively aggregate information from the previous layer through the attention mechanism. Hence, collecting information from 2 images (in the global attention) is a natural extension of collecting information within a single image.

The checkpoint you've seen is the result of this technique. So we used the first 12 layers of DINOv2-L for the encoder and the next 12 layers for the decoder, which made the model smaller than the original one (24 encoder + 12 global attention). We do finetune all layers and use a smaller learning rate because now the layers are somewhat initialized. However, with this trick the smaller model can actually reach similar performance as our old checkpoint.

We've also validated this trick in MapAnything. Highly recommended to have a try.

[infinity1096]

In case you want to have a try, here is the conversion script:

https://github.com/castacks/UniCeption/blob/main/scripts/convert_dino_to_info_sharing.py