Hiera run inference slower than ViT?

[NoTody]

I'm testing model throughput on my machine and it seems like Hiera is much slower than ViT in my case. Shouldn't it be the case that Hiera will be faster based on the paper? I run the benchmark with the provided benchmarking.py code and here is the output (All are run on one A100 GPU):

model = hiera.hiera_base_224() 
benchmark(model, device=0, input_size=(3, 224, 224), batch_size=64, runs=40, use_fp16=True, verbose=True)

Throughput: 2095.55 im/s

model_name = 'vit_base_patch16_224_in21k' 
model = timm.create_model(model_name, pretrained=False).cuda() 
benchmark(model, device=0, input_size=(3, 224, 224), batch_size=64, runs=40, use_fp16=True, verbose=True)

Throughput: 3190.02 im/s

[2453828936]

I also want to know why it's slow. I found that Hiera inference is twice as slow as Vit when using an RTX 3090 on Windows.

[piyushkhanna7]

I am also observing lower throughput for Hiera compared to ViT when using the benchmarking.py script, even with FlashAttention enabled for Hiera.

    hiera_model = hiera.hiera_base_224()
    print("Hiera Model Parameters: ", count_parameters(hiera_model))
    benchmark(hiera_model, device=0, input_size=(3, 224, 224), batch_size=256, runs=40, use_fp16=True, verbose=True)

Results on Nvidia A100 80 GB GPU:
ViT (vit_base_patch16_224_in21k) Model Parameters: 102595923
Throughput: 3331.44 im/s

Hiera Model (with flash attn) Parameters: 51524968
Throughput (with flash attn): 2599.96 im/s

@chayryali @dbolya Could you please help debug why Hiera is infering slower despite having fewer parameters? Any insights would be appreciated. Thanks!