Strategy B3

[june6423]

Greetings!

I read your paper with great interest and am trying to reproduce some of your experiments.

I want to reproduce your vanilla KD setting using strategy B1, B2, B3 based on your DIST_KD paper.

I found B1 and B2 strategy on your strategies folder, but I couldn't find B3 setting.

configs/strategies/deit/deit_tiny.yaml appears to be B3, but I'm not sure, which leaves me with a question.

Could you give me B3 setting with vanilla KD with temperature 4?

[hunto]

Hi @june6423 ,

Our B3 experiment on Swin Transformer was implemented on the original training code of Swin-Transformer, so there's no b3 config in this repo.

Alternatively, if you want to implement B3 on this repo, the strategy is similar to deit_tiny, you can use the following config for KD (T=4):

aa: rand-m9-mstd0.5
batch_size: 128 # x 8 gpus = 1024bs
color_jitter: 0.4
decay_by_epoch: false
decay_epochs: 3
decay_rate: 0.967
# dropout
drop: 0.0
drop_path_rate: 0.2

epochs: 300
log_interval: 50
lr: 1.e-3
min_lr: 5.0e-06
model_ema: False
model_ema_decay: 0.999
momentum: 0.9
opt: adamw
opt_betas: null
opt_eps: 1.0e-08
clip_grad_norm: true
clip_grad_max_norm: 5.0

interpolation: 'bicubic'

# random erase
remode: pixel
reprob: 0.25

# mixup
mixup: 0.8
cutmix: 1.0
mixup_prob: 1.0
mixup_switch_prob: 0.5
mixup_mode: 'batch'

sched: cosine
seed: 42
warmup_epochs: 20
warmup_lr: 5.e-7
weight_decay: 0.04
workers: 16

# kd
kd: 'kd'
ori_loss_weight: 1.
kd_loss_weight: 1.
teacher_model: 'timm_swin_large_patch4_window7_224'
teacher_pretrained: True

[june6423]

Thanks a lot!

Now I want to reproduce results of other KD methods including RKD and CRD (I am working on your Table5 in DIST_KD paper, CIFAR 100)

But I failed to find training config and code for training from scratch and other KD methods.

I am working on image_classification_sota with d9662f7 version.

I am wondering if there is already published code to experiment with these settings, or if I should implement them myself.

Thanks for your effort.

[Malaika68]

Hi @june6423 , how did you manage to get the data from meta folder for ImageNet?

[june6423]

Hi @june6423 , how did you manage to get the data from meta folder for ImageNet?

I made it meta data file.

Make train.txt and val.txt in data/imagenet/meta folder.

Here's the example of train.txt. (File path and class number)

image/n01440764/n01440764_10026.JPEG 0
image/n01440764/n01440764_10027.JPEG 0

[Malaika68]

Hi @june6423, thanks for help. I did that but my validation accuracy is 0. Did you also face this thing? I am trying to distill the knowledge from resnet34 to resnet18.

[Malaika68]

Hi @june6423, can you please share your meta files with me?

[june6423]

Hi @june6423, can you please share your meta files with me?

Here's my code making meta files.

## Make validation meta file
import os

folder_dict = {}

path1 = os.path.join(os.getcwd(), "folderlist.txt")
basepath = "/workspace/DIST_KD/classification/data/imagenet/val/image"

with open(path1, "r") as f:
    foldername = f.readlines()

foldername = [fold.replace('\n','') for fold in foldername]
for idx, name in enumerate(foldername):
    folder_dict[name] = idx

dirs = os.listdir(basepath)
filename = []

for dir in dirs:
    cls = folder_dict.get(dir, -1)
    if cls < 0:
        breakpoint()
        continue
    filelist = os.listdir(os.path.join(basepath, dir))

    for file in filelist:
        filename.append(str("image/"+dir+"/"+file+" "+ str(cls)))

dest_path = os.path.join(os.getcwd(), "val.txt")

with open(dest_path, "w") as f:
    f.write('\n'.join(filename))

## Make train meta file
import os

folder_dict = {}

path1 = os.path.join(os.getcwd(), "folderlist.txt")
path2 = os.path.join(os.getcwd(), "filelist_train.txt")

with open(path1, "r") as f:
    foldername = f.readlines()

foldername = [fold.replace('\n','') for fold in foldername]
for idx, name in enumerate(foldername):
    folder_dict[name] = idx

with open(path2, "r") as f:
    filename = f.readlines()

filename = [file.replace('\n', '') for file in filename]

for idx, name in enumerate(filename):
    basename = name.split('_')[0]
    cls = folder_dict.get(basename, -1)
    if cls < 0:
        breakpoint()
        continue

    filename[idx] = str("image/"+basename+"/"+name + " " + str(cls))

dest_path = os.path.join(os.getcwd(), "train.txt")

with open(dest_path, "w") as f:
    f.write('\n'.join(filename))

## Make folderlist (Make sure your working on meta folder)
ls ../train/image > folderlist.txt

[Malaika68]

Thanks @june6423 for sending the code, what was your accuracy for baseline model of teacher and student
ResNet-18 student and ResNet-34 teacher.