Only gain 64.39 on cifar100 using VPT

[RongchangLi]

In the paper, the result of VPT on Cifar100 is 78.8. But I repeoduce a worse results: 64.39.

Here is my command:
bash configs/VPT/VTAB/ubuntu_train_vpt_vtab.sh experiments/VPT/ViT-B_prompt_vpt_100.yaml ViT-B_16.npz
lr is 0.001, weight decay is 0.0001.

Here is the outputs:

{"train_lr": 1.0000000000000002e-06, "train_loss": 4.60572919845581, "test_loss": 4.605674954909313, "test_acc1": 1.0, "test_acc5": 5.06, "epoch": 0, "n_parameters": 1152100}
{"train_lr": 0.0009001000000000005, "train_loss": 3.097556209564209, "test_loss": 2.907223858410799, "test_acc1": 24.05, "test_acc5": 63.74, "epoch": 10, "n_parameters": 1152100}
{"train_lr": 0.0009144048842659077, "train_loss": 1.6018924713134766, "test_loss": 1.7854226706903191, "test_acc1": 52.95, "test_acc5": 84.68, "epoch": 20, "n_parameters": 1152100}
{"train_lr": 0.0008083889915582233, "train_loss": 1.1247423966725667, "test_loss": 1.485605075389524, "test_acc1": 60.25, "test_acc5": 88.44, "epoch": 30, "n_parameters": 1152100}
{"train_lr": 0.0006726752705214194, "train_loss": 1.0146915316581726, "test_loss": 1.5092519914047628, "test_acc1": 61.44, "test_acc5": 87.76, "epoch": 40, "n_parameters": 1152100}
{"train_lr": 0.0005205483257436735, "train_loss": 0.9674314936002095, "test_loss": 1.4829329569128495, "test_acc1": 62.81, "test_acc5": 88.66, "epoch": 50, "n_parameters": 1152100}
{"train_lr": 0.0003668994025105816, "train_loss": 0.8998952905337015, "test_loss": 1.4677390011051032, "test_acc1": 63.51, "test_acc5": 88.85, "epoch": 60, "n_parameters": 1152100}
{"train_lr": 0.00022676872796319536, "train_loss": 0.8787842233975728, "test_loss": 1.4812716230561462, "test_acc1": 63.59, "test_acc5": 89.04, "epoch": 70, "n_parameters": 1152100}
{"train_lr": 0.00011387326887403328, "train_loss": 0.8721011678377787, "test_loss": 1.46410808382155, "test_acc1": 64.24, "test_acc5": 89.54, "epoch": 80, "n_parameters": 1152100}
{"train_lr": 3.9264019367658406e-05, "train_loss": 0.8584762454032898, "test_loss": 1.4647658761543563, "test_acc1": 64.45, "test_acc5": 89.28, "epoch": 90, "n_parameters": 1152100}
{"train_lr": 1.0976769428005575e-05, "train_loss": 0.868200953801473, "test_loss": 1.4616762444942812, "test_acc1": 64.39, "test_acc5": 89.4, "epoch": 99, "n_parameters": 1152100}

Can you help to find the reasons?

[RongchangLi]

Sorry I did not explain that the ubuntu_train_vpt_vtab.sh is a file equals to slurm_train_vpt_vtab.sh.

I tried to set lr=1e-2, weight_decay=1e-4, the result improved to 71.21. If changing the number of prompts to 50 (maybe it's also used in your paper), the result is 73.57. But it is still much lower that 78.8 (reported in your paper).

So, could you please peovide the training details for VPT? It will help a lot~

[ZhuWenjie98]

I can use VPT official code and model imagenet21k_ViT-B_16.npz from https://github.com/jeonsworld/ViT-pytorch (the link from VPT codebse License) to reproduce the VPT results of cifar100. However, it seems that they replace backbone to ViT-B_16.npz from google, I'm confused why the results of using two types of ViT-B are so different?

[haoweiz23]

Sorry I did not explain that the ubuntu_train_vpt_vtab.sh is a file equals to slurm_train_vpt_vtab.sh.

I tried to set lr=1e-2, weight_decay=1e-4, the result improved to 71.21. If changing the number of prompts to 50 (maybe it's also used in your paper), the result is 73.57. But it is still much lower that 78.8 (reported in your paper).

So, could you please peovide the training details for VPT? It will help a lot~

I met the same problem.