Meta-training results are low on Meta-dataset

[rafa-cxg]

Hi, thanks for your works!

However, when I meta-train on Meta-dataset, eval results reach plautou about 4 epoches and unexpectly low.
Here is the training command (on 4gpus):
main.py --output xxx --dataset meta_dataset --data-path /xxx/meta_dataset_h5 --num_workers 4 --base_sources ilsvrc_2012 --epoch 100 --lr 5e-4 --dist-eval --arch deit_small_patch16 --fp16

The log.txt is as following:
{"aircraft": 50.81870883305867, "cu_birds": 78.36959873835245, "dtd": 85.6595253944397, "fungi": 42.095450512568156, "ilsvrc_2012": 81.6119522412618, "omniglot": 69.60026222864786, "quickdraw": 58.54106171925863, "vgg_flower": 88.16416889826456, "n_ways": 15.921875, "n_imgs": 438.29168701171875, "acc1": 69.35759735107422, "acc5": 92.26686096191406, "loss": 1.2670416831970215, "acc_std": 8.292503356933594, "epoch": -1} {"train_lr": 1.0000000000000114e-06, "train_n_ways": 18.953, "train_n_imgs": 560.719, "train_loss": 0.8147808248996735, "test_aircraft": 50.759508816401166, "test_cu_birds": 78.65988321304322, "test_dtd": 85.62619215647379, "test_fungi": 42.7772557258606, "test_ilsvrc_2012": 82.3101764678955, "test_omniglot": 69.4898078918457, "test_quickdraw": 58.49048713048299, "test_vgg_flower": 88.0607219060262, "test_n_ways": 15.921875, "test_n_imgs": 438.29168701171875, "test_acc1": 69.5217514038086, "test_acc5": 92.32942962646484, "test_loss": 1.2561185359954834, "test_acc_std": 8.548413276672363, "epoch": 0, "n_parameters": 21665666, "best_test_acc": 69.5217514038086} {"train_lr": 1.0000000000000114e-06, "train_n_ways": 18.433, "train_n_imgs": 562.5655, "train_loss": 0.7735185374133289, "test_aircraft": 51.15178027153015, "test_cu_birds": 78.97712380091349, "test_dtd": 85.60754156112671, "test_fungi": 43.50443981488546, "test_ilsvrc_2012": 82.86554934183756, "test_omniglot": 69.33503462473551, "test_quickdraw": 58.46635762850443, "test_vgg_flower": 88.00162099202474, "test_n_ways": 15.921875, "test_n_imgs": 438.29168701171875, "test_acc1": 69.73867797851562, "test_acc5": 92.42477416992188, "test_loss": 1.2484898567199707, "test_acc_std": 8.60382080078125, "epoch": 1, "n_parameters": 21665666, "best_test_acc": 69.73867797851562} {"train_lr": 5.799999999999955e-06, "train_n_ways": 18.791, "train_n_imgs": 559.0005, "train_loss": 0.7137562246918678, "test_aircraft": 51.35167574882507, "test_cu_birds": 79.78397973378499, "test_dtd": 85.69444621404013, "test_fungi": 44.82360962231954, "test_ilsvrc_2012": 84.14077679316203, "test_omniglot": 69.86654510498047, "test_quickdraw": 58.82178834279378, "test_vgg_flower": 87.92798795700074, "test_n_ways": 15.921875, "test_n_imgs": 438.29168701171875, "test_acc1": 70.30134582519531, "test_acc5": 92.64244842529297, "test_loss": 1.2236076593399048, "test_acc_std": 8.570518493652344, "epoch": 2, "n_parameters": 21665666, "best_test_acc": 70.30134582519531} {"train_lr": 1.0600000000000063e-05, "train_n_ways": 18.918, "train_n_imgs": 561.3185, "train_loss": 0.6610476011410356, "test_aircraft": 51.13846492767334, "test_cu_birds": 80.05027437210083, "test_dtd": 85.66587451299031, "test_fungi": 45.56454202334086, "test_ilsvrc_2012": 84.97727718353272, "test_omniglot": 70.65821135838827, "test_quickdraw": 58.66301472981771, "test_vgg_flower": 88.20873959859212, "test_n_ways": 15.921875, "test_n_imgs": 438.29168701171875, "test_acc1": 70.61579895019531, "test_acc5": 92.80927276611328, "test_loss": 1.2053977251052856, "test_acc_std": 8.310075759887695, "epoch": 3, "n_parameters": 21665666, "best_test_acc": 70.61579895019531} {"train_lr": 1.5399999999999832e-05, "train_n_ways": 18.6235, "train_n_imgs": 556.044, "train_loss": 0.6298330032750964, "test_aircraft": 51.13672161102295, "test_cu_birds": 80.03401034673055, "test_dtd": 85.48135102589926, "test_fungi": 45.545631710688276, "test_ilsvrc_2012": 85.2327094078064, "test_omniglot": 71.08325242996216, "test_quickdraw": 58.79606787363688, "test_vgg_flower": 88.12906545003256, "test_n_ways": 15.921875, "test_n_imgs": 438.29168701171875, "test_acc1": 70.67985534667969, "test_acc5": 92.89509582519531, "test_loss": 1.194559097290039, "test_acc_std": 8.290506362915039, "epoch": 4, "n_parameters": 21665666, "best_test_acc": 70.67985534667969} {"train_lr": 2.019999999999986e-05, "train_n_ways": 18.201, "train_n_imgs": 553.031, "train_loss": 0.603521515801549, "test_aircraft": 51.163675435384114, "test_cu_birds": 79.97277530034383, "test_dtd": 85.52420816421508, "test_fungi": 45.31500533421834, "test_ilsvrc_2012": 85.5517194112142, "test_omniglot": 71.11022526423136, "test_quickdraw": 58.554710388183594, "test_vgg_flower": 88.09597803751628, "test_n_ways": 15.921875, "test_n_imgs": 438.29168701171875, "test_acc1": 70.66104125976562, "test_acc5": 92.87564849853516, "test_loss": 1.1894696950912476, "test_acc_std": 8.291881561279297, "epoch": 5, "n_parameters": 21665666, "best_test_acc": 70.67985534667969} {"train_lr": 2.4852260087141618e-05, "train_n_ways": 18.973, "train_n_imgs": 557.8095, "train_loss": 0.6119356644563377, "test_aircraft": 51.21725977261861, "test_cu_birds": 79.87562297185262, "test_dtd": 85.37619231541952, "test_fungi": 45.25002218882243, "test_ilsvrc_2012": 85.66028550465902, "test_omniglot": 71.08135503133138, "test_quickdraw": 58.59248196283976, "test_vgg_flower": 88.00407594045004, "test_n_ways": 15.921875, "test_n_imgs": 438.29168701171875, "test_acc1": 70.63216400146484, "test_acc5": 92.87222290039062, "test_loss": 1.1859983205795288, "test_acc_std": 8.306001663208008, "epoch": 6, "n_parameters": 21665666, "best_test_acc": 70.67985534667969} {"train_lr": 2.4787447008744322e-05, "train_n_ways": 18.445, "train_n_imgs": 553.3545, "train_loss": 0.587374390617013, "test_aircraft": 51.12672254244487, "test_cu_birds": 79.85350618362426, "test_dtd": 85.43849369684855, "test_fungi": 45.33987803459168, "test_ilsvrc_2012": 85.78725407918294, "test_omniglot": 70.81301383972168, "test_quickdraw": 58.37208652496338, "test_vgg_flower": 88.05837783813476, "test_n_ways": 15.921875, "test_n_imgs": 438.29168701171875, "test_acc1": 70.59866333007812, "test_acc5": 92.83828735351562, "test_loss": 1.185286045074463, "test_acc_std": 8.268087387084961, "epoch": 7, "n_parameters": 21665666, "best_test_acc": 70.67985534667969} {"train_lr": 2.47110011432649e-05, "train_n_ways": 19.112, "train_n_imgs": 566.3605, "train_loss": 0.5799718932434916, "test_aircraft": 51.0446756362915, "test_cu_birds": 79.84302895863851, "test_dtd": 85.37817624409993, "test_fungi": 45.13587988217672, "test_ilsvrc_2012": 85.84268973668416, "test_omniglot": 70.57321942647299, "test_quickdraw": 58.300805997848514, "test_vgg_flower": 88.06614100138346, "test_n_ways": 15.921875, "test_n_imgs": 438.29168701171875, "test_acc1": 70.5230712890625, "test_acc5": 92.82667541503906, "test_loss": 1.1848111152648926, "test_acc_std": 8.225890159606934, "epoch": 8, "n_parameters": 21665666, "best_test_acc": 70.67985534667969}

I do not know what is wrong. Thanks for helps!

[hushell]

Sorry I don't keep track of the issues as this project is defocused for a while. I will find my training log later this week and upload here for you to compare. The results look normal as you only meta-train on ilsvrc_2012 and since DINO is pre-trained on the same domain, meta-training is expected to plateau early. Perhaps try to compare with the released ckpts to see if you can reproduce the ImageNet-only experiments. Many thanks!