The result record in train.py has two items.

[BurningFr]

In train.py(L145), I find the code also records the best performance of test domain in logger.save_best().

Could you point out which result the paper uses, is it the best performance of test domain (test_res.max()) or the best validation performance (test_res[idx_best])?

Thanks a lot!

[Justinhzy]

Hi, it's the best validation result. Fix the random seed, and run multiple times to report the average.

[BurningFr]

Hi, thank you for answering my question!

I find there are some missing files in ./models (alexnet.py). I run the 'train.py' on my server and the reproduction results cannot achieve as high as the paper released (PACS dataset on resnet18). If I miss something important? I have annotated some codes in class 'JigsawNewDataset' which are for jigsaw data generate process to run RSC model.

My reproduction results is 79.45% for Sketch and 80.22% for Art Painting.

[Justinhzy]

I have uploaded my environment. Could you try to run the code in that environment? I attach some results I just got for Sketch and Art painting.
Thanks for pointing out the legacy code in class 'JigsawNewDataset'.

sketch
Dataset size: train 5446, val 616, test 3929
Step size: 24
0/42 of epoch 1/30 class : 2.189 - acc class : 7.03 [bs:128]
30/42 of epoch 1/30 class : 0.479 - acc class : 87.50 [bs:128]
Accuracies on val: class : 88.96
Accuracies on test: class : 64.77
New epoch - lr: 0.008
0/42 of epoch 2/30 class : 1.116 - acc class : 58.59 [bs:128]
30/42 of epoch 2/30 class : 0.335 - acc class : 92.19 [bs:128]
Accuracies on val: class : 92.05
Accuracies on test: class : 66.56
New epoch - lr: 0.008
0/42 of epoch 3/30 class : 0.646 - acc class : 72.66 [bs:128]
30/42 of epoch 3/30 class : 0.299 - acc class : 94.53 [bs:128]
Accuracies on val: class : 90.75
Accuracies on test: class : 69.92
New epoch - lr: 0.008
0/42 of epoch 4/30 class : 0.513 - acc class : 71.88 [bs:128]
30/42 of epoch 4/30 class : 0.434 - acc class : 83.59 [bs:128]
Accuracies on val: class : 93.51
Accuracies on test: class : 57.19
New epoch - lr: 0.008
0/42 of epoch 5/30 class : 0.324 - acc class : 89.84 [bs:128]
30/42 of epoch 5/30 class : 0.235 - acc class : 94.53 [bs:128]
Accuracies on val: class : 96.59
Accuracies on test: class : 71.21
New epoch - lr: 0.008
0/42 of epoch 6/30 class : 0.156 - acc class : 98.44 [bs:128]
30/42 of epoch 6/30 class : 0.208 - acc class : 92.97 [bs:128]
Accuracies on val: class : 93.83
Accuracies on test: class : 77.50
New epoch - lr: 0.008
0/42 of epoch 7/30 class : 0.243 - acc class : 92.97 [bs:128]
30/42 of epoch 7/30 class : 0.255 - acc class : 91.41 [bs:128]
Accuracies on val: class : 95.13
Accuracies on test: class : 72.94
New epoch - lr: 0.008
0/42 of epoch 8/30 class : 0.183 - acc class : 95.31 [bs:128]
30/42 of epoch 8/30 class : 0.173 - acc class : 94.53 [bs:128]
Accuracies on val: class : 94.48
Accuracies on test: class : 76.25
New epoch - lr: 0.008
0/42 of epoch 9/30 class : 0.107 - acc class : 98.44 [bs:128]
30/42 of epoch 9/30 class : 0.282 - acc class : 89.06 [bs:128]
Accuracies on val: class : 94.32
Accuracies on test: class : 77.76
New epoch - lr: 0.008
0/42 of epoch 10/30 class : 0.152 - acc class : 94.53 [bs:128]
30/42 of epoch 10/30 class : 0.081 - acc class : 98.44 [bs:128]
Accuracies on val: class : 95.94
Accuracies on test: class : 75.03
New epoch - lr: 0.008
0/42 of epoch 11/30 class : 0.092 - acc class : 97.66 [bs:128]
30/42 of epoch 11/30 class : 0.246 - acc class : 91.41 [bs:128]
Accuracies on val: class : 95.45
Accuracies on test: class : 78.62
New epoch - lr: 0.008
0/42 of epoch 12/30 class : 0.084 - acc class : 98.44 [bs:128]
30/42 of epoch 12/30 class : 0.047 - acc class : 100.00 [bs:128]
Accuracies on val: class : 95.45
Accuracies on test: class : 72.36
New epoch - lr: 0.008
0/42 of epoch 13/30 class : 0.084 - acc class : 98.44 [bs:128]
30/42 of epoch 13/30 class : 0.146 - acc class : 96.09 [bs:128]
Accuracies on val: class : 95.78
Accuracies on test: class : 75.44
New epoch - lr: 0.008
0/42 of epoch 14/30 class : 0.036 - acc class : 100.00 [bs:128]
30/42 of epoch 14/30 class : 0.037 - acc class : 100.00 [bs:128]
Accuracies on val: class : 94.97
Accuracies on test: class : 73.63
New epoch - lr: 0.008
0/42 of epoch 15/30 class : 0.106 - acc class : 96.09 [bs:128]
30/42 of epoch 15/30 class : 0.083 - acc class : 98.44 [bs:128]
Accuracies on val: class : 94.48
Accuracies on test: class : 79.05
New epoch - lr: 0.008
0/42 of epoch 16/30 class : 0.053 - acc class : 99.22 [bs:128]
30/42 of epoch 16/30 class : 0.035 - acc class : 100.00 [bs:128]
Accuracies on val: class : 93.99
Accuracies on test: class : 75.21
New epoch - lr: 0.008
0/42 of epoch 17/30 class : 0.064 - acc class : 98.44 [bs:128]
30/42 of epoch 17/30 class : 0.038 - acc class : 99.22 [bs:128]
Accuracies on val: class : 95.62
Accuracies on test: class : 74.62
New epoch - lr: 0.008
0/42 of epoch 18/30 class : 0.061 - acc class : 99.22 [bs:128]
30/42 of epoch 18/30 class : 0.031 - acc class : 99.22 [bs:128]
Accuracies on val: class : 94.97
Accuracies on test: class : 76.10
30/42 of epoch 19/30 class : 0.057 - acc class : 98.44 [bs:128]
Accuracies on val: class : 94.97
Accuracies on test: class : 77.22
New epoch - lr: 0.008
0/42 of epoch 20/30 class : 0.017 - acc class : 100.00 [bs:128]
30/42 of epoch 20/30 class : 0.023 - acc class : 100.00 [bs:128]
Accuracies on val: class : 95.13
Accuracies on test: class : 76.43
New epoch - lr: 0.008
0/42 of epoch 21/30 class : 0.049 - acc class : 96.88 [bs:128]
30/42 of epoch 21/30 class : 0.077 - acc class : 99.22 [bs:128]
Accuracies on val: class : 94.81
Accuracies on test: class : 79.97
New epoch - lr: 0.008
0/42 of epoch 22/30 class : 0.013 - acc class : 100.00 [bs:128]
30/42 of epoch 22/30 class : 0.100 - acc class : 98.44 [bs:128]
Accuracies on val: class : 94.16
Accuracies on test: class : 76.69
New epoch - lr: 0.008
0/42 of epoch 23/30 class : 0.013 - acc class : 100.00 [bs:128]
30/42 of epoch 23/30 class : 0.011 - acc class : 100.00 [bs:128]
Accuracies on val: class : 95.13
Accuracies on test: class : 78.72
New epoch - lr: 0.008
0/42 of epoch 24/30 class : 0.068 - acc class : 98.44 [bs:128]
30/42 of epoch 24/30 class : 0.017 - acc class : 100.00 [bs:128]
Accuracies on val: class : 95.29
Accuracies on test: class : 78.44
New epoch - lr: 0.0008
0/42 of epoch 25/30 class : 0.022 - acc class : 100.00 [bs:128]
30/42 of epoch 25/30 class : 0.027 - acc class : 99.22 [bs:128]
Accuracies on val: class : 96.59
Accuracies on test: class : 80.55
New epoch - lr: 0.0008
0/42 of epoch 26/30 class : 0.012 - acc class : 100.00 [bs:128]
30/42 of epoch 26/30 class : 0.014 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.75
Accuracies on test: class : 80.07
New epoch - lr: 0.0008
0/42 of epoch 27/30 class : 0.032 - acc class : 99.22 [bs:128]
30/42 of epoch 27/30 class : 0.059 - acc class : 98.44 [bs:128]
Accuracies on val: class : 96.27
Accuracies on test: class : 80.91
New epoch - lr: 0.0008
0/42 of epoch 28/30 class : 0.023 - acc class : 100.00 [bs:128]
30/42 of epoch 28/30 class : 0.014 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.75
Accuracies on test: class : 80.35
New epoch - lr: 0.0008
0/42 of epoch 29/30 class : 0.010 - acc class : 100.00 [bs:128]
30/42 of epoch 29/30 class : 0.047 - acc class : 99.22 [bs:128]
Accuracies on val: class : 96.75
Accuracies on test: class : 81.27
New epoch - lr: 0.0008
0/42 of epoch 30/30 class : 0.059 - acc class : 100.00 [bs:128]
30/42 of epoch 30/30 class : 0.047 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.43
Accuracies on test: class : 80.33
Best val 0.967532, corresponding test 0.812675 - best test: 0.812675, best epoch: 28
It took 557.549

art_painting
Dataset size: train 7137, val 806, test 2048
Step size: 24
0/55 of epoch 1/30 class : 3.264 - acc class : 0.78 [bs:128]
30/55 of epoch 1/30 class : 0.754 - acc class : 78.91 [bs:128]
Accuracies on val: class : 89.21
Accuracies on test: class : 76.32
New epoch - lr: 0.008
0/55 of epoch 2/30 class : 0.825 - acc class : 66.41 [bs:128]
30/55 of epoch 2/30 class : 0.510 - acc class : 87.50 [bs:128]
Accuracies on val: class : 93.42
Accuracies on test: class : 73.29
New epoch - lr: 0.008
0/55 of epoch 3/30 class : 0.423 - acc class : 85.16 [bs:128]
30/55 of epoch 3/30 class : 0.330 - acc class : 93.75 [bs:128]
Accuracies on val: class : 93.18
Accuracies on test: class : 77.54
New epoch - lr: 0.008
0/55 of epoch 4/30 class : 0.277 - acc class : 93.75 [bs:128]
30/55 of epoch 4/30 class : 0.379 - acc class : 85.94 [bs:128]
Accuracies on val: class : 91.56
Accuracies on test: class : 76.03
New epoch - lr: 0.008
0/55 of epoch 5/30 class : 0.296 - acc class : 88.28 [bs:128]
30/55 of epoch 5/30 class : 0.253 - acc class : 94.53 [bs:128]
Accuracies on val: class : 94.54
Accuracies on test: class : 77.83
New epoch - lr: 0.008
0/55 of epoch 6/30 class : 0.176 - acc class : 96.88 [bs:128]
30/55 of epoch 6/30 class : 0.198 - acc class : 96.09 [bs:128]
Accuracies on val: class : 94.67
Accuracies on test: class : 77.44
New epoch - lr: 0.008
0/55 of epoch 7/30 class : 0.242 - acc class : 90.62 [bs:128]
30/55 of epoch 7/30 class : 0.200 - acc class : 95.31 [bs:128]
Accuracies on val: class : 95.16
Accuracies on test: class : 79.54
New epoch - lr: 0.008
0/55 of epoch 8/30 class : 0.125 - acc class : 98.44 [bs:128]
30/55 of epoch 8/30 class : 0.153 - acc class : 96.09 [bs:128]
Accuracies on val: class : 92.80
Accuracies on test: class : 75.05
New epoch - lr: 0.008
0/55 of epoch 9/30 class : 0.079 - acc class : 98.44 [bs:128]
30/55 of epoch 9/30 class : 0.089 - acc class : 99.22 [bs:128]
Accuracies on val: class : 93.80
Accuracies on test: class : 75.78
New epoch - lr: 0.008
0/55 of epoch 10/30 class : 0.115 - acc class : 96.88 [bs:128]
30/55 of epoch 10/30 class : 0.131 - acc class : 95.31 [bs:128]
Accuracies on val: class : 95.41
Accuracies on test: class : 78.56
New epoch - lr: 0.008
0/55 of epoch 11/30 class : 0.091 - acc class : 99.22 [bs:128]
30/55 of epoch 11/30 class : 0.113 - acc class : 96.88 [bs:128]
Accuracies on val: class : 94.54
Accuracies on test: class : 81.25
New epoch - lr: 0.008
0/55 of epoch 12/30 class : 0.109 - acc class : 96.88 [bs:128]
30/55 of epoch 12/30 class : 0.140 - acc class : 96.88 [bs:128]
Accuracies on val: class : 95.78
Accuracies on test: class : 74.61
New epoch - lr: 0.008
0/55 of epoch 13/30 class : 0.043 - acc class : 100.00 [bs:128]
30/55 of epoch 13/30 class : 0.038 - acc class : 100.00 [bs:128]
Accuracies on val: class : 92.31
Accuracies on test: class : 80.57
New epoch - lr: 0.008
0/55 of epoch 14/30 class : 0.060 - acc class : 98.44 [bs:128]
30/55 of epoch 14/30 class : 0.123 - acc class : 95.31 [bs:128]
Accuracies on val: class : 95.41
Accuracies on test: class : 76.76
New epoch - lr: 0.008
0/55 of epoch 15/30 class : 0.082 - acc class : 96.09 [bs:128]
30/55 of epoch 15/30 class : 0.046 - acc class : 99.22 [bs:128]
Accuracies on val: class : 95.53
Accuracies on test: class : 79.88
New epoch - lr: 0.008
0/55 of epoch 16/30 class : 0.040 - acc class : 98.44 [bs:128]
30/55 of epoch 16/30 class : 0.030 - acc class : 99.22 [bs:128]
Accuracies on val: class : 95.66
Accuracies on test: class : 80.57
New epoch - lr: 0.008
0/55 of epoch 17/30 class : 0.062 - acc class : 98.44 [bs:128]
30/55 of epoch 17/30 class : 0.092 - acc class : 98.44 [bs:128]
Accuracies on val: class : 95.78
Accuracies on test: class : 78.42
New epoch - lr: 0.008
0/55 of epoch 18/30 class : 0.029 - acc class : 100.00 [bs:128]
30/55 of epoch 18/30 class : 0.042 - acc class : 99.22 [bs:128]
Accuracies on val: class : 94.04
Accuracies on test: class : 73.34
New epoch - lr: 0.008
0/55 of epoch 19/30 class : 0.079 - acc class : 96.88 [bs:128]
30/55 of epoch 19/30 class : 0.116 - acc class : 95.31 [bs:128]
Accuracies on val: class : 94.54
Accuracies on test: class : 76.81
New epoch - lr: 0.008
0/55 of epoch 20/30 class : 0.034 - acc class : 99.22 [bs:128]
30/55 of epoch 20/30 class : 0.019 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.15
Accuracies on test: class : 79.10
New epoch - lr: 0.008
0/55 of epoch 21/30 class : 0.032 - acc class : 99.22 [bs:128]
30/55 of epoch 21/30 class : 0.022 - acc class : 99.22 [bs:128]
Accuracies on val: class : 95.41
Accuracies on test: class : 76.32
New epoch - lr: 0.008
0/55 of epoch 22/30 class : 0.042 - acc class : 98.44 [bs:128]
30/55 of epoch 22/30 class : 0.024 - acc class : 100.00 [bs:128]
Accuracies on val: class : 95.41
Accuracies on test: class : 81.64
New epoch - lr: 0.008
0/55 of epoch 23/30 class : 0.034 - acc class : 99.22 [bs:128]
30/55 of epoch 23/30 class : 0.013 - acc class : 100.00 [bs:128]
Accuracies on val: class : 95.78
Accuracies on test: class : 81.84
New epoch - lr: 0.008
0/55 of epoch 24/30 class : 0.009 - acc class : 100.00 [bs:128]
30/55 of epoch 24/30 class : 0.032 - acc class : 99.22 [bs:128]
Accuracies on val: class : 95.78
Accuracies on test: class : 78.08
New epoch - lr: 0.0008
0/55 of epoch 25/30 class : 0.019 - acc class : 99.22 [bs:128]
30/55 of epoch 25/30 class : 0.010 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.28
Accuracies on test: class : 81.88
New epoch - lr: 0.0008
0/55 of epoch 26/30 class : 0.025 - acc class : 100.00 [bs:128]
30/55 of epoch 26/30 class : 0.012 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.53
Accuracies on test: class : 81.88
New epoch - lr: 0.0008
0/55 of epoch 27/30 class : 0.013 - acc class : 100.00 [bs:128]
30/55 of epoch 27/30 class : 0.009 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.40
Accuracies on test: class : 82.37
New epoch - lr: 0.0008
0/55 of epoch 28/30 class : 0.028 - acc class : 100.00 [bs:128]
30/55 of epoch 28/30 class : 0.005 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.40
Accuracies on test: class : 81.10
New epoch - lr: 0.0008
0/55 of epoch 29/30 class : 0.035 - acc class : 100.00 [bs:128]
30/55 of epoch 29/30 class : 0.009 - acc class : 100.00 [bs:128]
Accuracies on val: class : 97.02
Accuracies on test: class : 82.81
New epoch - lr: 0.0008
0/55 of epoch 30/30 class : 0.014 - acc class : 100.00 [bs:128]
30/55 of epoch 30/30 class : 0.009 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.65
Accuracies on test: class : 83.20
Best val 0.970223, corresponding test 0.828125 - best test: 0.832031, best epoch: 28
It took 562.823

[BurningFr]

Thank you for sharing your results. I fix the environment the same with your 'environment.yml' and run the PACS dataset with resnet18 for 5 times.

best val

PACS	a	c	s	p	avg
1	78.7109	77.4744	79.6386	94.4311	82.56375
2	80.4199	79.3089	80.4276	94.7305	83.72173
3	79.8828	78.8823	77.8061	93.9521	82.63083
4	77.9297	78.2423	74.7773	94.6108	81.39003
5	81.2012	75	75.6936	94.8503	81.68628
avg	79.6289	77.78158	77.66864	94.51496	82.39852

best test

PACS	a	c	s	p	avg
1	81.5918	78.4983	80.1985	94.6707	83.73983
2	82.0801	80.5887	80.6821	95.1497	84.62515
3	81.6406	79.8208	78.3151	95.2695	83.7615
4	82.1777	81.2713	81.0384	94.9701	84.86438
5	81.2012	79.029	82.0565	95.0299	84.32915
avg	81.73828	79.84162	80.45812	95.01798	84.264

I find the results is not stable with random initialization.

[Justinhzy]

Hi, I run one experiment at a time in the server, and the best Val result usually appears at the last 5 epoch. It should be more stable and better.
I will try to fix the randomness anyway.

[BurningFr]

Got it! Thank you.