bench covering both NNUE and classical, PGO, CI #2931

vondele · 2020-08-07T19:59:08Z

bench covering both NNUE and classical, PGO, CI

Add additional options to the bench command

classical, NNUE, mixed.

Default mixed, examples:

```
./stockfish bench 16 1 13 default depth mixed
./stockfish bench 16 1 13 default depth classical
./stockfish bench 16 1 13 default depth NNUE
```

Requires the default net to be present,
download automatically if needed for profile-build.

requires fishtest work, and presumably travis work, before commit.

Bench: 3607702

This is a partial PR concerning NNUE CI and PGO builds. It modifies the appveyor script so it downloads the default NNUE net before the stockfish bench is run. No functional change.

Adjust travis to cover more archs, speedup instrumented testing, and download net before building to be able to compute hybrid bench. in preparation for better NNUE integration in CI. No functional change.

Add additional options to the bench command classical, NNUE, mixed. requires the default net to be present, can be obtained with ``` make net ``` Default bench is mixed, further examples: ``` ./stockfish bench 16 1 13 default depth mixed ./stockfish bench 16 1 13 default depth classical ./stockfish bench 16 1 13 default depth NNUE ``` Requires the default net to be present, download automatically if needed for profile-build. PGO gives a nice speedup on fishtest, passed STC: LLR: 2.93 (-2.94,2.94) {-0.50,1.50} Total: 3360 W: 469 L: 343 D: 2548 Ptnml(0-2): 20, 246, 1030, 356, 28 https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569 requires fishtest work, and travis work, before commit. Bench: 4290577

…rofile-build) of the NNUE part of the code. Joint work gvreuls / vondele * Download the default NNUE net in AppVeyor * Download net in travis CI `make net` * Adjust tests to cover more archs, speedup instrumented testing * Introduce 'mixed' bench as default, with further options: classical, NNUE, mixed. mixed (default) and NNUE require the default net to be present, which can be obtained with ``` make net ``` Further examples (first is equivalent to `./stockfish bench`): ``` ./stockfish bench 16 1 13 default depth mixed ./stockfish bench 16 1 13 default depth classical ./stockfish bench 16 1 13 default depth NNUE ``` The net is now downloaded automatically if needed for `profile-build` (usual `build` works fine without net present) PGO gives a nice speedup on fishtest: passed STC: LLR: 2.93 (-2.94,2.94) {-0.50,1.50} Total: 3360 W: 469 L: 343 D: 2548 Ptnml(0-2): 20, 246, 1030, 356, 28 https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569 passed LTC: LLR: 2.97 (-2.94,2.94) {0.25,1.75} Total: 8824 W: 609 L: 502 D: 7713 Ptnml(0-2): 8, 430, 3438, 519, 17 https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c closes official-stockfish#2931 fixes official-stockfish#2907 requires fishtest updates before commit Bench: 4290577

…rofile-build) of the NNUE part of the code. Joint work gvreuls / vondele * Download the default NNUE net in AppVeyor * Download net in travis CI `make net` * Adjust tests to cover more archs, speedup instrumented testing * Introduce 'mixed' bench as default, with further options: classical, NNUE, mixed. mixed (default) and NNUE require the default net to be present, which can be obtained with ``` make net ``` Further examples (first is equivalent to `./stockfish bench`): ``` ./stockfish bench 16 1 13 default depth mixed ./stockfish bench 16 1 13 default depth classical ./stockfish bench 16 1 13 default depth NNUE ``` The net is now downloaded automatically if needed for `profile-build` (usual `build` works fine without net present) PGO gives a nice speedup on fishtest: passed STC: LLR: 2.93 (-2.94,2.94) {-0.50,1.50} Total: 3360 W: 469 L: 343 D: 2548 Ptnml(0-2): 20, 246, 1030, 356, 28 https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569 passed LTC: LLR: 2.97 (-2.94,2.94) {0.25,1.75} Total: 8824 W: 609 L: 502 D: 7713 Ptnml(0-2): 8, 430, 3438, 519, 17 https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c closes official-stockfish/Stockfish#2931 fixes official-stockfish/Stockfish#2907 requires fishtest updates before commit Bench: 4290577

…rofile-build) of the NNUE part of the code. Joint work gvreuls / vondele * Download the default NNUE net in AppVeyor * Download net in travis CI `make net` * Adjust tests to cover more archs, speedup instrumented testing * Introduce 'mixed' bench as default, with further options: classical, NNUE, mixed. mixed (default) and NNUE require the default net to be present, which can be obtained with ``` make net ``` Further examples (first is equivalent to `./stockfish bench`): ``` ./stockfish bench 16 1 13 default depth mixed ./stockfish bench 16 1 13 default depth classical ./stockfish bench 16 1 13 default depth NNUE ``` The net is now downloaded automatically if needed for `profile-build` (usual `build` works fine without net present) PGO gives a nice speedup on fishtest: passed STC: LLR: 2.93 (-2.94,2.94) {-0.50,1.50} Total: 3360 W: 469 L: 343 D: 2548 Ptnml(0-2): 20, 246, 1030, 356, 28 https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569 passed LTC: LLR: 2.97 (-2.94,2.94) {0.25,1.75} Total: 8824 W: 609 L: 502 D: 7713 Ptnml(0-2): 8, 430, 3438, 519, 17 https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c closes official-stockfish#2931 fixes official-stockfish#2907 requires fishtest updates before commit Bench: 4290577

…for PGO (profile-build) of the NNUE part of the code. Joint work gvreuls / vondele * Download the default NNUE net in AppVeyor * Download net in travis CI `make net` * Adjust tests to cover more archs, speedup instrumented testing * Introduce 'mixed' bench as default, with further options: classical, NNUE, mixed. mixed (default) and NNUE require the default net to be present, which can be obtained with ``` make net ``` Further examples (first is equivalent to `./stockfish bench`): ``` ./stockfish bench 16 1 13 default depth mixed ./stockfish bench 16 1 13 default depth classical ./stockfish bench 16 1 13 default depth NNUE ``` The net is now downloaded automatically if needed for `profile-build` (usual `build` works fine without net present) PGO gives a nice speedup on fishtest: passed STC: LLR: 2.93 (-2.94,2.94) {-0.50,1.50} Total: 3360 W: 469 L: 343 D: 2548 Ptnml(0-2): 20, 246, 1030, 356, 28 https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569 passed LTC: LLR: 2.97 (-2.94,2.94) {0.25,1.75} Total: 8824 W: 609 L: 502 D: 7713 Ptnml(0-2): 8, 430, 3438, 519, 17 https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c closes official-stockfish#2931 fixes official-stockfish#2907 requires fishtest updates before commit Bench: 4290577

gvreuls mentioned this pull request Aug 7, 2020

Enable NNUE PGO build #2918

Closed

vondele mentioned this pull request Aug 7, 2020

Possible data race #2933

Closed

mstembera mentioned this pull request Aug 7, 2020

[NNUE] Address single bench and PGO issues #2879

Closed

This was referenced Aug 8, 2020

Single bench covering classic and NNUE eval #2902

Closed

Download net in travis CI #2926

Closed

Download the default NNUE net in AppVeyor #2928

Closed

MS PowerShell : download default NNUE net. Help needed. #2924

Closed

vondele force-pushed the hybridBenchCI branch 4 times, most recently from 40c42b7 to b61fd4f Compare August 10, 2020 11:25

Download the default NNUE net in AppVeyor

4436bc5

This is a partial PR concerning NNUE CI and PGO builds. It modifies the appveyor script so it downloads the default NNUE net before the stockfish bench is run. No functional change.

vondele force-pushed the hybridBenchCI branch 5 times, most recently from fc6b7ee to 0a47bad Compare August 10, 2020 20:55

vondele added 2 commits August 10, 2020 23:31

Download net in travis CI and adjust tests

ab92c6d

Adjust travis to cover more archs, speedup instrumented testing, and download net before building to be able to compute hybrid bench. in preparation for better NNUE integration in CI. No functional change.

vondele force-pushed the hybridBenchCI branch from 0a47bad to 21ff092 Compare August 10, 2020 21:33

vondele added the to be merged Will be merged shortly label Aug 11, 2020

vondele closed this in ea6220f Aug 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bench covering both NNUE and classical, PGO, CI #2931

bench covering both NNUE and classical, PGO, CI #2931

vondele commented Aug 7, 2020

bench covering both NNUE and classical, PGO, CI #2931

bench covering both NNUE and classical, PGO, CI #2931

Conversation

vondele commented Aug 7, 2020