Update big net to nn-b1e55edbea57.nnue #4942
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Created by retraining the master big net `nn-0000000000a0.nnue` on the same dataset with the ranger21 optimizer and more WDL skipping at training time. More WDL skipping is meant to increase lambda accuracy and train on fewer misevaluated positions where position scores are unlikely to correlate with game outcomes. Inspired by: - repeated reports in discord #events-discuss about SF misplaying due to wrong endgame evals, possibly due to Leela's endgame weaknesses reflected in training data - an attempt to reduce the skewed dataset piece count distribution where there are much more positions with less than 16 pieces, since the target piece count distribution in the trainer is symmetric around 16 The faster convergence seen with ranger21 is meant to: - prune experiment ideas more quickly since fewer epochs are needed to reach elo maxima - research faster potential trainings by shortening each run ```yaml experiment-name: 2560-S7-Re-514G-ranger21-more-wdl-skip training-dataset: /data/S6-514G.binpack early-fen-skipping: 28 start-from-engine-test-net: True nnue-pytorch-branch: linrock/nnue-pytorch/r21-more-wdl-skip num-epochs: 1200 lr: 4.375e-4 gamma: 0.995 start-lambda: 1.0 end-lambda: 0.7 ``` Experiment yaml configs converted to easy_train.sh commands with: https://github.com/linrock/nnue-tools/blob/4339954/yaml_easy_train.py Implementations based off of Sopel's NNUE training & experimentation log: https://docs.google.com/document/d/1gTlrr02qSNKiXNZ_SuO4-RjK4MXBiFlLE6jvNqqMkAY - Experiment 336 - ranger21 https://github.com/Sopel97/nnue-pytorch/tree/experiment_336 - Experiment 351 - more WDL skipping The version of the ranger21 optimizer used is: https://github.com/lessw2020/Ranger21/blob/b507df6/ranger21/ranger21.py The dataset is the exact same as in: official-stockfish#4782 Local elo at 25k nodes per move: nn-epoch619.nnue : 6.2 +/- 4.2 Passed STC: https://tests.stockfishchess.org/tests/view/658a029779aa8af82b94fbe6 LLR: 2.93 (-2.94,2.94) <0.00,2.00> Total: 46528 W: 11985 L: 11650 D: 22893 Ptnml(0-2): 154, 5489, 11688, 5734, 199 Passed LTC: https://tests.stockfishchess.org/tests/view/658a448979aa8af82b95010f LLR: 2.94 (-2.94,2.94) <0.50,2.50> Total: 265326 W: 66378 L: 65574 D: 133374 Ptnml(0-2): 153, 30175, 71254, 30877, 204 This was additionally tested with the latest DualNNUE and passed SPRTs: Passed STC vs. official-stockfish#4919 https://tests.stockfishchess.org/tests/view/658bcd5c79aa8af82b951846 LLR: 2.93 (-2.94,2.94) <0.00,2.00> Total: 296128 W: 76273 L: 75554 D: 144301 Ptnml(0-2): 1223, 35768, 73617, 35979, 1477 Passed LTC vs. official-stockfish#4919 https://tests.stockfishchess.org/tests/view/658c988d79aa8af82b95240f LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 75618 W: 19085 L: 18680 D: 37853 Ptnml(0-2): 45, 8420, 20497, 8779, 68 bench 1049162
Disservin
added
to be merged
rn5f107s2
pushed a commit
to rn5f107s2/Stockfish
that referenced
this pull request
Jan 14, 2024
Created by retraining the master big net `nn-0000000000a0.nnue` on the same dataset with the ranger21 optimizer and more WDL skipping at training time. More WDL skipping is meant to increase lambda accuracy and train on fewer misevaluated positions where position scores are unlikely to correlate with game outcomes. Inspired by: - repeated reports in discord #events-discuss about SF misplaying due to wrong endgame evals, possibly due to Leela's endgame weaknesses reflected in training data - an attempt to reduce the skewed dataset piece count distribution where there are much more positions with less than 16 pieces, since the target piece count distribution in the trainer is symmetric around 16 The faster convergence seen with ranger21 is meant to: - prune experiment ideas more quickly since fewer epochs are needed to reach elo maxima - research faster potential trainings by shortening each run ```yaml experiment-name: 2560-S7-Re-514G-ranger21-more-wdl-skip training-dataset: /data/S6-514G.binpack early-fen-skipping: 28 start-from-engine-test-net: True nnue-pytorch-branch: linrock/nnue-pytorch/r21-more-wdl-skip num-epochs: 1200 lr: 4.375e-4 gamma: 0.995 start-lambda: 1.0 end-lambda: 0.7 ``` Experiment yaml configs converted to easy_train.sh commands with: https://github.com/linrock/nnue-tools/blob/4339954/yaml_easy_train.py Implementations based off of Sopel's NNUE training & experimentation log: https://docs.google.com/document/d/1gTlrr02qSNKiXNZ_SuO4-RjK4MXBiFlLE6jvNqqMkAY - Experiment 336 - ranger21 https://github.com/Sopel97/nnue-pytorch/tree/experiment_336 - Experiment 351 - more WDL skipping The version of the ranger21 optimizer used is: https://github.com/lessw2020/Ranger21/blob/b507df6/ranger21/ranger21.py The dataset is the exact same as in: official-stockfish#4782 Local elo at 25k nodes per move: nn-epoch619.nnue : 6.2 +/- 4.2 Passed STC: https://tests.stockfishchess.org/tests/view/658a029779aa8af82b94fbe6 LLR: 2.93 (-2.94,2.94) <0.00,2.00> Total: 46528 W: 11985 L: 11650 D: 22893 Ptnml(0-2): 154, 5489, 11688, 5734, 199 Passed LTC: https://tests.stockfishchess.org/tests/view/658a448979aa8af82b95010f LLR: 2.94 (-2.94,2.94) <0.50,2.50> Total: 265326 W: 66378 L: 65574 D: 133374 Ptnml(0-2): 153, 30175, 71254, 30877, 204 This was additionally tested with the latest DualNNUE and passed SPRTs: Passed STC vs. official-stockfish#4919 https://tests.stockfishchess.org/tests/view/658bcd5c79aa8af82b951846 LLR: 2.93 (-2.94,2.94) <0.00,2.00> Total: 296128 W: 76273 L: 75554 D: 144301 Ptnml(0-2): 1223, 35768, 73617, 35979, 1477 Passed LTC vs. official-stockfish#4919 https://tests.stockfishchess.org/tests/view/658c988d79aa8af82b95240f LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 75618 W: 19085 L: 18680 D: 37853 Ptnml(0-2): 45, 8420, 20497, 8779, 68 closes official-stockfish#4942 Bench: 1304666
windfishballad
pushed a commit
to windfishballad/Stockfish
that referenced
this pull request
Jan 23, 2024
Created by retraining the master big net `nn-0000000000a0.nnue` on the same dataset with the ranger21 optimizer and more WDL skipping at training time. More WDL skipping is meant to increase lambda accuracy and train on fewer misevaluated positions where position scores are unlikely to correlate with game outcomes. Inspired by: - repeated reports in discord #events-discuss about SF misplaying due to wrong endgame evals, possibly due to Leela's endgame weaknesses reflected in training data - an attempt to reduce the skewed dataset piece count distribution where there are much more positions with less than 16 pieces, since the target piece count distribution in the trainer is symmetric around 16 The faster convergence seen with ranger21 is meant to: - prune experiment ideas more quickly since fewer epochs are needed to reach elo maxima - research faster potential trainings by shortening each run ```yaml experiment-name: 2560-S7-Re-514G-ranger21-more-wdl-skip training-dataset: /data/S6-514G.binpack early-fen-skipping: 28 start-from-engine-test-net: True nnue-pytorch-branch: linrock/nnue-pytorch/r21-more-wdl-skip num-epochs: 1200 lr: 4.375e-4 gamma: 0.995 start-lambda: 1.0 end-lambda: 0.7 ``` Experiment yaml configs converted to easy_train.sh commands with: https://github.com/linrock/nnue-tools/blob/4339954/yaml_easy_train.py Implementations based off of Sopel's NNUE training & experimentation log: https://docs.google.com/document/d/1gTlrr02qSNKiXNZ_SuO4-RjK4MXBiFlLE6jvNqqMkAY - Experiment 336 - ranger21 https://github.com/Sopel97/nnue-pytorch/tree/experiment_336 - Experiment 351 - more WDL skipping The version of the ranger21 optimizer used is: https://github.com/lessw2020/Ranger21/blob/b507df6/ranger21/ranger21.py The dataset is the exact same as in: official-stockfish#4782 Local elo at 25k nodes per move: nn-epoch619.nnue : 6.2 +/- 4.2 Passed STC: https://tests.stockfishchess.org/tests/view/658a029779aa8af82b94fbe6 LLR: 2.93 (-2.94,2.94) <0.00,2.00> Total: 46528 W: 11985 L: 11650 D: 22893 Ptnml(0-2): 154, 5489, 11688, 5734, 199 Passed LTC: https://tests.stockfishchess.org/tests/view/658a448979aa8af82b95010f LLR: 2.94 (-2.94,2.94) <0.50,2.50> Total: 265326 W: 66378 L: 65574 D: 133374 Ptnml(0-2): 153, 30175, 71254, 30877, 204 This was additionally tested with the latest DualNNUE and passed SPRTs: Passed STC vs. official-stockfish#4919 https://tests.stockfishchess.org/tests/view/658bcd5c79aa8af82b951846 LLR: 2.93 (-2.94,2.94) <0.00,2.00> Total: 296128 W: 76273 L: 75554 D: 144301 Ptnml(0-2): 1223, 35768, 73617, 35979, 1477 Passed LTC vs. official-stockfish#4919 https://tests.stockfishchess.org/tests/view/658c988d79aa8af82b95240f LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 75618 W: 19085 L: 18680 D: 37853 Ptnml(0-2): 45, 8420, 20497, 8779, 68 closes official-stockfish#4942 Bench: 1304666
linrock
added a commit
to linrock/Stockfish
that referenced
this pull request
Feb 16, 2024
Created by retraining the previous main net `nn-baff1edbea57.nnue` with: - some of the same options as before: ranger21, more WDL skipping - the addition of T80 nov+dec 2023 data - increasing loss by 15% when prediction is too high, up from 10% - use of torch.compile to speed up training by over 25% ```yaml experiment-name: 2560--S9-514G-T80-augtodec2023-more-wdl-skip-15p-more-loss-high-q-sk28 training-dataset: # official-stockfish#4782 - /data/S6-514G-1ee1aba5ed.binpack - /data/test80-aug2023-2tb7p.v6.min.binpack - /data/test80-sep2023-2tb7p.binpack - /data/test80-oct2023-2tb7p.binpack - /data/test80-nov2023-2tb7p.binpack - /data/test80-dec2023-2tb7p.binpack early-fen-skipping: 28 start-from-engine-test-net: True nnue-pytorch-branch: linrock/nnue-pytorch/r21-more-wdl-skip-15p-more-loss-high-q-torch-compile num-epochs: 1000 lr: 4.375e-4 gamma: 0.995 start-lambda: 1.0 end-lambda: 0.7 ``` Epoch 819 trained with the above config led to this PR. Use of torch.compile decorators in nnue-pytorch model.py was found to speed up training by at least 25% on Ampere gpus when using recent pytorch compiled with cuda 12: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch See recent main net PRs for more info on - ranger21 and more WDL skipping: official-stockfish#4942 - increasing loss when Q is too high: official-stockfish#4972 Training data can be found at: https://robotmoon.com/nnue-training-data/ Passed STC: https://tests.stockfishchess.org/tests/view/65cd76151d8e83c78bfd2f52 LLR: 2.98 (-2.94,2.94) <0.00,2.00> Total: 78336 W: 20504 L: 20115 D: 37717 Ptnml(0-2): 317, 9225, 19721, 9562, 343 Passed LTC: https://tests.stockfishchess.org/tests/view/65ce5be61d8e83c78bfd43e9 LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 41016 W: 10492 L: 10159 D: 20365 Ptnml(0-2): 22, 4533, 11071, 4854, 28 Bench: 1265463
Disservin
pushed a commit
that referenced
this pull request
Feb 17, 2024
Created by retraining the previous main net `nn-baff1edbea57.nnue` with: - some of the same options as before: ranger21, more WDL skipping - the addition of T80 nov+dec 2023 data - increasing loss by 15% when prediction is too high, up from 10% - use of torch.compile to speed up training by over 25% ```yaml experiment-name: 2560--S9-514G-T80-augtodec2023-more-wdl-skip-15p-more-loss-high-q-sk28 training-dataset: # #4782 - /data/S6-514G-1ee1aba5ed.binpack - /data/test80-aug2023-2tb7p.v6.min.binpack - /data/test80-sep2023-2tb7p.binpack - /data/test80-oct2023-2tb7p.binpack - /data/test80-nov2023-2tb7p.binpack - /data/test80-dec2023-2tb7p.binpack early-fen-skipping: 28 start-from-engine-test-net: True nnue-pytorch-branch: linrock/nnue-pytorch/r21-more-wdl-skip-15p-more-loss-high-q-torch-compile num-epochs: 1000 lr: 4.375e-4 gamma: 0.995 start-lambda: 1.0 end-lambda: 0.7 ``` Epoch 819 trained with the above config led to this PR. Use of torch.compile decorators in nnue-pytorch model.py was found to speed up training by at least 25% on Ampere gpus when using recent pytorch compiled with cuda 12: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch See recent main net PRs for more info on - ranger21 and more WDL skipping: #4942 - increasing loss when Q is too high: #4972 Training data can be found at: https://robotmoon.com/nnue-training-data/ Passed STC: https://tests.stockfishchess.org/tests/view/65cd76151d8e83c78bfd2f52 LLR: 2.98 (-2.94,2.94) <0.00,2.00> Total: 78336 W: 20504 L: 20115 D: 37717 Ptnml(0-2): 317, 9225, 19721, 9562, 343 Passed LTC: https://tests.stockfishchess.org/tests/view/65ce5be61d8e83c78bfd43e9 LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 41016 W: 10492 L: 10159 D: 20365 Ptnml(0-2): 22, 4533, 11071, 4854, 28 closes #5056 Bench: 1351997
xu-shawn
pushed a commit
to xu-shawn/Stockfish
that referenced
this pull request
Feb 17, 2024
Created by retraining the previous main net `nn-baff1edbea57.nnue` with: - some of the same options as before: ranger21, more WDL skipping - the addition of T80 nov+dec 2023 data - increasing loss by 15% when prediction is too high, up from 10% - use of torch.compile to speed up training by over 25% ```yaml experiment-name: 2560--S9-514G-T80-augtodec2023-more-wdl-skip-15p-more-loss-high-q-sk28 training-dataset: # official-stockfish#4782 - /data/S6-514G-1ee1aba5ed.binpack - /data/test80-aug2023-2tb7p.v6.min.binpack - /data/test80-sep2023-2tb7p.binpack - /data/test80-oct2023-2tb7p.binpack - /data/test80-nov2023-2tb7p.binpack - /data/test80-dec2023-2tb7p.binpack early-fen-skipping: 28 start-from-engine-test-net: True nnue-pytorch-branch: linrock/nnue-pytorch/r21-more-wdl-skip-15p-more-loss-high-q-torch-compile num-epochs: 1000 lr: 4.375e-4 gamma: 0.995 start-lambda: 1.0 end-lambda: 0.7 ``` Epoch 819 trained with the above config led to this PR. Use of torch.compile decorators in nnue-pytorch model.py was found to speed up training by at least 25% on Ampere gpus when using recent pytorch compiled with cuda 12: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch See recent main net PRs for more info on - ranger21 and more WDL skipping: official-stockfish#4942 - increasing loss when Q is too high: official-stockfish#4972 Training data can be found at: https://robotmoon.com/nnue-training-data/ Passed STC: https://tests.stockfishchess.org/tests/view/65cd76151d8e83c78bfd2f52 LLR: 2.98 (-2.94,2.94) <0.00,2.00> Total: 78336 W: 20504 L: 20115 D: 37717 Ptnml(0-2): 317, 9225, 19721, 9562, 343 Passed LTC: https://tests.stockfishchess.org/tests/view/65ce5be61d8e83c78bfd43e9 LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 41016 W: 10492 L: 10159 D: 20365 Ptnml(0-2): 22, 4533, 11071, 4854, 28 closes official-stockfish#5056 Bench: 1351997
xu-shawn
pushed a commit
to xu-shawn/Stockfish
that referenced
this pull request
Feb 19, 2024
Created by retraining the previous main net `nn-baff1edbea57.nnue` with: - some of the same options as before: ranger21, more WDL skipping - the addition of T80 nov+dec 2023 data - increasing loss by 15% when prediction is too high, up from 10% - use of torch.compile to speed up training by over 25% ```yaml experiment-name: 2560--S9-514G-T80-augtodec2023-more-wdl-skip-15p-more-loss-high-q-sk28 training-dataset: # official-stockfish#4782 - /data/S6-514G-1ee1aba5ed.binpack - /data/test80-aug2023-2tb7p.v6.min.binpack - /data/test80-sep2023-2tb7p.binpack - /data/test80-oct2023-2tb7p.binpack - /data/test80-nov2023-2tb7p.binpack - /data/test80-dec2023-2tb7p.binpack early-fen-skipping: 28 start-from-engine-test-net: True nnue-pytorch-branch: linrock/nnue-pytorch/r21-more-wdl-skip-15p-more-loss-high-q-torch-compile num-epochs: 1000 lr: 4.375e-4 gamma: 0.995 start-lambda: 1.0 end-lambda: 0.7 ``` Epoch 819 trained with the above config led to this PR. Use of torch.compile decorators in nnue-pytorch model.py was found to speed up training by at least 25% on Ampere gpus when using recent pytorch compiled with cuda 12: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch See recent main net PRs for more info on - ranger21 and more WDL skipping: official-stockfish#4942 - increasing loss when Q is too high: official-stockfish#4972 Training data can be found at: https://robotmoon.com/nnue-training-data/ Passed STC: https://tests.stockfishchess.org/tests/view/65cd76151d8e83c78bfd2f52 LLR: 2.98 (-2.94,2.94) <0.00,2.00> Total: 78336 W: 20504 L: 20115 D: 37717 Ptnml(0-2): 317, 9225, 19721, 9562, 343 Passed LTC: https://tests.stockfishchess.org/tests/view/65ce5be61d8e83c78bfd43e9 LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 41016 W: 10492 L: 10159 D: 20365 Ptnml(0-2): 22, 4533, 11071, 4854, 28 closes official-stockfish#5056 Bench: 1351997
xu-shawn
pushed a commit
to xu-shawn/Stockfish
that referenced
this pull request
Feb 19, 2024
Created by retraining the previous main net `nn-baff1edbea57.nnue` with: - some of the same options as before: ranger21, more WDL skipping - the addition of T80 nov+dec 2023 data - increasing loss by 15% when prediction is too high, up from 10% - use of torch.compile to speed up training by over 25% ```yaml experiment-name: 2560--S9-514G-T80-augtodec2023-more-wdl-skip-15p-more-loss-high-q-sk28 training-dataset: # official-stockfish#4782 - /data/S6-514G-1ee1aba5ed.binpack - /data/test80-aug2023-2tb7p.v6.min.binpack - /data/test80-sep2023-2tb7p.binpack - /data/test80-oct2023-2tb7p.binpack - /data/test80-nov2023-2tb7p.binpack - /data/test80-dec2023-2tb7p.binpack early-fen-skipping: 28 start-from-engine-test-net: True nnue-pytorch-branch: linrock/nnue-pytorch/r21-more-wdl-skip-15p-more-loss-high-q-torch-compile num-epochs: 1000 lr: 4.375e-4 gamma: 0.995 start-lambda: 1.0 end-lambda: 0.7 ``` Epoch 819 trained with the above config led to this PR. Use of torch.compile decorators in nnue-pytorch model.py was found to speed up training by at least 25% on Ampere gpus when using recent pytorch compiled with cuda 12: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch See recent main net PRs for more info on - ranger21 and more WDL skipping: official-stockfish#4942 - increasing loss when Q is too high: official-stockfish#4972 Training data can be found at: https://robotmoon.com/nnue-training-data/ Passed STC: https://tests.stockfishchess.org/tests/view/65cd76151d8e83c78bfd2f52 LLR: 2.98 (-2.94,2.94) <0.00,2.00> Total: 78336 W: 20504 L: 20115 D: 37717 Ptnml(0-2): 317, 9225, 19721, 9562, 343 Passed LTC: https://tests.stockfishchess.org/tests/view/65ce5be61d8e83c78bfd43e9 LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 41016 W: 10492 L: 10159 D: 20365 Ptnml(0-2): 22, 4533, 11071, 4854, 28 closes official-stockfish#5056 Bench: 1351997
xu-shawn
pushed a commit
to xu-shawn/Stockfish
that referenced
this pull request
Feb 19, 2024
Created by retraining the previous main net `nn-baff1edbea57.nnue` with: - some of the same options as before: ranger21, more WDL skipping - the addition of T80 nov+dec 2023 data - increasing loss by 15% when prediction is too high, up from 10% - use of torch.compile to speed up training by over 25% ```yaml experiment-name: 2560--S9-514G-T80-augtodec2023-more-wdl-skip-15p-more-loss-high-q-sk28 training-dataset: # official-stockfish#4782 - /data/S6-514G-1ee1aba5ed.binpack - /data/test80-aug2023-2tb7p.v6.min.binpack - /data/test80-sep2023-2tb7p.binpack - /data/test80-oct2023-2tb7p.binpack - /data/test80-nov2023-2tb7p.binpack - /data/test80-dec2023-2tb7p.binpack early-fen-skipping: 28 start-from-engine-test-net: True nnue-pytorch-branch: linrock/nnue-pytorch/r21-more-wdl-skip-15p-more-loss-high-q-torch-compile num-epochs: 1000 lr: 4.375e-4 gamma: 0.995 start-lambda: 1.0 end-lambda: 0.7 ``` Epoch 819 trained with the above config led to this PR. Use of torch.compile decorators in nnue-pytorch model.py was found to speed up training by at least 25% on Ampere gpus when using recent pytorch compiled with cuda 12: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch See recent main net PRs for more info on - ranger21 and more WDL skipping: official-stockfish#4942 - increasing loss when Q is too high: official-stockfish#4972 Training data can be found at: https://robotmoon.com/nnue-training-data/ Passed STC: https://tests.stockfishchess.org/tests/view/65cd76151d8e83c78bfd2f52 LLR: 2.98 (-2.94,2.94) <0.00,2.00> Total: 78336 W: 20504 L: 20115 D: 37717 Ptnml(0-2): 317, 9225, 19721, 9562, 343 Passed LTC: https://tests.stockfishchess.org/tests/view/65ce5be61d8e83c78bfd43e9 LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 41016 W: 10492 L: 10159 D: 20365 Ptnml(0-2): 22, 4533, 11071, 4854, 28 closes official-stockfish#5056 Bench: 1351997
TierynnB
pushed a commit
to TierynnB/Stockfish
that referenced
this pull request
Feb 22, 2024
Created by retraining the previous main net `nn-baff1edbea57.nnue` with: - some of the same options as before: ranger21, more WDL skipping - the addition of T80 nov+dec 2023 data - increasing loss by 15% when prediction is too high, up from 10% - use of torch.compile to speed up training by over 25% ```yaml experiment-name: 2560--S9-514G-T80-augtodec2023-more-wdl-skip-15p-more-loss-high-q-sk28 training-dataset: # official-stockfish#4782 - /data/S6-514G-1ee1aba5ed.binpack - /data/test80-aug2023-2tb7p.v6.min.binpack - /data/test80-sep2023-2tb7p.binpack - /data/test80-oct2023-2tb7p.binpack - /data/test80-nov2023-2tb7p.binpack - /data/test80-dec2023-2tb7p.binpack early-fen-skipping: 28 start-from-engine-test-net: True nnue-pytorch-branch: linrock/nnue-pytorch/r21-more-wdl-skip-15p-more-loss-high-q-torch-compile num-epochs: 1000 lr: 4.375e-4 gamma: 0.995 start-lambda: 1.0 end-lambda: 0.7 ``` Epoch 819 trained with the above config led to this PR. Use of torch.compile decorators in nnue-pytorch model.py was found to speed up training by at least 25% on Ampere gpus when using recent pytorch compiled with cuda 12: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch See recent main net PRs for more info on - ranger21 and more WDL skipping: official-stockfish#4942 - increasing loss when Q is too high: official-stockfish#4972 Training data can be found at: https://robotmoon.com/nnue-training-data/ Passed STC: https://tests.stockfishchess.org/tests/view/65cd76151d8e83c78bfd2f52 LLR: 2.98 (-2.94,2.94) <0.00,2.00> Total: 78336 W: 20504 L: 20115 D: 37717 Ptnml(0-2): 317, 9225, 19721, 9562, 343 Passed LTC: https://tests.stockfishchess.org/tests/view/65ce5be61d8e83c78bfd43e9 LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 41016 W: 10492 L: 10159 D: 20365 Ptnml(0-2): 22, 4533, 11071, 4854, 28 closes official-stockfish#5056 Bench: 1351997
TierynnB
added a commit
to TierynnB/Stockfish
that referenced
this pull request
Feb 23, 2024
commit 524083a Merge: 8a0206f 4a5ba40 Author: Tierynn Byrnes <[email protected]> Date: Fri Feb 23 08:42:30 2024 +1000 Merge branch 'TM_Change_2' of https://github.com/TierynnB/Stockfish into TM_Change_2 commit 8a0206f Author: Tierynn Byrnes <[email protected]> Date: Mon Feb 19 07:13:26 2024 +1000 use current time instead of '1' for timeLeft formula. make timeLeft a double, timepoint seemed unecessary since it was always casting back to double anyway. fixed comments Squashed commits commit 4a5ba40 Merge: ce952bf 676a1d7 Author: Tierynn Byrnes <[email protected]> Date: Fri Feb 23 08:01:21 2024 +1000 Merge branch 'TM_Change_2' of https://github.com/TierynnB/Stockfish into TM_Change_2 commit ce952bf Author: cj5716 <[email protected]> Date: Tue Feb 13 17:46:37 2024 +0800 Simplify PV node reduction Reduce less on PV nodes even with an upperbound TT entry. Passed STC: https://tests.stockfishchess.org/tests/view/65cb3a861d8e83c78bfd0497 LLR: 2.96 (-2.94,2.94) <-1.75,0.25> Total: 118752 W: 30441 L: 30307 D: 58004 Ptnml(0-2): 476, 14179, 29921, 14335, 465 Passed LTC: https://tests.stockfishchess.org/tests/view/65cd3b951d8e83c78bfd2b0d LLR: 2.95 (-2.94,2.94) <-1.75,0.25> Total: 155058 W: 38549 L: 38464 D: 78045 Ptnml(0-2): 85, 17521, 42219, 17632, 72 closes official-stockfish#5057 Bench: 1303971 commit 4acf810 Author: Linmiao Xu <[email protected]> Date: Tue Feb 6 11:21:15 2024 -0500 Update default main net to nn-b1a57edbea57.nnue Created by retraining the previous main net `nn-baff1edbea57.nnue` with: - some of the same options as before: ranger21, more WDL skipping - the addition of T80 nov+dec 2023 data - increasing loss by 15% when prediction is too high, up from 10% - use of torch.compile to speed up training by over 25% ```yaml experiment-name: 2560--S9-514G-T80-augtodec2023-more-wdl-skip-15p-more-loss-high-q-sk28 training-dataset: # official-stockfish#4782 - /data/S6-514G-1ee1aba5ed.binpack - /data/test80-aug2023-2tb7p.v6.min.binpack - /data/test80-sep2023-2tb7p.binpack - /data/test80-oct2023-2tb7p.binpack - /data/test80-nov2023-2tb7p.binpack - /data/test80-dec2023-2tb7p.binpack early-fen-skipping: 28 start-from-engine-test-net: True nnue-pytorch-branch: linrock/nnue-pytorch/r21-more-wdl-skip-15p-more-loss-high-q-torch-compile num-epochs: 1000 lr: 4.375e-4 gamma: 0.995 start-lambda: 1.0 end-lambda: 0.7 ``` Epoch 819 trained with the above config led to this PR. Use of torch.compile decorators in nnue-pytorch model.py was found to speed up training by at least 25% on Ampere gpus when using recent pytorch compiled with cuda 12: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch See recent main net PRs for more info on - ranger21 and more WDL skipping: official-stockfish#4942 - increasing loss when Q is too high: official-stockfish#4972 Training data can be found at: https://robotmoon.com/nnue-training-data/ Passed STC: https://tests.stockfishchess.org/tests/view/65cd76151d8e83c78bfd2f52 LLR: 2.98 (-2.94,2.94) <0.00,2.00> Total: 78336 W: 20504 L: 20115 D: 37717 Ptnml(0-2): 317, 9225, 19721, 9562, 343 Passed LTC: https://tests.stockfishchess.org/tests/view/65ce5be61d8e83c78bfd43e9 LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 41016 W: 10492 L: 10159 D: 20365 Ptnml(0-2): 22, 4533, 11071, 4854, 28 closes official-stockfish#5056 Bench: 1351997 commit 40c6cdf Author: cj5716 <[email protected]> Date: Tue Feb 13 17:50:16 2024 +0800 Simplify TT PV reduction This also removes some incorrect fail-high logic. Passed STC: https://tests.stockfishchess.org/tests/view/65cb3b641d8e83c78bfd04a9 LLR: 2.94 (-2.94,2.94) <-1.75,0.25> Total: 87968 W: 22634 L: 22468 D: 42866 Ptnml(0-2): 315, 10436, 22323, 10588, 322 Passed LTC: https://tests.stockfishchess.org/tests/view/65cccee21d8e83c78bfd222c LLR: 2.95 (-2.94,2.94) <-1.75,0.25> Total: 70794 W: 17846 L: 17672 D: 35276 Ptnml(0-2): 44, 7980, 19189, 8126, 58 closes official-stockfish#5055 Bench: 1474424 commit 9299d01 Author: Gahtan Nahdi <[email protected]> Date: Sat Feb 10 03:51:05 2024 +0700 Remove penalty for quiet ttMove that fails low Passed STC non-reg: https://tests.stockfishchess.org/tests/view/65c691a7c865510db0286e6e LLR: 2.95 (-2.94,2.94) <-1.75,0.25> Total: 234336 W: 60258 L: 60255 D: 113823 Ptnml(0-2): 966, 28141, 58918, 28210, 933 Passed LTC non-reg: https://tests.stockfishchess.org/tests/view/65c8d0d31d8e83c78bfcd4a6 LLR: 2.95 (-2.94,2.94) <-1.75,0.25> Total: 235206 W: 59134 L: 59132 D: 116940 Ptnml(0-2): 135, 26908, 63517, 26906, 137 official-stockfish#5054 Bench: 1287996 commit 676a1d7 Merge: 7d0cd7b 3c3f88b Author: Tierynn Byrnes <[email protected]> Date: Fri Feb 23 07:58:38 2024 +1000 Merge branch 'TM_Change_2' of https://github.com/TierynnB/Stockfish into TM_Change_2 commit 7d0cd7b Author: Tierynn Byrnes <[email protected]> Date: Mon Feb 19 07:13:26 2024 +1000 parent 8b67b7e author Tierynn Byrnes <[email protected]> 1708290806 +1000 committer Tierynn Byrnes <[email protected]> 1708638981 +1000 use current time instead of '1' for timeLeft formula. make timeLeft a double, timepoint seemed unecessary since it was always casting back to double anyway. commit 3c3f88b Merge: 76c50a0 61e8083 Author: Tierynn Byrnes <[email protected]> Date: Fri Feb 23 07:54:46 2024 +1000 Merge branch 'TM_Change_2' of https://github.com/TierynnB/Stockfish into TM_Change_2 commit 76c50a0 Author: Tierynn Byrnes <[email protected]> Date: Mon Feb 19 07:13:26 2024 +1000 use current time instead of '1' for timeLeft formula. make timeLeft a double, timepoint seemed unecessary since it was always casting back to double anyway. fixed comments commit 61e8083 Merge: 5cf3f49 8afec41 Author: Tierynn Byrnes <[email protected]> Date: Thu Feb 22 19:44:21 2024 +1000 Merge branch 'TM_Change_2' of https://github.com/TierynnB/Stockfish into TM_Change_2 commit 5cf3f49 Author: Tierynn Byrnes <[email protected]> Date: Mon Feb 19 07:13:26 2024 +1000 use current time instead of '1' for timeLeft formula. make timeLeft a double, timepoint seemed unecessary since it was always casting back to double anyway. commit 8afec41 Author: Tierynn Byrnes <[email protected]> Date: Mon Feb 19 07:50:30 2024 +1000 fixed comments commit de4a3c4 Author: Tierynn Byrnes <[email protected]> Date: Mon Feb 19 07:32:09 2024 +1000 make timeLeft a double, timepoint seemed unecessary since it was always casting back to double anyway. commit e1f6b87 Merge: 8b67b7e fc41f64 Author: Lemmy <[email protected]> Date: Mon Feb 19 07:14:27 2024 +1000 Merge branch 'official-stockfish:master' into TM_Change_2 commit 8b67b7e Author: Tierynn Byrnes <[email protected]> Date: Mon Feb 19 07:13:26 2024 +1000 use current time instead of '1' for timeLeft formula.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Created by retraining the master big net
nn-0000000000a0.nnueon the same dataset with the ranger21 optimizer and more WDL skipping at training time.More WDL skipping is meant to increase lambda accuracy and train on fewer misevaluated positions where position scores are unlikely to correlate with game outcomes. Inspired by:
The faster convergence seen with ranger21 is meant to:
Experiment yaml configs converted to easy_train.sh commands with:
https://github.com/linrock/nnue-tools/blob/4339954/yaml_easy_train.py
Implementations based off of Sopel's NNUE training & experimentation log:
https://docs.google.com/document/d/1gTlrr02qSNKiXNZ_SuO4-RjK4MXBiFlLE6jvNqqMkAY
The version of the ranger21 optimizer used is:
https://github.com/lessw2020/Ranger21/blob/b507df6/ranger21/ranger21.py
The dataset is the exact same as in:
#4782
Local elo at 25k nodes per move:
nn-epoch619.nnue : 6.2 +/- 4.2
Passed STC:
https://tests.stockfishchess.org/tests/view/658a029779aa8af82b94fbe6
LLR: 2.93 (-2.94,2.94) <0.00,2.00>
Total: 46528 W: 11985 L: 11650 D: 22893
Ptnml(0-2): 154, 5489, 11688, 5734, 199
Passed LTC:
https://tests.stockfishchess.org/tests/view/658a448979aa8af82b95010f
LLR: 2.94 (-2.94,2.94) <0.50,2.50>
Total: 265326 W: 66378 L: 65574 D: 133374
Ptnml(0-2): 153, 30175, 71254, 30877, 204
This was additionally tested with the latest DualNNUE and passed SPRTs:
Passed STC vs. #4919
https://tests.stockfishchess.org/tests/view/658bcd5c79aa8af82b951846
LLR: 2.93 (-2.94,2.94) <0.00,2.00>
Total: 296128 W: 76273 L: 75554 D: 144301
Ptnml(0-2): 1223, 35768, 73617, 35979, 1477
Passed LTC vs. #4919
https://tests.stockfishchess.org/tests/view/658c988d79aa8af82b95240f
LLR: 2.95 (-2.94,2.94) <0.50,2.50>
Total: 75618 W: 19085 L: 18680 D: 37853
Ptnml(0-2): 45, 8420, 20497, 8779, 68
bench 1049162