Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Contempt 20 #1366

Closed
Closed

Conversation

snicolet
Copy link
Member

@snicolet snicolet commented Jan 22, 2018

Set the default contempt value of Stockfish to 20 centipawns.

The so-called contempt feature of Stockfish tries to prevent the
engine from simplifying the position too quickly when it feels that it
is very slightly behind, instead keeping the tension a little bit longer.

Various tests in November 2017 have proved that our current imple-
mentation works well against SF7 (which is about 130 Elo weaker than
current master) and that the Elo gain is an increasing function of
contempt, going (against SF7) from +0 Elo when contempt is set at
zero centipawns, to +32 Elo when contempt is 40 centipawns.

See pull request 1325 for details:
#1325

This November discussion left open the decision of which "default"
value for contempt we should use for Stockfish, taking into account
the various uses of Stockfish (opening preparation for humans, computer
online tournaments, analysis tool for web pages, human/computer play,
etc).

This pull request proposes to set the default contempt value of SF
to twenty centipawns, which turns out to be the highest value which
is not a regression against current master, as this seemed to be a
good compromise between risk and safety. A couple of SPRT[-3..1]
tests at long time control were done to bisect this value:

LTC:
Contempt 10: http://tests.stockfishchess.org/tests/view/5a5d42d20ebc5902977e2901 (PASSED)
Contempt 15: http://tests.stockfishchess.org/tests/view/5a5d41740ebc5902977e28fa (PASSED)
Contempt 20: http://tests.stockfishchess.org/tests/view/5a5d42060ebc5902977e28fc (PASSED)
Contempt 25: http://tests.stockfishchess.org/tests/view/5a5d433f0ebc5902977e2904 (FAILED)

Surprisingly, a test at "very long time control" hinted that using
contempt 20 would not only be non-regressive against contempt 0, but
may actually exhibit some small Elo gain (+2.4 Elo), giving a likelihood
of superiority of 90% after 8500 games:

VLTC:
ELO: 2.44 +-3.7 (95%) LOS: 90.2%
Total: 8546 W: 1104 L: 1044 D: 6398
http://tests.stockfishchess.org/tests/view/5a60b2820ebc590297b9b7e0

Finally, there were some concerns that a contempt value of 20 would
be worse than a value of 7, but a test with 20000 games at STC was
neutral:

STC:
ELO: 0.45 +-3.1 (95%) LOS: 61.2%
Total: 20000 W: 4222 L: 4196 D: 11582
http://tests.stockfishchess.org/tests/view/5a64d2fd0ebc590297903868

See the comments in pull request 1361 for the long, nice discussion
(180 entries :-)) leading to the decision to propose contempt 20 as
the default value:
#1361

Whether Stockfish should strictly adhere to the Komodo and Houdini
semantics, adding the UCI commands to force the contempt to be White
in the so-called "analysis mode" is still under discussion, and may be or
may not be the object of a future commit.

Bench: 5571216

Set the default contempt value of Stockfish to 20 centipawns.

The contempt feature of Stockfish tries to prevent the engine from
simplifying the position too quickly when it feels that it is very
slightly behind, instead keeping the tension a little bit longer.

Various tests in November 2017 have proved that our current imple-
mentation works well against SF7 (which is about 130 Elo weaker than
current master) and than the Elo gain is an increasing function of
contempt, going (against SF7) from +0 Elo when contempt is set at
zero centipawns, to +30 Elo when contempt is 40 centipawns.

See pull request 1325 for details:

official-stockfish/Stockfish#1325

This november discussion left open the decision of which "default"
value for contempt we should use for Stockfish, taking into account
the various uses ofStockfish (opening preparation for humans, computer
online tournaments,analysis tool for web pages, human/computer play,
etc).

This pull request proposes to set the default contempt value of SF
to twenty centipawns, which turns out to be the highest value which
is not a regression against current master, as this seemed to be a
good compromise between risk and safety. A couple of SPRT[-3..1]
tests were done to bisect this value:

Contempt 10: http://tests.stockfishchess.org/tests/view/5a5d42d20ebc5902977e2901 (PASSED)
Contempt 15: http://tests.stockfishchess.org/tests/view/5a5d41740ebc5902977e28fa (PASSED)
Contempt 20: http://tests.stockfishchess.org/tests/view/5a5d42060ebc5902977e28fc (PASSED)
Contempt 25: http://tests.stockfishchess.org/tests/view/5a5d433f0ebc5902977e2904 (FAILED)

Surprisingly, a test at "very long time control" hinted that using
contempt 20 is not only be non-regressive against contempt 0, but
may actually exhibit some small Elo gain, giving a likehood of superio-
rity of 88.7% after 8500 games:

VLTC:
ELO: 2.28 +-3.7 (95%) LOS: 88.7%
Total: 8521 W: 1096 L: 1040 D: 6385
http://tests.stockfishchess.org/tests/view/5a60b2820ebc590297b9b7e0

Finally, there was some concerns that a contempt value of 20 would
be worse than a value of 7, but a test with 20000 games at STC was
neutral:

STC:
ELO: 0.45 +-3.1 (95%) LOS: 61.2%
Total: 20000 W: 4222 L: 4196 D: 11582
http://tests.stockfishchess.org/tests/view/5a64d2fd0ebc590297903868

See the comments in pull request 1361 for the long, nice discussion
(180 entries :-)) leading to the decision to propose contempt 20 as
the default value:

official-stockfish/Stockfish#1361

Whether Stockfish should strictly adhere to the Komodo and Houdini
semantics and add the UCI commands to force the contempt to be White
in the so-called "analysis mode" is still under discussion, and may
be or may not be the object of a future commit.

Bench: 5571216
@AlexandreMasta
Copy link

Very nice and well put.

Hope SF9's release is near.

@mcostalba
Copy link

Merged with 254d995

Congratulations! Impressive work and impressive log message. Very high quality stuff here :-)

@mcostalba mcostalba closed this Jan 23, 2018
@mcostalba mcostalba mentioned this pull request Jan 23, 2018
@Ipmanchess
Copy link

Thank you Marco..looks like my testings shows again right ;)

@atumanian
Copy link

atumanian commented Jan 23, 2018

@mcostalba, it was too early to commit this patch. @syzygy1 has correctly pointed out that the current implementation of contempt isn't suitable for analysis.
There are also other reasons for rejecting the default contempt setting. If the purpose is to boost the position in rating lists, then the value of 20 is too small, because most other engines are much weaker. I don't see the point of setting the default value based on [-3, 1] tests.

@mcostalba
Copy link

@atumanian the correct line is that user should set contempt zero for analysis. Please note that this is true for any engine at any time, and (to me) it is acceptable to think this fact is in the knwoledge of users that usually do chess analysis.

The new fact here is that before contempt was at zero by default now it should be explicitly set.

@atumanian
Copy link

@mcostalba, thank you for your reply. I've recently created a pull request #1369. We can discuss the issue there.

@NKONSTANTAKIS NKONSTANTAKIS mentioned this pull request Nov 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants