Add tests for the suggest method #1316

teytaud · 2021-12-03T07:46:58Z

Types of changes

Docs change / refactoring / dependency upgrade
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Motivation and Context / Related issue

We want to be sure that "suggest" works.
It turns out that it does work.

How Has This Been Tested (if it applies)

Checklist

The documentation is up-to-date with the changes I made.
I have read the CONTRIBUTING document and completed the CLA (see CLA).
All tests passed, and additional code has been covered with new tests.

jrapin

There is a lot of very redundant code, you could easily change the target depending on the case instead

nevergrad/optimization/test_optimizerlib.py

Co-authored-by: Jérémy Rapin <[email protected]>

…gest

…suggest

teytaud · 2021-12-15T08:03:34Z

There is a lot of very redundant code, you could easily change the target depending on the case instead

The different tests are distinct though. Some cases are continuous and others are discrete. Also, in some cases we just check "if the optimum has been suggested then we find it" and in other cases we check the stronger requirement "if a good suboptimal point has been provided then we find the optimum".

Co-authored-by: Jérémy Rapin <[email protected]>

teytaud · 2021-12-15T08:05:26Z

There is a lot of very redundant code, you could easily change the target depending on the case instead

The different tests are distinct though. Some cases are continuous and others are discrete. Also, in some cases we just check "if the optimum has been suggested then we find it" and in other cases we check the stronger requirement "if a good suboptimal point has been provided then we find the optimum".

I think compacting more would decrease readability.

jrapin · 2021-12-17T17:43:55Z

There is a lot of very redundant code, you could easily change the target depending on the case instead

The different tests are distinct though. Some cases are continuous and others are discrete. Also, in some cases we just check "if the optimum has been suggested then we find it" and in other cases we check the stronger requirement "if a good suboptimal point has been provided then we find the optimum".

I think compacting more would decrease readability.

But increase cost of maintenance, there is too much shared code here, I am sure it's possible to be both readable and factorized

* Add architecture search for Gym (#1315) * as * black * add_Test * type_fix * typing * fix * fix_bug * fix_weird * wtf * fix * fix * fix * black * switch_to_choice * mypy * fix * fix * fix * fix * fix * fix_type * Adding optimization specifically for RL (#1303) * Adding optimization specifically for RL * fix * fix * Import benchmarks online for fixing dependencies (#1310) * fix_dependencies * fix_tests * : missing * fix_lint * fix * fix * fix * XPs with sparsity on Open AI Gym (#1319) * XPs with sparsity on Open AI Gym * Update gymexperiments.py * Adding state of the art results on OpenAI Gym results (#1318) * Update plotting.py * fix * ok_good * fix * Adding example for OpenAI Gym (#1320) * testing more algorithms for permutations (#1321) * testing more algorithms for permutations * fix * Adding yahdlbbbob and simplifying stuff (#1145) * Update experiments.py * black * fix_case_1 * fix * Update experiments.py * Add Olympus benchmark (#1190) * Add olympus function and benchmark * remove surfaces with error * Add noise and use average of noise free surfaces for evaluation_finction * fix static tests * rename olympus to olympussurfaces * fix test_core olympus * replace olympus with olymp in requirements * fix olymp version in requirements * black reformatting * Add discrete and GaussianMixture surfaces * Add discrete and GaussianMixture surfaces * Move surfaces in a class variable * no_need_for_42_because_noiseless * nowindowsforcarraz * fix * fix * fix * fix * fix * fix * fix * fix * bettertest * Add olympus emulators * add BayesNeuralNet * reformat code * minor fix * minor fix * minor fix * add missing package and correct seed problem * remove unused packages * fix silence_tensorflow version * try after merge * fix * gamma_fix * fix * fix * fix_naming * fix * fix * fix * fix * fix * fix * fix * fix * Update requirements/bench.txt Co-authored-by: Jérémy Rapin <[email protected]> * YEEEEEEEES_it_works Co-authored-by: ncarraz <[email protected]> Co-authored-by: Jérémy Rapin <[email protected]> * Documentation for permutations (#1322) * change sparsity in gym (#1328) * change sparsity in gym * fix * Unstack ngopt (#1330) * fix plot (#1332) * fix plot * black * Fix gym: thread safety (#1338) * fixgym * fix * fix * fix * Avoid bool check on params (#1349) * Keep only final optimizer in portfolio (#1350) * Fix nan in cma (#1347) * fixnan * fixnan * fixnan * fo * Keep data in cache for CMA (#1351) * Add installation instruction for MuJoCo (#1352) * adding installation instruction for MuJoCo * raise error * better error handling Co-authored-by: Mathurin Videau <[email protected]> * enable_pickling for NGOpt (#1356) * Bugfix in scrambled progressive optimization. (#1357) * Add anisotropic progressive optimization (#1344) * Adding anisotropic progressive optimization * black * Finish adding enable_pickling for all optimizers (Rescaled) (#1358) * Residual controllers in RL. (#1359) * Residual controllers in RL. It's not exactly residual, it's initializing with something closer to identity. It works pretty well. * Update multigym.py * fix * fix * fix * black * fix * seed * Update test_core.py * fix * fix * fix * Bump version to 0.4.3.post10 (#1364) * Removing an incomplete sentence from the doc (#1367) * Fix broken CI (#1370) * docs: add GH button in support of Ukraine (#1369) * Add the FAO crop model (#1343) * aquacrop * fix * fix * fix * Update ac.py * black * Update experiments.py (#1361) * fix * Update bench.txt * fix * fix * fix * tentative_pip3 * yet_another_tentative_fi * yet_another_tentative_fi * fix * fix_suffering * desperate_try * desperate_try * desperate_try * desperate_try * fix * desperate_try * desperate_try * desperate_try * desperate_try * fix * Update config.yml * fix * Update setup.py * Update main.txt * fix * Use up-to-date headers (#1371) * Add NLOPT as a solver (#1340) * Update version and changelog to 0.5.0 (#1372) * Deactivate mutation test in CI (#1374) * Reduce noise in gym (#1333) * Reduce noise in gym * Update multigym.py * Add comment * NaN robustness in Gym (#1289) * NaN robustness in Gym * Update nevergrad/functions/gym/multigym.py Co-authored-by: Jérémy Rapin <[email protected]> * fix * fix Co-authored-by: Jérémy Rapin <[email protected]> * Add more models for Gym control (#1346) * more models for Gym control * Update gymexperiments.py * Update multigym.py * fix * Update gymexperiments.py * update moremodels (#1378) * Bump version to 0.4.3.post10 (#1364) * Removing an incomplete sentence from the doc (#1367) * Fix broken CI (#1370) * docs: add GH button in support of Ukraine (#1369) * Add the FAO crop model (#1343) * aquacrop * fix * fix * fix * Update ac.py * black * Update experiments.py (#1361) * fix * Update bench.txt * fix * fix * fix * tentative_pip3 * yet_another_tentative_fi * yet_another_tentative_fi * fix * fix_suffering * desperate_try * desperate_try * desperate_try * desperate_try * fix * desperate_try * desperate_try * desperate_try * desperate_try * fix * Update config.yml * fix * Update setup.py * Update main.txt * fix * Use up-to-date headers (#1371) * Add NLOPT as a solver (#1340) * Update version and changelog to 0.5.0 (#1372) * Deactivate mutation test in CI (#1374) * Reduce noise in gym (#1333) * Reduce noise in gym * Update multigym.py * Add comment Co-authored-by: Jérémy Rapin <[email protected]> Co-authored-by: Dmitry Vinnik <[email protected]> * fix * fix * im_lost * fix * fix * fix Co-authored-by: Jérémy Rapin <[email protected]> Co-authored-by: Dmitry Vinnik <[email protected]> * Add a conformant GP experiment (#1337) * Adding a conformant GP experiment Because sometimes conformant planning, in spite of being super simple, performs incredibly well. * fix * High-speed differential evolution (#1366) * High-speed differential evolution * Update base.py * Update optimizerlib.py * fix * fix * clean * clean * Update differentialevolution.py * Update nevergrad/optimization/differentialevolution.py Co-authored-by: Jérémy Rapin <[email protected]> * Update nevergrad/optimization/differentialevolution.py Co-authored-by: Jérémy Rapin <[email protected]> * fi * fix * fix * fix Co-authored-by: Jérémy Rapin <[email protected]> * cleaning * cleaning * cleaning Co-authored-by: ncarraz <[email protected]> Co-authored-by: Jérémy Rapin <[email protected]> Co-authored-by: mathuvu <[email protected]> Co-authored-by: Mathurin Videau <[email protected]> Co-authored-by: Jeremy Reizenstein <[email protected]> Co-authored-by: Dmitry Vinnik <[email protected]>

teytaud · 2022-03-15T14:42:26Z

There is a lot of very redundant code, you could easily change the target depending on the case instead

The different tests are distinct though. Some cases are continuous and others are discrete. Also, in some cases we just check "if the optimum has been suggested then we find it" and in other cases we check the stronger requirement "if a good suboptimal point has been provided then we find the optimum".

I think compacting more would decrease readability.

But increase cost of maintenance, there is too much shared code here, I am sure it's possible to be both readable and factorized

ok for you now ?

jrapin

given it's added 3 very specific tests on one particular topic, it may be better to move to a new file test_suggest.py

nevergrad/optimization/test_optimizerlib.py

jrapin · 2022-03-15T18:18:11Z

nevergrad/optimization/test_optimizerlib.py

+def good_at_c0_suggest(r: str) -> bool:
+    return "NGOpt" == r  # Let us check on NGOpt only for now.


is that really needed as a function?

jrapin

the code seems very redundant, but not sure it could be restructured better

Co-authored-by: Jérémy Rapin <[email protected]>

…suggest

teytaud · 2022-03-18T07:59:33Z

the code seems very redundant, but not sure it could be restructured better

is it ok for you now ? i refactored cycling.py but not the underlying computations: i assume that original authors would be lost if I changed the deep stuff.

jrapin

I would still move all suggestions tests to one test_suggest.py file because we start to have quite a few of them, and test_optimizerlib is too big

testing_suggest

04f56c9

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Dec 3, 2021

teytaud added 5 commits December 3, 2021 09:28

fix

63d1667

fix

b83e40b

fix

a8ccd92

fix

632848e

fix

7a13bc9

jrapin requested changes Dec 6, 2021

View reviewed changes

nevergrad/optimization/test_optimizerlib.py Outdated Show resolved Hide resolved

nevergrad/optimization/test_optimizerlib.py Outdated Show resolved Hide resolved

jrapin changed the title ~~testing suggest~~ Add tests for the suggest method Dec 6, 2021

teytaud and others added 3 commits December 14, 2021 09:05

Update nevergrad/optimization/test_optimizerlib.py

cd98640

Co-authored-by: Jérémy Rapin <[email protected]>

Merge branch 'main' of github.com:facebookresearch/nevergrad into sug…

e8d7b6e

…gest

Merge branch 'suggest' of github.com:facebookresearch/nevergrad into …

cf6537c

…suggest

Update nevergrad/optimization/test_optimizerlib.py

22c4483

Co-authored-by: Jérémy Rapin <[email protected]>

teytaud and others added 3 commits March 14, 2022 09:46

fix

6ad413a

Merge branch 'main' into suggest

5cbc120

jrapin reviewed Mar 15, 2022

View reviewed changes

teytaud and others added 6 commits March 16, 2022 15:36

Update nevergrad/optimization/test_optimizerlib.py

2a98e4c

Co-authored-by: Jérémy Rapin <[email protected]>

Update test_optimizerlib.py

473aa33

fix

70e8c38

Merge branch 'suggest' of github.com:facebookresearch/nevergrad into …

8bf6104

…suggest

Update test_optimizerlib.py

08712e9

Update test_optimizerlib.py

da3e7d3

jrapin approved these changes Mar 21, 2022

View reviewed changes

teytaud and others added 3 commits March 21, 2022 17:39

fix

da5df91

Update test_suggest.py

b9f8cd3

fix

cbad33b

teytaud merged commit 213bbc8 into main Mar 22, 2022

teytaud deleted the suggest branch March 22, 2022 07:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tests for the suggest method #1316

Add tests for the suggest method #1316

teytaud commented Dec 3, 2021

jrapin left a comment

teytaud commented Dec 15, 2021

teytaud commented Dec 15, 2021

jrapin commented Dec 17, 2021

teytaud commented Mar 15, 2022

jrapin left a comment

jrapin Mar 15, 2022

teytaud Mar 16, 2022

jrapin left a comment

teytaud commented Mar 18, 2022

jrapin left a comment

		def good_at_c0_suggest(r: str) -> bool:
		return "NGOpt" == r # Let us check on NGOpt only for now.

Add tests for the suggest method #1316

Add tests for the suggest method #1316

Conversation

teytaud commented Dec 3, 2021

Types of changes

Motivation and Context / Related issue

How Has This Been Tested (if it applies)

Checklist

jrapin left a comment

Choose a reason for hiding this comment

teytaud commented Dec 15, 2021

teytaud commented Dec 15, 2021

jrapin commented Dec 17, 2021

teytaud commented Mar 15, 2022

jrapin left a comment

Choose a reason for hiding this comment

jrapin Mar 15, 2022

Choose a reason for hiding this comment

teytaud Mar 16, 2022

Choose a reason for hiding this comment

jrapin left a comment

Choose a reason for hiding this comment

teytaud commented Mar 18, 2022

jrapin left a comment

Choose a reason for hiding this comment