Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make it possible to infer noise #1345

Closed
wants to merge 1 commit into from
Closed

Conversation

dme65
Copy link
Contributor

@dme65 dme65 commented Dec 28, 2022

Summary: We currently set the noise to 0 for all benchmark problems even though we often infer the noise in practice. This diff adds an infer_noise flag that we can use to control whether different problems should set the noise to a fixed value or None, where the latter will trigger inferring the noise in the model. I'm setting infer_noise=True by default for all problems.

Reviewed By: Balandat

Differential Revision: D39413742

@facebook-github-bot facebook-github-bot added CLA Signed Do not delete this pull request or issue due to inactivity. fb-exported labels Dec 28, 2022
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D39413742

dme65 pushed a commit to dme65/Ax that referenced this pull request Dec 28, 2022
Summary:
Pull Request resolved: facebook#1345

We currently set the noise to 0 for all benchmark problems even though we often infer the noise in practice. This diff adds an `infer_noise` flag that we can use to control whether different problems should set the noise to a fixed value or `None`, where the latter will trigger inferring the noise in the model. I'm setting `infer_noise=True` by default for all problems.

Reviewed By: Balandat

Differential Revision: D39413742

fbshipit-source-id: bdbe81048008b346f22d82552fb2a0074a1b1207
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D39413742

dme65 pushed a commit to dme65/Ax that referenced this pull request Dec 28, 2022
Summary:
Pull Request resolved: facebook#1345

We currently set the noise to 0 for all benchmark problems even though we often infer the noise in practice. This diff adds an `infer_noise` flag that we can use to control whether different problems should set the noise to a fixed value or `None`, where the latter will trigger inferring the noise in the model. I'm setting `infer_noise=True` by default for all problems.

Reviewed By: saitcakmak, Balandat

Differential Revision: D39413742

fbshipit-source-id: 844960e3ab90c19b14c3615b3d91aceea60f2930
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D39413742

@codecov-commenter
Copy link

codecov-commenter commented Dec 28, 2022

Codecov Report

Base: 94.67% // Head: 94.67% // Decreases project coverage by -0.00% ⚠️

Coverage data is based on head (044095a) compared to base (5e61497).
Patch coverage: 88.88% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1345      +/-   ##
==========================================
- Coverage   94.67%   94.67%   -0.01%     
==========================================
  Files         420      420              
  Lines       38562    38565       +3     
==========================================
+ Hits        36508    36510       +2     
- Misses       2054     2055       +1     
Impacted Files Coverage Δ
ax/benchmark/problems/hpo/torchvision.py 64.70% <ø> (ø)
ax/benchmark/problems/registry.py 100.00% <ø> (ø)
ax/benchmark/problems/synthetic/hss/jenatton.py 100.00% <ø> (ø)
ax/storage/json_store/decoder.py 100.00% <ø> (ø)
ax/storage/json_store/encoders.py 100.00% <ø> (ø)
ax/benchmark/problems/surrogate.py 58.18% <50.00%> (-1.82%) ⬇️
ax/benchmark/benchmark_problem.py 100.00% <100.00%> (ø)
ax/benchmark/problems/hpo/pytorch_cnn.py 51.57% <100.00%> (+0.51%) ⬆️
ax/metrics/botorch_test_problem.py 88.88% <100.00%> (ø)
ax/metrics/jenatton.py 51.85% <100.00%> (+1.85%) ⬆️
... and 1 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Summary:
Pull Request resolved: facebook#1345

We currently set the noise to 0 for all benchmark problems even though we often infer the noise in practice. This diff adds an `infer_noise` flag that we can use to control whether different problems should set the noise to a fixed value or `None`, where the latter will trigger inferring the noise in the model. I'm setting `infer_noise=True` by default for all problems.

Reviewed By: saitcakmak, Balandat

Differential Revision: D39413742

fbshipit-source-id: e50ea9d684411a5da09d06899b1a11dac9aca3b3
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D39413742

@facebook-github-bot
Copy link
Contributor

This pull request has been merged in 9b8df90.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed Do not delete this pull request or issue due to inactivity. fb-exported Merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants