Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tolerance in tests ? #276

Open
Abelarm opened this issue Apr 15, 2024 · 3 comments
Open

Tolerance in tests ? #276

Abelarm opened this issue Apr 15, 2024 · 3 comments
Labels

Comments

@Abelarm
Copy link
Contributor

Abelarm commented Apr 15, 2024

Hi @phoebe-p,

I keep getting annoying messages from Github about test failing in github action, for my fork of solcore.
All of them are because the tolerance is way too low.
I checked and also the test for solcore they usually fail for the same reason

I wanted to ask if is it by design that solcore need this strict tolerance or we can relax it a bit to not make the test fail?

Examples:
image
image
image

@phoebe-p
Copy link
Member

Sorry, missed this somehow. We can definitely increase the tolerance on the tests, this is a silly reason to have failed tests.

@Abelarm
Copy link
Contributor Author

Abelarm commented Jun 8, 2024

No worries, what's a good tolerance we like?

@phoebe-p
Copy link
Member

phoebe-p commented Jun 27, 2024

I've increased the tolerance for some tests I have seen failing due to this problem in #278. There are probably others that I missed. There were a couple where no tolerance was set in approx at all (so it's using the default of 1e-12 absolute/1e-6 relative), which I set to a relative tolerance of 1e-5. There were some other tests where a tolerance of 1e-4 was set which sometimes failed, changed those to 1e-3. let me know if it keeps happening with other tests! I will also keep an eye on the daily scheduled tests to see if there are other failures related to this (now that they are working again not failing for other reasons in the build system...)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants