Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Test estimators with hypothesis #4960

Open
1 of 5 tasks
csadorf opened this issue Oct 28, 2022 · 0 comments
Open
1 of 5 tasks

[FEA] Test estimators with hypothesis #4960

csadorf opened this issue Oct 28, 2022 · 0 comments
Labels
2 - In Progress Currenty a work in progress ? - Needs Triage Need team to review and classify feature request New feature or request

Comments

@csadorf
Copy link
Contributor

csadorf commented Oct 28, 2022

Many of cuml's tests are aimed at comparing results between different implementations, e.g., the GPU and CPU implementation of the same estimator and against third-party implementations, notably scikit-learn. We expect estimators to overall behave very similarly and their results to be identical up to numerical precision.

Further, estimators are usually tested only against a specific combination of inputs and example datasets, an approach that likely fails to test rare edge cases and cannot provide confidence for the equivalence of a wide-range of inputs and datasets. Using hypothesis to test estimators and compare results has therefore two positive effects:

  1. The API surface is tested significantly more often against edge cases and extreme values.
  2. Any validation through the comparison of results is done on a more diverse set of input datasets.

A potential downside is an increase in test implementation complexity and test runtime. The former can be mitigated through a well-designed abstraction of hypothesis-strategies and may then actually lead to a reduction of complexity, the latter can be mitigated through limiting the number of hypothesis iterations and potentially only running hypothesis tests as part of the stress tests.

I suggest the following break-down for implementation:

#4960 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2 - In Progress Currenty a work in progress ? - Needs Triage Need team to review and classify feature request New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants
@csadorf and others