Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify that nan is supported in zero_division #2535

Open
robmarkcole opened this issue May 12, 2024 · 4 comments
Open

Clarify that nan is supported in zero_division #2535

robmarkcole opened this issue May 12, 2024 · 4 comments
Assignees
Labels
documentation Improvements or additions to documentation enhancement New feature or request v1.3.x
Milestone

Comments

@robmarkcole
Copy link

This option added in #2198 to create equivalence with sklearn also supports passing through a nan if I am not mistaken, based on the implementation of torchmetrics.utilities.compute._safe_divide.
I wanted to clarify that nan can indeed be passed through, ask if any code changed are required to support this officially, and suggest this could be added to the docs which state currently zero_division [float] – Should be 0 or 1., e.g. here

@robmarkcole robmarkcole added the documentation Improvements or additions to documentation label May 12, 2024
Copy link

Hi! thanks for your contribution!, great first issue!

@SkafteNicki SkafteNicki added this to the v1.3.x milestone May 31, 2024
@SkafteNicki
Copy link
Member

Hi @robmarkcole, thanks for raising this issue and sorry for the slow reply from my side.
So currently nan is not a supported value, only 0 and 1 are supported. I can try to look into what it will take to support nan as a value. To my understanding the nan option means in sklearn that values will be ignored when calculating the average. I can try to look into what it will take to implement this into torchmetrics

@SkafteNicki SkafteNicki added the enhancement New feature or request label May 31, 2024
@robmarkcole
Copy link
Author

@SkafteNicki thanks for clarifying, and yes ignoring these values in averages is the goal 👍

@DimitrisMantas
Copy link

DimitrisMantas commented Jun 1, 2024

I'm using v1.4.0post0 and can definitely pass NaN with zero_division to the metrics affected by #2198 and they work as expected (I think) when the value of average is either micro or none but return NaN in the case of macro (bug), so I'm computing it manually from the classwise results. Other metrics such as accuracy and specifity also accept zero_division because of inheritance, but complain when it is set to NaN.

I think that the ideal situation would be for absent and ignored classes to be assigned a NaN score or dropped, but this may not be that straightforward to implement because one class might be absent in one sample in a given batch but not in another, so we probably can't rely on batchwise class supports and have to work sample by sample. You can get StatScores to output samplewise NaNs for both absent and ignored classes, but then reducing this to a batchwise metric is a bit too much for me right now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request v1.3.x
Projects
None yet
Development

No branches or pull requests

4 participants