Add BinarySemanticSegmentationTask to properly compute IoU

[calebrob6]

ETCI2021 is a binary segmentation task with a positive class and a background class. As such, the IoU metric should only be computed over the positive class (i.e. computing IoU), however currently the 2 class mIoU is computed.

[adamjstewart]

@calebrob6 I think you mentioned that you hacked this to get this working for the paper. What did that hack look like? We can create a BinarySemanticSegmentationTask that subclasses SemanticSegmentationTask and includes this hack.

[sofstef]

@adamjstewart @calebrob6 is this something you're still planning on adding? I'm having a go at creating a binary segmentation task and it would be helpful to know if you've already done something similar!

[adamjstewart]

I don't think either of us have worked on this yet, so if you get one working that would be great!

[remtav]

Another binary semantic segmentation scenario is a Spacenet-type dataset, where labels come in as (N, 1, W, H), not (N, 2, W, H), ie. no background class. It would be great if the BinarySemanticSegmentationTask covers this scenario, using the sigmoid() operation rather than argmax(dim=1), for example. Also, as a heads up for whoever develops this new task, torchmetrics seems to have a consistency issue for this use case. I've been performing tests with the Spacenet1 data and haven't been able to get IoU nor accuracy working yet with torchmetrics, even with some hacks.

[ashnair1]

The next release of torchmetrics (v0.10.0) will contain binary, multi-class and multi-label variants of all the classification metrics. An RC has already been released.

[isaaccorley]

Should we actually have a task for binary segmentation with sigmoid outputs where 0 presumably means background or should we force binary datasets to have a background class so everything can be multiclass for consistency. I guess I'm not seeing the benefit of having a separate task over keeping the datasets consistent?

[ashnair1]

Forcing binary datasets to have a background class would be the simpler option. I had trained a model on the Inria dataset in the same way (setting num_classes=2) and it worked as expected. We could go one step further and always handle background classes. This way users would just specify num_classes = N in yaml config instead of N + 1 for segmentation and object detection tasks.

If we were to separate it out into a new BinarySemanticSegmentation task we would need to:

• Switch to binary metrics
• Switch to binary losses (BCE instead of CE) or enable binary mode for losses (Jaccard(mode=binary))
• Change the forward methods to provide sigmoid outputs.

[adamjstewart]

Given that torchmetrics is splitting binary and multi-class into separate metrics, I think it would make sense to have separate tasks. At the very least, there should be an option in SemanticSegmentationTask that allows you to select between binary and multi-class.