Consider adding option to ABI L1b reader for lower clipping radiances

[djhoese]

I was going through old emails between some NOAA/CIMSS folks and @simonrp84 where we discussed ways to fix holes in the ABI C07 imagery that shows up in Geo2Grid/GeoSphere, but I don't think it ever resulted in an issue here. The problem we're trying to solve is that sometimes C07 has negative radiances and these result in invalid BTs when the radiances are converted to BTs. The suggestion from Mat Gunshor and Tim Schmidt was to replace the values with the minimum possible BT for a band. We determined that the best way to do that is to (quote from my response in the email thread):

If you assume you are looking for the lowest count where Rad is 0 or greater then you can do:

rad = count * scale_factor + add_offset

which for rad=0 and solving for count becomes:

count = -add_offset / scale_factor

If we then do a "ceiling" operation on that we should get the integer count we're looking for. And indeed when I do that with the file I've been playing with I get 197.5K. 

My one concern was that this assumes add_offset is always negative. Mat responded with:

add_offset will not be positive for any of the GOES-R series ABI (GOES-16 through -19) because of the design to allow for negative radiances.  I can't be sure that's true for any other instruments though. 

I think this behavior could be added to Satpy with an additional keyword argument, but in Satpy it would probably not be the default given how much we're "fudging" the data. I'm also not sure if it should be done for radiances or only during the conversion to BT.

[djhoese]

This is now available as a reader keyword argument now that pytroll/satpy#2499 has been merged.

[djhoese]

@kathys Let's talk about this in our meeting today. FYI to @graemely, this feature as a default would fix the green "speckling" seen in GeoSphere's night microphysics composites.

[djhoese]

It took me forever to find this message from @simonrp84 on the pytroll slack about the speckling in the night microphysics ABI images so I'm putting it here for future reference. This is about how even after the feature mentioned in this issue was added and tested C07-based night microphysics images still had speckling and there doesn't seem to be a reasonable way to get rid of it.

OK, looking at the channel data, the green spots are not due to negative radiances - they're due to the quantisation applied either in the onboard ADC or via the ground segment.
At those cold temperatures, the fidelity of the digital counts if very low. So incrementing the digital count by 1 has a huge effect on the measured brightness temperature. It looks like a digital count of 1 (i.e: lowest measurable) is ~197K while a digital count of 2 is ~205K.

Nothing can be done about that unfortunately, it's just a side effect of the bit depth of the data.

[graemely]

Have you checked for specking in all of the input bands? It seems odd to me that an RGB formula would be that sensitive to changes of a single step in representable input values.

[djhoese]

I think the question is what the lower (or higher) ends of each of the bands is. More likely than not the range of values we're interested in in the RGBs is within the "good range" of the band.

If I'm remembering correctly C07 is good for detecting fires, right? The speckles we're seeing are in the cold clouds in a band used for detecting hot temperatures...at least that's what I'm telling myself. I'm not sure if other RGBs have band ranges like that or that single band products use limits at those extremes.

[simonrp84]

Yes, that is correct. The MIR band has sensitivity at the high end for fire detection but loses sensitivity at the low end. Other bands are more equally tuned for high/low radiances and so rarely suffer from this problem.