Add ability to filter VIIRS AOD product image using Quality Flag

[kathys]

I would like the ability with P2G to be able to filter the VIIRS EDR AOD product using the QCAll array. I would like to start by creating images using only the pixels that have a QCAll value of 0: high or 1: medium.

The VIIRS EDR AOD images are very nice, but there is alot of AOD that is retrieved in the image that look like either ground features, or in areas that look relatively clean. Image below.

I have a good NOAA20 example data set from over the summer.

bumi:/data/users/kathys/test_data/viirs_edr/j01/summer/asci/

JRR-AOD_v3r0_j01_s202306081920003_e202306081921248_c202306082036337.nc
JRR-AOD_v3r0_j01_s202306081921261_e202306081922506_c202306082036337.nc
JRR-AOD_v3r0_j01_s202306081922518_e202306081924163_c202306082036337.nc
JRR-AOD_v3r0_j01_s202306081924176_e202306081925403_c202306082036337.nc
JRR-AOD_v3r0_j01_s202306081925415_e202306081927061_c202306082039376.nc
JRR-AOD_v3r0_j01_s202306081927073_e202306081928318_c202306082040351.nc
JRR-AOD_v3r0_j01_s202306081928331_e202306081929576_c202306082040574.nc
JRR-AOD_v3r0_j01_s202306081929588_e202306081931215_c202306082041046.nc
JRR-AOD_v3r0_j01_s202306081931228_e202306081932473_c202306082043028.nc
JRR-AOD_v3r0_j01_s202306081932485_e202306081934131_c202306082043167.nc
JRR-AOD_v3r0_j01_s202306081934143_e202306081935388_c202306082043285.nc
JRR-AOD_v3r0_j01_s202306081935401_e202306081937028_c202306082043301.nc
JRR-AOD_v3r0_j01_s202306081937040_e202306081938285_c202306082045500.nc

[djhoese]

Would a --filter-aod be good enough and we'll settle on a hardcoded 0 or 1 for the filtering? Or maybe I should make it something like --aod-qc-filter 0 (or 1) to control it and we can choose the default in Polar2Grid.

[kathys]

That would be really useful. I like the --aod-qc-filter 0 option idea. I would make the default to be 0, but the QC flag can be a value between 0 and 3:
QCAll:long_name = Retrieval quality: 0: high; 1: medium; 2: low; 3: no retrieval
So let the user choose another one of these values if they want.

[djhoese]

I suppose, now that I'm actually trying to implement this, that this value has to be to keep anything where qcall <= X. Let me know if this is flawed thinking.

[djhoese]

Oh and technically a user could load any variable from the AOD files, but I'm guessing this should only be applied to the AOD550 variable? I'm not sure of another more generic rule to apply.

[kathys]

Trying to think about the qcall <=X option. If you choose 1, then it would filter anything that had a 0 or a 1 value? That is not what you would want. You really do want to filter based upon the actual values of either 0,1,2 or 3.

And yes, since we are only officially supporting AOD550 from these files, we only want the filtering to apply to that array.

[djhoese]

I'm not sure that makes sense. If you said "I want to include quality 1 - medium" then you'd also want the "high" (0) quality pixels too. The easiest way for me to implement this was to consider the specified value as the "maximum quality integer to include". So if you specify 1, you get 0 and 1. If you specify 2, you get 2, 1, and 0. Maybe "filter" isn't the best way of phrasing that?

[kathys]

Yes, you are right. I was not thinking about it correctly. You original plan will work fine I think ( qcall <=X ).

[djhoese]

No filtering

0

1

2

3

This would be everything marked as "no retrieval" or better so I didn't generate it as I think the only pixels that would be different from "No filtering" are where QCAll is invalid (-128).

[kathys]

Thanks Dave. I was not able to download the file filtered using 2 for some reason. If you click on it, it is blank, it does not expand.

[kathys]

I took a look at these images. Here are my thoughts:

• I could not see a difference between the unfiltered, and the low ( QCall = 2 ).
• There is a huge difference between 1: medium; 2: low. I wish there was a category between them.
• I guess I would choose 1:medium as the default. 0:high filters out too many good retrievals I think.