Merge pull request #703 from ssec/dockathys

Update documentation for Polar2Grid 3.1.0
ssec · Aug 13, 2024 · 038d017 · 038d017
2 parents 9135377 + a0cc8b8
commit 038d017
Show file tree

Hide file tree

Showing 11 changed files with 293 additions and 65 deletions.
diff --git a/NEWS.rst b/NEWS.rst
@@ -1,6 +1,21 @@
 Release Notes
 =============
 
+Version 3.1.0 (2024-08-13)
+--------------------------
+
+* Support for additional VIIRS EDRs, including
+
+  * Aerosol Optical Depth (AOD)
+  * Cloud Top Height and Cloud Top Temperature
+  * Surface Reflectances, including True Color
+  * Vegetation Indices (NDVI, EVI)
+
+* Support added for FY-3E MERSI-LL instrument
+* Re-organization of software directories
+* Optimizations
+* Bug fixes
+
 Version 3.0.0 (2023-02-21)
 --------------------------
 

diff --git a/doc/source/conf.py b/doc/source/conf.py
@@ -77,6 +77,11 @@
     "https://bin.ssec.wisc.edu/pub/CSPP/p2g_v_3_0_examples/acspo/noaa20_viirs_sst_20220810_184327_great_lakes_rescaled_wcolor.png",
     "https://bin.ssec.wisc.edu/pub/CSPP/p2g_v_3_0_examples/acspo/noaa20_viirs_sst_20220810_184327_great_lakes_rescaled_wcolor_colortable_resize.png",
     "https://bin.ssec.wisc.edu/pub/CSPP/p2g_v_3_0_examples/acspo/noaa20_viirs_sst_20220810_184327_great_lakes_sst_final_resize.png",
+    "https://bin.ssec.wisc.edu/pub/CSPP/p2g_v_3_0_examples/asci/VIIRS_AOD_example.png",
+    "https://bin.ssec.wisc.edu/pub/CSPP/p2g_v_3_0_examples/asci/VIIRS_AOD_example_with_overlays.png",
+    "https://bin.ssec.wisc.edu/pub/CSPP/p2g_v_3_0_examples/asci/VIIRS_CTT_example_with_overlays.png",
+    "https://bin.ssec.wisc.edu/pub/CSPP/p2g_v_3_0_examples/asci/VIIRS_NDVI_example_with_overlays.png",
+    "https://bin.ssec.wisc.edu/pub/CSPP/p2g_v_3_0_examples/asci/VIIRS_SurfReflectance_True_Color_example_with_overlays.png",
     "https://bin.ssec.wisc.edu/pub/CSPP/p2g_v_2_1_examples/acspo/npp_viirs_sst_20191216_072134_acspo_sst_rescaled_wcolor.png",
     "https://bin.ssec.wisc.edu/pub/CSPP/p2g_v_2_1_examples/acspo/npp_viirs_sst_20191216_072134_acspo_sst_rescaled_wcolor_colortable_resize.png",
     "https://bin.ssec.wisc.edu/pub/CSPP/p2g_v_2_1_examples/acspo/npp_viirs_sst_20191216_072134_acspo_sst_final_resize.png",
@@ -259,6 +264,7 @@ def setup(app):
             "examples/amsr2_example.rst",
             "examples/modis_example.rst",
             "examples/viirs_example.rst",
+            "examples/asci_example.rst",
             "readers/acspo.rst",
             "readers/amsr2_l1b.rst",
             "readers/avhrr.rst",

diff --git a/doc/source/examples/acspo_example.rst b/doc/source/examples/acspo_example.rst
@@ -31,12 +31,12 @@ band output file with a black background.
 
 .. code-block:: bash
 
-    polar2grid.sh -r acspo -w geotiff acspo gtiff --grid-coverage 0 --fill-value 0 \
+    polar2grid.sh -r acspo -w geotiff --grid-coverage 0 --fill-value 0 \
       -f viirs/20220810184327-CSPP-L2P_GHRSST-SSTskin-VIIRS_N20-ACSPO_V2.80-v02.0-fv01.0.nc
 
 The data set is re-projected into the WGS84 (Platte Carrée) projection
-by default. The image scaling is defined in the ``generic.yaml`` located in the
-``$POLAR2GRID_HOME/libexec/python_runtime/etc/polar2grid/enhancements`` directory.
+by default. The image scaling is defined in the ``generic.yaml`` file located in the
+``$POLAR2GRID_HOME/etc/polar2grid/enhancements`` directory.
 This file contains product scaling information for all data parameters supported by
 Polar2Grid. It replaces the ``rescale.ini`` file that was used in previous versions of Polar2Grid.
 
@@ -47,12 +47,12 @@ references our SST product is listed below.
 
 .. parsed-literal::
 
-      395   sea_surface_temperature4:
-      396     standard_name: sea_surface_subskin_temperature
-      397     operations:
-      398       - name: linear_stretch
-      399         method: !!python/name:satpy.enhancements.stretch
-      400         kwargs: {stretch: 'crude', min_stretch: 267.317, max_stretch: 309.816}
+      404   sea_surface_temperature4:
+      405     standard_name: sea_surface_subskin_temperature
+      406     operations:
+      407       - name: linear_stretch
+      408         method: !!python/name:satpy.enhancements.stretch
+      409         kwargs: {stretch: 'crude', min_stretch: 267.317, max_stretch: 309.816}
 
 This is used in the Polar2Grid software to define the range of brightness
 values in the output GeoTIFF file (0-255) to the temperatures they represent - in this

diff --git a/doc/source/examples/amsr2_example.rst b/doc/source/examples/amsr2_example.rst
@@ -64,7 +64,7 @@ page:  http://www.nrlmry.navy.mil/TC.html
 First, create a reprojected GeoTIFF in Lambert Conic Conformal (LCC) projection
 and rescale the data. The data in this example is from 10 September 2022. We are pointing
 to the rescale information that is stored in the `$POLAR2GRID_HOME/example_enhancements/amsr2_png/enhancements/generic.yaml` file.
-This will produce a linear scaled output of data rangeing from 180.0 K to 280.0 K brightness temperatures
+This will produce a linear scaled output of data ranging from 180.0 K to 280.0 K brightness temperatures
 for our default products.
 
 .. code-block:: bash
@@ -83,11 +83,11 @@ Executing this command produces these AMSR2 LCC GeoTIFF files:
     gcom-w1_amsr2_btemp_89\.0bv_20220910_233500_lcc_fit\.tif
 
 Once the data has been rescaled, you are ready to apply the NRL colormaps to the data. In this example we
-are using the 89A/H GHz file.
+are using the 89A/H GHz file. There is also a 36 GHz colormap that can be used.
 
 .. code-block:: bash
 
-    add_colormap.sh $POLAR2GRID_HOME/libexec/python_runtime/etc/polar2grid/colormaps/amsr2_89h.cmap gcom-w1_amsr2_btemp_89.0ah_20220910_233500_lcc_fit.tif
+    add_colormap.sh $POLAR2GRID_HOME/libexec/python_runtime/lib/python3.11/site-packages/polar2grid/etc/colormaps/amsr2_89h.cmap gcom-w1_amsr2_btemp_89.0ah_20220910_233500_lcc_fit.tif
 
 This command adds the enhancement to the original GeoTIFF.  The rescaled and final color enhanced product are shown below:
 

diff --git a/doc/source/examples/asci_example.rst b/doc/source/examples/asci_example.rst
@@ -0,0 +1,222 @@
+.. raw:: latex
+
+    \newpage
+
+Creating VIIRS Land and Atmosphere Product Images
+-------------------------------------------------
+
+This set of examples demonstrates how you can create VIIRS high quality
+land and atmosphere color enhanced science product images using Polar2Grid.
+The example data set is a NOAA-20 VIIRS direct broadcast 7 granule overpass
+starting at 18:50 UTC, 5 June 2024, and covers a region from
+Central America to a portion of North America over the United States.
+
+Find the options available for creating VIIRS Environmental Data Record (EDR) GeoTIFFs:
+
+   ``polar2grid.sh -r viirs_edr -w geotiff -h``
+
+List the products that can be created from your VIIRS EDR NetCDF
+files. In this case we have the VIIRS Aerosol Optical Depth (JRR-AOD*.nc),
+VIIRS Cloud Height (JRR-CloudHeight*.nc) and VIIRS Land Surface Reflectance
+(SurfRefl*.nc) NetCDF files available for our 7 granule overpass.
+
+.. code-block:: bash
+
+    polar2grid.sh -r viirs_edr -w geotiff --list-products -f *j01_s20240605185*.nc
+
+Execution of this command results in the following list of Standard Products:
+
+.. parsed-literal::
+
+    ### Standard Available Polar2Grid Products
+           AOD550
+           CldTopHght
+           CldTopTemp
+           EVI
+           NDVI
+           surf_refl_i01
+           surf_refl_i02
+           surf_refl_i03
+           surf_refl_m01
+           surf_refl_m02
+           surf_refl_m03
+           surf_refl_m04
+           surf_refl_m05
+           surf_refl_m07
+           surf_refl_m08
+           surf_refl_m10
+           surf_refl_m11
+           true_color_surf
+
+The standard Land Surface Reflectance products include Normalized Difference Vegetation Index
+(NDVI), Enhanced Vegetation Index (EVI), and a true color image created from the surface
+reflectance bands (`true_color_surf`). Polar2Grid defaults to producing all available
+Standard Products. Users can create one or more selected products by using the `-p` option.
+For instance, if I want to create just an image of the Aersol Optical Depth (AOD) for these
+data files, I would use this command:
+
+.. code-block:: bash
+
+    polar2grid.sh -r acspo -w geotiff -p AOD550 -f viirs/JRR-AOD_*j01_s20240605185*.nc
+
+An aggregated GeoTIFF image will be created from the 7 granule input files with the data
+re-projected into the WGS84 (Platte Carrée) projection by default. The image scaling
+is defined in the ``viirs.yaml`` file located in the
+``$POLAR2GRID_HOME/etc/polar2grid/enhancements`` directory.
+This file contains VIIRS product scaling information.
+
+The default scaling used for the VIIRS AOD files can be found under
+`aod550`. The section of the viirs.yaml file that references the VIIRS AOD
+product is listed below.
+
+.. parsed-literal::
+
+      136   aod550:
+      137     name: AOD550
+      138     sensor: viirs
+      139     operations:
+      140       - name: colorize
+      141         method: !!python/name:polar2grid.enhancements.colorize
+      142         kwargs:
+      143           palettes:
+      144             - min_value: 0.0
+      145               max_value: 1.0
+      146               colors: "rainbow"
+
+This is used in the Polar2Grid software to scale the range of brightness
+values in the output GeoTIFF file (0-255) to the AOD values they represent - in this
+case 0.0 to 1.0. In addition, this product is by default color enhanced using the
+``rainbow`` color map. AOD values above 1.0 are color enhanced using the last color value (dark red).
+The scaling is done linearly. AOD values are filtered based upon a Quality Flag (QCAll)
+that is either 0 or 1 (high or medium).  The output GeoTIFF image below shows the
+end result of the polar2grid.sh command execution for this data.
+
+.. raw:: latex
+
+    \newpage
+
+.. figure:: ../_static/example_images/VIIRS_AOD_example.png
+    :name: VIIRS_AOD_example.png
+    :width: 80%
+    :align: center
+
+    CSPP NOAA-20 VIIRS Aerosol Optical Depth GeoTIFF image from 5 June 2024, 18:50 UTC (noaa20_viirs_AOD550_20240605_185031_wgs84_fit.tif).
+
+Note that AOD retrievals are not made in sun glint regions.
+
+We can add overlays to the image including a color bar, title and maps using the
+``add_coastlines.sh`` script:
+
+.. code-block:: bash
+
+    add_coastlines.sh noaa20_viirs_AOD550_20240605_185031_wgs84_fit.tif --add-colorbar \
+      --colorbar-text-color="black" --colorbar-title="VIIRS Aerosol Optical Depth" \
+      --add-coastlines --coastlines-outline "black" --coastlines-level 1 \
+      --coastlines-resolution=i --add-borders --borders-level 2 --borders-outline "gray" \
+      --borders-width 1 --coastlines-width 2 --colorbar-tick-marks 0.1 \
+      --colorbar-minor-tick-marks 0.05 --colorbar-height 125 --colorbar-text-size 100
+
+More thorough examples of rescaling and adding overlays can be found in the :doc:`acspo_example`.
+
+The annotated image with overlays is shown below.
+
+.. raw:: latex
+
+    \newpage
+
+.. figure:: ../_static/example_images/VIIRS_AOD_example_with_overlays.png
+    :name: VIIRS_AOD_example_with_overlays.png
+    :width: 100%
+    :align: center
+
+    CSPP VIIRS NOAA-20 Aerosol Optical Depth PNG image with added borders, coastlines and an annotated colorbar. The retrievals were created from June 5, 2024, 18:50 UTC observations.
+
+Other CSPP VIIRS EDR product images can be created in a similar manner.
+For example, the Polar2Grid commands to create a VIIRS Cloud Top Temperature
+color enhanced image with overlays are shown below.
+
+.. code-block:: bash
+
+    polar2grid.sh -r viirs_edr -w geotiff -p CldTopTemp -f JRR-CloudHeigh*.nc
+
+    add_coastlines.sh noaa20_viirs_CldTopTemp_20240605_185031_wgs84_fit.tif \
+    --add-colorbar --colorbar-text-color="black" \
+    --colorbar-title="VIIRS Cloud Top Temperature (°K)" --add-coastlines \
+    --coastlines-outline "black" --coastlines-level 1 \
+    --coastlines-resolution=i --add-borders --borders-level 2 \
+    --borders-outline gray --coastlines-width 2 --colorbar-tick-marks 10 \
+    --colorbar-height 125 --colorbar-text-size 100
+
+And the resulting image is shown below:
+
+.. raw:: latex
+
+    \newpage
+
+.. figure:: ../_static/example_images/VIIRS_CTT_example_with_overlays.png
+    :name: VIIRS_CTT_example_with_overlays.png
+    :width: 100%
+    :align: center
+
+    CSPP VIIRS NOAA-20 Cloud Top Temperature PNG image with added borders, coastlines and an annotated colorbar. The retrievals were created from June 5, 2024, 18:50 UTC observations.
+
+Similarly, the commands to create a Normalized Difference Vegetation Index (NDVI)
+color enhanced image with overlays from the VIIRS Surface Reflectance products
+is shown below, followed by the output image.
+
+.. code-block:: bash
+
+    polar2grid.sh -r viirs_edr -w geotiff -p NDVI -f SurfRefl_*.nc
+
+    add_coastlines.sh noaa20_viirs_NDVI_20240605_185031_wgs84_fit.tif \
+    --add-colorbar --colorbar-text-color="red" \
+    --colorbar-title="Normalized Difference Vegetation Index (NDVI)" --add-coastlines \
+    --coastlines-outline "black" --coastlines-level 1 \
+    --coastlines-resolution=i --add-borders --borders-level 2 \
+    --borders-outline gray --coastlines-width 2 --colorbar-tick-marks 10 \
+    --colorbar-height 150 --colorbar-text-size 100
+
+.. raw:: latex
+
+    \newpage
+
+.. figure:: ../_static/example_images/VIIRS_NDVI_example_with_overlays.png
+    :name: VIIRS_NDVI_example_with_overlays.png
+    :width: 100%
+    :align: center
+
+
+    CSPP VIIRS NOAA-20 NDVI PNG image with added borders, coastlines and an annotated colorbar. The retrievals were created from June 5, 2024, 18:50 UTC observations.
+
+Polar2Grid supports the creation of individual band and true color images from VIIRS EDR Land Surface
+Reflectance output files (SurfRefl*.nc).  Surface reflectances differ from
+Top-Of-Atmosphere (TOA) reflectances in that they are corrected to remove the influence
+of the atmosphere, thereby preserving only the portion that is being reflected
+from the surface below. For more information about the CSPP Surface Reflectance products,
+please visit the `CSPP Distribution Website: <https://cimss.ssec.wisc.edu/cspp/viirs_lsr_v1.1.shtml>`_
+
+The commands to create a true color image from the surface reflectance files
+with overlays are shown below.  These images are sharpened to 375 m spatial resolution; the
+images are not cloud cleared nor water cleared, although the reflectances are
+valid only over land.  The commands are followed by the resulting output image.
+
+.. code-block:: bash
+
+    polar2grid.sh -r viirs_edr -w geotiff -p true_color_surf -f SurfRefl_*.nc
+
+    add_coastlines.sh noaa20_viirs_true_color_surf_20240605_185031_wgs84_fit.tif \
+    --add-coastlines --coastlines-outline "black" --coastlines-level 1 \
+    --coastlines-resolution=i --add-borders --borders-level 2 \
+    --borders-outline yellow --coastlines-width 2
+
+.. raw:: latex
+
+    \newpage
+
+.. figure:: ../_static/example_images/VIIRS_SurfReflectance_True_Color_example_with_overlays.png
+    :name: VIIRS_SurfReflectance_True_Color_example_with_overlays.png
+    :width: 100%
+    :align: center
+
+
+    CSPP VIIRS NOAA-20 Land Surface Reflectance True Color image with added borders and coastlines. The retrievals were created from June 5, 2024, 18:50 UTC observations.
diff --git a/doc/source/examples/index.rst b/doc/source/examples/index.rst
@@ -7,6 +7,7 @@ Examples
     :polar2grid:viirs_example
     :polar2grid:modis_example
     :polar2grid:acspo_example
+    :polar2grid:asci_example
     :polar2grid:amsr2_example
     :geo2grid:abi_example
     :geo2grid:abi_l2_example

diff --git a/doc/source/introduction.rst b/doc/source/introduction.rst
@@ -48,20 +48,14 @@ What's New?
 
 .. ifconfig:: not is_geo2grid
 
-    Polar2Grid Version 3.0 is now available. This is a major
-    update that includes changes to basic Polar2Grid execution.
-    These changes bring Polar2Grid in conformity with the execution strategy
-    of Geo2Grid, and takes advantage of the Xarray and Dask python libraries.
+    Polar2Grid Version 3.1 is now available. This is a minor
+    update that includes support for more VIIRS Science Products.
 
-    Please see the example executions listed at the end of every reader
-    description in this document, as well as the updated examples in the
-    :doc:`examples/index` section. Finally, the Appendix includes a
-    longer list of changes and direct comparisons of Polar2Grid V2.3
-    to V3.0 executions. See :doc:`version3_implementation`.
+    Included in this release:
 
     .. include:: NEWS.rst
         :start-line: 6
-        :end-line: 14
+        :end-line: 17
 
     For more details on what's new in this version and past versions see the
     `Polar2Grid Release Notes <https://raw.githubusercontent.com/ssec/polar2grid/main/NEWS.rst>`_
@@ -147,32 +141,33 @@ System Requirements
 
     * Intel or AMD CPU with 64-bit instruction support (2+ cores - 2.4GHz)
     * 16 GB RAM (minimum)
-    * Rocky Linux 9.3; the software has been tested successfully on Rocky Linux 8.9 and (#####)
+    * Rocky 8 or Rocky 9 64-bit Linux (or other compatible 64-bit Linux distribution)
     * 5 GB disk space (minimum)
 
     Improved Execution Times
     -------------------------
 
-    Updates in Polar2grid Version 3.0 result in improved image creation times.  The table
+    We continue to work to improve the efficiencies of Polar2grid.  The table
     below presents a comparison of the unix `real` time required to create
     VIIRS and MODIS imager GeoTIFF files for the given segments of data in the default
     WGS84 projection. In these examples, the default 4 computer threads were used in the
-    Version 3.0 executions. Execution times decrease when fewer bands and smaller data
-    segments are processed.
+    Version 3.1 executions. Execution times decrease when fewer bands and smaller data
+    segments are processed. The table compares the execution times using Polar2Grid Version
+    3.1 with with those of Version 2.3.
 
     **Table of Execution Times for Creating GeoTIFF Default Projection Images**
 
     +------------------+-----------------+-----------------+------------------------+-----------------------+
-    |**Instrument**    |**Polar2Grid**   |**Polar2grid**   |**Polar2Grid2 V2.3 All**|**Polar2Grid V3.0 All**|
-    |**Input**         |**V2.3 True and**|**V3.0 True and**|**Bands plus True**     |**Bands plus True**    |
+    |**Instrument**    |**Polar2Grid**   |**Polar2grid**   |**Polar2Grid2 V2.3 All**|**Polar2Grid V3.1 All**|
+    |**Input**         |**V2.3 True and**|**V3.1 True and**|**Bands plus True**     |**Bands plus True**    |
     |                  |**False Color**  |**False Color**  |**and False Color**     |**and False Color**    |
     +==================+=================+=================+========================+=======================+
     |**VIIRS SDR**     |                 |                 |                        |                       |
-    |10 - 86 second    |    4m52s        |      2m46s      |      12m54s            |       4m32s           |
+    |10 - 86 second    |    4m52s        |      1m35s      |      12m54s            |       4m23s           |
     |granules          |                 |                 |                        |                       |
     +------------------+-----------------+-----------------+------------------------+-----------------------+
     |**MODIS Level 1B**|                 |                 |                        |                       |
-    |3 - 5 minute      |    4m11s        |      3m55s      |      9m08s             |      4m51s            |
+    |3 - 5 minute      |    4m11s        |      2m34s      |      9m08s             |      2m32s            |
     |granules          |                 |                 |                        |                       |
     +------------------+-----------------+-----------------+------------------------+-----------------------+