Feature #2278 qm_docs

docs/Users_Guide/reformat_point.rst

@@ -219,7 +219,31 @@ _____________________
   obs_bufr_var = [ 'QOB', 'TOB', 'ZOB', 'UOB', 'VOB' ];
 
 
-Each PrepBUFR message will likely contain multiple observation variables. The **obs_bufr_var** variable is used to specify which observation variables should be retained or derived. The variable name comes from BUFR file which includes BUFR table. The following BUFR names may be retained: QOB, TOB, ZOB, UOB, and VOB for specific humidity, temperature, height, and the u and v components of winds. The following BUFR names may be derived: D_DPT, D_WIND, D_RH, D_MIXR, D_PRMSL, D_PBL, D_CAPE, and D_MLCAPE for dew point, wind speed, relative humidity, mixing ratio, pressure reduced to MSL, planetary boundary layer height, convective available potential energy, and mixed layer convective available potential energy. This configuration replaces **obs_grib_code**. If the list is empty, all BUFR variables are retained.
+Each PrepBUFR message will likely contain multiple observation variables. The **obs_bufr_var** variable is used to specify which observation variables should be retained or derived. The observation variable names are retrieved from the BUFR table embedded within the file. Users can run PB2NC with the **-index** command line argument to list out the variable names present in the file, and those names can be listed in this setting. If the list is empty, all BUFR variables present in the file are retained. This setting replaces the deprecated **obs_grib_code**.
+
+
+The example **obs_bufr_var** setting above retains observations of QOB, TOB, ZOB, UOB, and VOB for specific humidity, temperature, height, and the u and v components of winds. Observations of those types are reported at the corresponding POB pressure level. In addition, PB2NC can derive several other variables from these observations. By convention, all observations that are derivable are named with a **D_** prefix:
+
+• **D_DPT** for dew point (from POB and QOB)
+
+• **D_WDIR** for wind direction (from UOB and VOB)
+
+• **D_WIND** for wind speed (from UOB and VOB)
+
+• **D_RH** for relative humidity (from POB, QOB, and TOB)
+
+• **D_MIXR** for mixing ratio (from QOB)
+
+• **D_PRMSL** for pressure reduced to mean sea level (from POB, TOB, and ZOB)
+
+• **D_PBL** for planetary boundary layer height (from POB, QOB, TOB, ZOB, UOB, and VOB)
+
+• **D_CAPE** for convective available potential energy (from POB, QOB, and TOB)
+
+• **D_MLCAPE** for mixed layer convective available potential energy (from POB, QOB, and TOB)
+
+
+In BUFR, lower quality mark values indicate higher quality observations. The quality marks for derived observations are computed as the maximum of the quality marks for its components. For example, **D_DPT** derived from **POB** with quality mark 1 and **QOB** with quality mark 2 is assigned a quality mark value of 2. **D_PBL**, **D_CAPE**, and **D_MLCAPE** are derived using data from multiple vertical levels. Their quality marks are computed as the maximum of their components over all vertical levels.
 
 _____________________
 
@@ -302,6 +326,9 @@ The summaries will only be calculated for the observations specified in the **gr
 
 The **vld_freq** and **vld_thresh** entries specify the required ratio of valid data for an output time summary value to be computed. This option is only applied when these entries are set to non-zero values. The **vld_freq** entry specifies the expected frequency of observations in seconds. The width of the time window is divided by this frequency to compute the expected number of observations for the time window. The actual number of valid observations is divided by the expected number to compute the ratio of valid data. An output time summary value will only be written if that ratio is greater than or equal to the **vld_thresh** entry. Detailed information about which observations are excluded is provided at debug level 4.
 
+
+The quality mark for time summaries is always reported by PB2NC as bad data. Time summaries are computed by several MET point pre-processing tools using common library code. While BUFR quality marks are integers, the quality flags for other point data formats (MADIS NetCDF, for example) are stored as strings. MET does not currently contain logic to determine which quality flag strings are better or worse. Note however that any point observation whose quality mark does not meet the **quality_mark_thresh** criteria is not used in the computation of time summaries.
+
 .. _pb2nc output:
 
 pb2nc output

scripts/utility/print_pointnc2ascii.py

@@ -84,7 +84,8 @@ def dump(self, out_handler, show_header=True, use_comma=False):
             obs_val = [ f'{i:.0f}' for i in self.obs_val ]
         else:
             obs_precision = self.get_precision(self.obs_val)
-            obs_val = [ f'{i:.1f}' if abs((i*10)-int(i*10)) < 0.000001 else
+            obs_val = [ 'NA'       if np.ma.is_masked(i) else
+                        f'{i:.1f}' if abs((i*10)-int(i*10)) < 0.000001 else
                         f'{i:.1f}' if abs((i*10)-int(i*10)) > 0.999998 else
                         f'{i:.2f}' if abs((i*100)-int(i*100)) < 0.000001 else
                         f'{i:.2f}' if abs((i*100)-int(i*100)) > 0.999998 else

src/tools/other/pb2nc/pb2nc.cc

@@ -1345,8 +1345,10 @@ void process_pbfile(int i_pb) {
          }
       }
 
+      // Initialize for CAPE and PBL
       if (cal_cape || cal_mlcape) {
          cape_level = 0;
+         cape_qm = bad_data_float;
       }
 
       do_pbl = cal_pbl && 0 == strcmp("ADPUPA", hdr_typ);
@@ -1356,6 +1358,7 @@ void process_pbfile(int i_pb) {
 
       // Search through the vertical levels
       for(lv=0, n_hdr_obs = 0; lv<buf_nlev; lv++) {
+
          // If the observation vertical level is not within the
          // specified valid range, continue to the next vertical
          // level
@@ -1505,12 +1508,15 @@ void process_pbfile(int i_pb) {
                      if (cape_level < MAX_CAPE_LEVEL) cape_data_temp[cape_level] = obs_arr[4];
                      cape_member_cnt++;
                   }
-                  if (is_cape_input && (cape_level == 0)) {
+
+                  // Track the maximum quality mark for CAPE components
+                  if (is_cape_input && (is_bad_data(cape_qm) || quality_mark > cape_qm)) {
                      cape_qm = quality_mark;
                   }
                }
 
-               if (do_pbl && (pbl_level == 0)) {
+               // Track the maximum quality mark for PBL components
+               if (do_pbl && (is_bad_data(pbl_qm) || quality_mark > pbl_qm)) {
                   pbl_qm = quality_mark;
                }
             }
@@ -1908,6 +1914,7 @@ void process_pbfile(int i_pb) {
             pqtzuv_list.clear();
             pqtzuv_map_tq.clear();
             pqtzuv_map_uv.clear();
+            pbl_qm = bad_data_float;
          }
          //is_same_header = false;
          prev_hdr_vld_ut = hdr_vld_ut;