gen_vx_mask to generate template mask

[shaunlcooper]

Hello MET team!

I am very new to MET at this stage. There are a number of us that the Bureau of Meteorology that are undertaking training in MET/METplus and I have begun trying to use the tools on some of our data.

I have been able to use plot_data_plane and pcp_combine to generate 3 hourly precipitation accumulations from some of our ensemble model outputs and radar observations. However, our radar network does not cover the entire model domain - with the unobserved areas masked out. I plan to use grid_stat to continue my exploration of MET. However, due to the masked area in the observations, I thought I might need to generate a mask file that would then be passed into other tools to mask out the unobserved regions. Is this assumption correct?

I attempted to produce such a mask file using the following:

gen_vx_mask -type data \
-input_data 'name="accum_prcp"; level="(,)";'
-mask_field 'name="precipitation"; level="(,)";'
forecast.nc obs.nc mask.nc -v 4

This produces a mask.nc that is the same as the obs.nc with the missing data value altered to MET's -9999, with the observation values retained. This is not what I was expecting. I was expecting either a binary mask, or the forecast values with the mask applied (NOTE: first time using the tool!).

I do receive the following warning:

WARNING: apply_data_mask() -> since "-thresh" was not used to specify a threshold for data masking, the raw data values will be written.

I tried to add: '-thresh 0' to the above call assuming it might help generate a binary mask and get the following error:

ERROR : Dictionary::lookup_thresh() -> lookup failed for name "threshold"

My questions are:

1. What data should I be expecting the mask.nc file to contain?
2. Why is the '-thresh' option not working?
3. Is the output mask file passed to other tools or does one need apply the mask to the forecast data prior to passing them to grid_stat for example?
4. In this instance, do I actually need to generate a mask or will the obs.nc masks work as it is? (noting the different missing data value)

I have attached example data and logs.

mask.tar.gz

Thanks in advance,

Shaun

[JohnHalleyGotway]

Hi @shaunlcooper, we're glad to hear that BOM is getting up and running with METplus, and we're happy to help! Thanks for sending along sample data to illustrate your questions.

Whenever getting going with new data in MET, I always start by running the plot_data_plane tool to visualize. Here's what I see in the forecast.nc and obs.nc files you sent:

> plot_data_plane forecast.nc forecast.ps 'name="accum_prcp"; level="(*,*)";' -title "Forecast accum_prcp"
> plot_data_plane obs.nc obs.ps 'name="precipitation"; level="(*,*)";' -title "Observation precipitation"

And here's the result:

I understand that you only want to compute stats at grid points where the forecast and observations both contain valid data.

1. First, you do not actually need to pre-process the data any more than this. MET reads the _FillValue and missing_value variable attributes from obs.nc and resets any instances of that value to it's internal bad data value of -9999.

	double precipitation(lat, lon) ;
		precipitation:_FillValue = -32768. ;
		precipitation:missing_value = -32768. ;

That's why the minimum value in the plot of the obs above is 0, and not -32768. MET is correctly interpreting that as missing data. Please run Grid-Stat to compare forecast.nc to obs.nc and it'll only use grid points where both the forecast and observations contain valid data. Basically, it'll ignore the obs bad data points just like you want it to.

2. Second, plot_data_plane does correctly extract the timing info from the forecast data. You can see this by rerunning using the "-v 4" command line option to increase the logging verbosity level:

DEBUG 4: Data plane information:
DEBUG 4:    plane min: 0
DEBUG 4:    plane max: 153.191
DEBUG 4:    valid time: 20220227_210000
DEBUG 4:    lead time: 000000
DEBUG 4:    init time: 20220227_210000
DEBUG 4:    accum time: NA

However, it does NOT correctly extract the timing info from the obs file. Instead, it just uses the default valid time of unixtime 0 (which is 19700101):

DEBUG 4: Data plane information:
DEBUG 4:    plane min: 0
DEBUG 4:    plane max: 2.10354
DEBUG 4:    valid time: 19700101_000000
DEBUG 4:    lead time: 000000
DEBUG 4:    init time: 19700101_000000
DEBUG 4:    accum time: 000000

I see that the file specifies the start_time and valid_time, however, I believe that according to CF-1.7 conventions, "valid_time" should actually just be named "time". When I rename valid_time to time, plot_data_plane does parse it, as expected:

DEBUG 4: Data plane information:
DEBUG 4:    plane min: 0
DEBUG 4:    plane max: 2.10354
DEBUG 4:    valid time: 20220221_000000
DEBUG 4:    lead time: 000000
DEBUG 4:    init time: 20220221_000000
DEBUG 4:    accum time: 000000
DEBUG 4: 

It isn't clear to me whether setting standard_name = "time"; complies with the CF convention. If you believe so, I could ask a team member to take a closer look.

3. Next, based on the "log.txt" file you sent, I think you ran this Gen-Vx-Mask command.

gen_vx_mask -type data forecast.nc obs.nc mask.nc -mask_field 'name="precipitation"; level="(*,*)";'

And the output file just contains the same data as obs.nc. So Gen-Vx-Mask supports several types of mask definition. The "-type data" option tells it to read a field of data and apply some threshold, specified using the "-thresh" command line option. If no threshold is specified, it just write the un-thresholded input values to the output (in some circumstances, that actually is useful).

For example, you could run:

gen_vx_mask -type data forecast.nc obs.nc mask.nc -mask_field 'name="precipitation"; level="(*,*)";' -thresh gt0

And the resulting mask of 0's and 1's are all the points where the obs precip is strictly greater than 0.

I'd recommend reading through the Gen-Vx-Mask chapter of the user's guide and stepping through the online tutorial section about it. In particular, if you're interested in points close to radars, you may find the circle masking type to be of use.

Please don't hesitate to ask if more questions or issues arise in your use of MET. We've also recently been recording a series of 1-hour METplus Training videos. Just click on each session name to see the embedded recording. We covered Gen-Vx-Mask in the December 14, 2021 session.

[shaunlcooper]

Hi John,

Thanks for you swift response!

One last quick question. If I wanted to apply the mask that already exists in the observations file, to the forecast file, is there a manner of doing this with MET?

Thanks again,

Shaun

[JohnHalleyGotway]

I'm interpreting your question as, can I apply the bad data mask from obs.nc to the values in forecast.nc. Assuming that's the right question, yes, you can using Gen-Vx-Mask, like this:

gen_vx_mask -type data \
forecast.nc obs.nc forecast_masked_by_obs.nc \
-input_field 'name="accum_prcp"; level="(*,*)";' \
-mask_field 'name="precipitation"; level="(*,*)";' \
-thresh eq-9999 -value -9999

Then plot the result.

plot_data_plane forecast_masked_by_obs.nc forecast_masked_by_obs.ps 'name="data_mask"; level="(*,*)";'

Here's some details...

• If "-input_field" is specified for "-type data", then the value at each grid point is initialized to the value from that field of data instead of 0, which is the default.
• The "-mask_field" is read from the "mask_file" (2nd req'd argument) and is defines the data for data masking.
• The "-thresh eq-9999" says we're interested in grid points that contain bad data since -9999 is MET's bad data value.
• By default, the mask value written to the output is 1. Setting "-value -9999" writes the bad data value instead.

So yes, this is possible, but this does not represent a typical usage of the Gen-Vx-Mask tool. It's usually used to define fields of 0's and 1's to define spatial masking areas. The NetCDF mask files are then referenced in the Point-Stat and Grid-Stat config files, for example.

[shaunlcooper]

Thanks so much John.

Feel free to close this off!