Missing variables when getting the "DatasetBuildError: key present and new value is different" warning

[enyfeo]

When attempting to read grib file from EFAS which contained 3 variables, cfgrib fails with the error below.

The grib contains 6 hourly time steps 0/to/240/by/6 for var sd which is read ok as the first variable of 41 steps. The remaining params tp06 and dis06 are values for total precipitation and discharge over the last 6 hours and only have 40 time steps, timesteps are written a range 0-6 6-12 ... 234-240.

Reading the variables from separate gribs works ok. It seems due to the number of steps between the 2 variables not being equal.
I cannot upload the grib but drop me an email and I will send it to you.

>>> cfgrib.__version__
'0.9.6.post1'
>>> eccodes.__version__
'2.12.0'
>>> ds = xr.open_dataset('test/all_20190201.grb',engine='cfgrib')
skipping variable: paramId==240023 shortName='dis06'
Traceback (most recent call last):
  File "/home/ma/maca/.local/lib/python3.6/site-packages/cfgrib/data

set.py", line 467, in build_dataset_components
    dict_merge(dimensions, dims)
  File "/home/ma/maca/.local/lib/python3.6/site-packages/cfgrib/dataset.py", line 430, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='step' value=41 new_value=40
skipping variable: paramId==260267 shortName='tp06'
Traceback (most recent call last):
  File "/home/ma/maca/.local/lib/python3.6/site-packages/cfgrib/dataset.py", line 467, in build_dataset_components
    dict_merge(dimensions, dims)
  File "/home/ma/maca/.local/lib/python3.6/site-packages/cfgrib/dataset.py", line 430, in dict_merge
    "key=%r value=%r new_value=%r" % (key, master[key], value))
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='step' value=41 new_value=40
>>> 

[alexamici]

@enyfeo after testing the GRIB sample you sent me via email I can confirm that this is a known, but badly documented issue.

Unfortunately it is not easy to address the root problem within the xarray interfaces and we are left with using the filter_by_keys work-around, in this case you can access variables one by one specifying the GRIB short name:

>>> xr.open_dataset('EFAS.grib', engine='cfgrib', backend_kwargs={'filter_by_keys': {'shortName': 'dis06'}})
<xarray.Dataset>
Dimensions:     (step: 3, values: 950000)
Coordinates:
    time        datetime64[ns] ...
  * step        (step) timedelta64[ns] 06:00:00 12:00:00 18:00:00
    surface     int64 ...
    latitude    (values) float64 ...
    longitude   (values) float64 ...
    valid_time  (step) datetime64[ns] ...
Dimensions without coordinates: values
Data variables:
    dis06       (step, values) float32 ...
Attributes:
    GRIB_edition:            2
    GRIB_centre:             ecmf
    GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
    GRIB_subCentre:          0
    Conventions:             CF-1.7
    institution:             European Centre for Medium-Range Weather Forecasts
    history:                 2019-03-21T11:24:08 GRIB to CDM+CF via cfgrib-0....

Still it would be nicer if the error message suggested the possible work-around.

Note that I changed the title to make it more searchable.

[enyfeo]

@alexamici
A workaround for me was to use grib_copy and split the file by its parameters and process each file individually.

[alexamici]

@enyfeo using the new heuristic for cfgrib.open_datasets introduced with version 0.9.7 you should get all, or almost all, variable correctly.

Please open a new issue if you still find that some variable is not usable.

>>> import cfgrib
>>> cfgrib.open_datasets('EFAS.grib')
[<xarray.Dataset>
 Dimensions:     (step: 4, values: 950000)
 Coordinates:
     time        datetime64[ns] 2019-02-01
   * step        (step) timedelta64[ns] 00:00:00 06:00:00 12:00:00 18:00:00
     surface     int64 0
     latitude    (values) float64 ...
     longitude   (values) float64 ...
     valid_time  (step) datetime64[ns] ...
 Dimensions without coordinates: values
 Data variables:
     sd          (step, values) float32 ...
 Attributes:
     GRIB_edition:            2
     GRIB_centre:             ecmf
     GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
     GRIB_subCentre:          0
     Conventions:             CF-1.7
     institution:             European Centre for Medium-Range Weather Forecasts,
 <xarray.Dataset>
 Dimensions:     (step: 3, values: 950000)
 Coordinates:
     time        datetime64[ns] 2019-02-01
   * step        (step) timedelta64[ns] 06:00:00 12:00:00 18:00:00
     surface     int64 0
     latitude    (values) float64 ...
     longitude   (values) float64 ...
     valid_time  (step) datetime64[ns] ...
 Dimensions without coordinates: values
 Data variables:
     dis06       (step, values) float32 ...
 Attributes:
     GRIB_edition:            2
     GRIB_centre:             ecmf
     GRIB_centreDescription:  European Centre for Medium-Range Weather Forecasts
     GRIB_subCentre:          0
     Conventions:             CF-1.7
     institution:             European Centre for Medium-Range Weather Forecasts]