Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Raise all non-unique values in the file not just the one for the variable. #54 #57

Merged
merged 2 commits into from
Feb 24, 2019
Merged

Raise all non-unique values in the file not just the one for the variable. #54 #57

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bf3c56a
Raise all non-unique values in the file not just the one for the vari…
alexamici Feb 24, 2019
2681af6
Raise all non-unique values in the file not just the one for the vari…
alexamici Feb 24, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ Changelog for cfgrib
0.9.6 (unreleased)
------------------

- Fixed a serious bug in the computation of the suggested ``filter_by_keys`` for non-cubic
GRIB files. As a result ``cfgrib.xarray_store.open_datasets`` was not finding all the
variables in the files.
See: `#54 <https://github.com/ecmwf/cfgrib/issues/54>`_.
- The *conda-forge* package is in progress.
See: `#5 <https://github.com/ecmwf/cfgrib/issues/5>`_.
- Add support for *Windows* by installing *ecCodes* via *conda*.
Expand Down
108 changes: 108 additions & 0 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -351,6 +351,49 @@ and it is not part of the stable API. In the future it may change or be removed
>>> from cfgrib import xarray_store
>>> xarray_store.open_datasets('nam.t00z.awp21100.tm00.grib2', backend_kwargs={'errors': 'ignore'})
[<xarray.Dataset>
Dimensions: (x: 93, y: 65)
Coordinates:
time datetime64[ns] ...
step timedelta64[ns] ...
meanSea int64 ...
latitude (y, x) float64 ...
longitude (y, x) float64 ...
valid_time datetime64[ns] ...
Dimensions without coordinates: x, y
Data variables:
prmsl (y, x) float32 ...
mslet (y, x) float32 ...
Attributes:
GRIB_edition: 2
GRIB_centre: kwbc
GRIB_centreDescription: US National Weather Service - NCEP...
GRIB_subCentre: 0
Conventions: CF-1.7
institution: US National Weather Service - NCEP...
history: ..., <xarray.Dataset>
Dimensions: (x: 93, y: 65)
Coordinates:
time datetime64[ns] ...
step timedelta64[ns] ...
surface int64 ...
latitude (y, x) float64 ...
longitude (y, x) float64 ...
valid_time datetime64[ns] ...
Dimensions without coordinates: x, y
Data variables:
gust (y, x) float32 ...
sp (y, x) float32 ...
orog (y, x) float32 ...
tp (y, x) float32 ...
acpcp (y, x) float32 ...
Attributes:
GRIB_edition: 2
GRIB_centre: kwbc
GRIB_centreDescription: US National Weather Service - NCEP...
GRIB_subCentre: 0
Conventions: CF-1.7
institution: US National Weather Service - NCEP...
history: ..., <xarray.Dataset>
Dimensions: (isobaricInhPa: 19, x: 93, y: 65)
Coordinates:
time datetime64[ns] ...
Expand All @@ -375,6 +418,47 @@ Attributes:
Conventions: CF-1.7
institution: US National Weather Service - NCEP...
history: ..., <xarray.Dataset>
Dimensions: (x: 93, y: 65)
Coordinates:
time datetime64[ns] ...
step timedelta64[ns] ...
heightAboveGround int64 ...
latitude (y, x) float64 ...
longitude (y, x) float64 ...
valid_time datetime64[ns] ...
Dimensions without coordinates: x, y
Data variables:
t2m (y, x) float32 ...
r2 (y, x) float32 ...
u10 (y, x) float32 ...
v10 (y, x) float32 ...
Attributes:
GRIB_edition: 2
GRIB_centre: kwbc
GRIB_centreDescription: US National Weather Service - NCEP...
GRIB_subCentre: 0
Conventions: CF-1.7
institution: US National Weather Service - NCEP...
history: ..., <xarray.Dataset>
Dimensions: (x: 93, y: 65)
Coordinates:
time datetime64[ns] ...
step timedelta64[ns] ...
level int64 ...
latitude (y, x) float64 ...
longitude (y, x) float64 ...
valid_time datetime64[ns] ...
Dimensions without coordinates: x, y
Data variables:
pwat (y, x) float32 ...
Attributes:
GRIB_edition: 2
GRIB_centre: kwbc
GRIB_centreDescription: US National Weather Service - NCEP...
GRIB_subCentre: 0
Conventions: CF-1.7
institution: US National Weather Service - NCEP...
history: ..., <xarray.Dataset>
Dimensions: (x: 93, y: 65)
Coordinates:
time datetime64[ns] ...
Expand Down Expand Up @@ -450,6 +534,30 @@ Dimensions without coordinates: x, y
Data variables:
gh (y, x) float32 ...
r (y, x) float32 ...
Attributes:
GRIB_edition: 2
GRIB_centre: kwbc
GRIB_centreDescription: US National Weather Service - NCEP...
GRIB_subCentre: 0
Conventions: CF-1.7
institution: US National Weather Service - NCEP...
history: ..., <xarray.Dataset>
Dimensions: (pressureFromGroundLayer: 5, x: 93, y: 65)
Coordinates:
time datetime64[ns] ...
step timedelta64[ns] ...
* pressureFromGroundLayer (pressureFromGroundLayer) int64 3000 6000 ... 15000
latitude (y, x) float64 ...
longitude (y, x) float64 ...
valid_time datetime64[ns] ...
Dimensions without coordinates: x, y
Data variables:
t (pressureFromGroundLayer, y, x) float32 ...
r (pressureFromGroundLayer, y, x) float32 ...
u (pressureFromGroundLayer, y, x) float32 ...
v (pressureFromGroundLayer, y, x) float32 ...
pli (y, x) float32 ...
4lftx (y, x) float32 ...
Attributes:
GRIB_edition: 2
GRIB_centre: kwbc
Expand Down
23 changes: 17 additions & 6 deletions cfgrib/dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,14 +178,12 @@ def enforce_unique_attributes(index, attributes_keys, filter_by_keys={}):
for key in attributes_keys:
values = index[key]
if len(values) > 1:
error_message = "multiple values for unique key, try re-open the file with one of:"
fbks = []
for value in values:
fbk = {key: value}
fbk.update(filter_by_keys)
fbks.append(fbk)
error_message += "\n filter_by_keys=%r" % fbk
raise DatasetBuildError(error_message, fbks)
raise DatasetBuildError("multiple values for key %r" % key, key, fbks)
if values and values[0] not in ('undef', 'unknown'):
attributes['GRIB_' + key] = values[0]
return attributes
Expand Down Expand Up @@ -443,9 +441,22 @@ def build_dataset_components(
variables = collections.OrderedDict()
for param_id, short_name, var_name in zip(param_ids, index['shortName'], index['cfVarName']):
var_index = index.subindex(paramId=param_id)
dims, data_var, coord_vars = build_variable_components(
var_index, encode_cf, filter_by_keys, errors=errors,
)
try:
dims, data_var, coord_vars = build_variable_components(
var_index, encode_cf, filter_by_keys, errors=errors,
)
except DatasetBuildError as ex:
# NOTE: When a variable has more than one value for an attribute we need to raise all
# the values in the file, not just the ones associated with that variable. See #54.
key = ex.args[1]
error_message = "multiple values for unique key, try re-open the file with one of:"
fbks = []
for value in index[key]:
fbk = {key: value}
fbk.update(filter_by_keys)
fbks.append(fbk)
error_message += "\n filter_by_keys=%r" % fbk
raise DatasetBuildError(error_message, key, fbks)
if 'parameter' in encode_cf and var_name not in ('undef', 'unknown'):
short_name = var_name
vars = collections.OrderedDict([(short_name, data_var)])
Expand Down
2 changes: 1 addition & 1 deletion cfgrib/xarray_store.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ def open_datasets(path, backend_kwargs={}, no_warn=False, **kwargs):
try:
datasets.append(open_dataset(path, backend_kwargs=backend_kwargs, **kwargs))
except DatasetBuildError as ex:
fbks.extend(ex.args[1])
fbks.extend(ex.args[2])
# NOTE: the recursive call needs to stay out of the exception handler to avoid showing
# to the user a confusing error message due to exception chaining
for fbk in fbks:
Expand Down