Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reducing memory leak related to accessing GRIB keys #320

Merged
merged 4 commits into from
Nov 16, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,14 @@ venv.bak/
# mypy
.mypy_cache/

# editors
.vscode

# local ignores
tests/sample-data/cds*.grib
tests/sample-data/cds*.nc
*.idx
*.idx
_dev

# mac
.DS_Store
22 changes: 15 additions & 7 deletions cfgrib/messages.py
Original file line number Diff line number Diff line change
Expand Up @@ -127,19 +127,27 @@ def message_get(self, item, key_type=None, default=_MARKER):
# type: (str, T.Optional[type], T.Any) -> T.Any
"""Get value of a given key as its native or specified type."""
try:
values = eccodes.codes_get_array(self.codes_id, item, key_type)
is_array = eccodes.codes_get_size(self.codes_id, item) > 1
if is_array:
values = eccodes.codes_get_array(self.codes_id, item, key_type)
else:
values = eccodes.codes_get(self.codes_id, item, key_type)

if values is None:
values = ["unsupported_key_type"]
return "unsupported_key_type"

if is_array and len(values) == 1:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks to me like this condition will never be met

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, it does seem to be met occasionally

if isinstance(values, np.ndarray):
values = values.tolist()
return values[0]

return values

except eccodes.KeyValueNotFoundError:
if default is _MARKER:
raise KeyError(item)
else:
return default
if len(values) == 1:
if isinstance(values, np.ndarray):
values = values.tolist()
return values[0]
return values

def message_set(self, item: str, value: T.Any) -> None:
arr = isinstance(value, (np.ndarray, T.Sequence)) and not isinstance(value, str)
Expand Down