Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

added TELEMAC support and tests #91

Merged
merged 1 commit into from
May 29, 2024
Merged

added TELEMAC support and tests #91

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Thalassa is currently supporting visualization of the output of the following so

- [Schism](https://github.com/schism-dev/schism)
- [ADCIRC](https://adcirc.org/)
- [openTELEMAC](https://www.opentelemac.org/)

Adding support for new solvers is relatively straight-forward.

Expand All @@ -30,6 +31,11 @@ Adding support for new solvers is relatively straight-forward.
```
pip install thalassa
```
for Selafin files (openTELEMAC outputs):
```
pip install thalassa
pip install xarray-selafin
```

### Conda

Expand Down
1 change: 1 addition & 0 deletions docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ Thalassa is currently supporting visualization of the output of the following so

- [Schism](https://github.com/schism-dev/schism)
- [ADCIRC](https://adcirc.org/)
- [TELEMAC](https://www.opentelemac.org)

Adding support for new solvers is relatively straight-forward.

Expand Down
5 changes: 5 additions & 0 deletions docs/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@
```
pip install thalassa
```
for Selafin files (openTELEMAC outputs):
```
pip install thalassa
pip install xarray-selafin
```

### Conda

Expand Down
2,779 changes: 1,411 additions & 1,368 deletions poetry.lock
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ ipykernel = "*"
ipython = "*"
papermill = "*"
pandas-stubs = "*"
xarray-selafin = "*"

[tool.poetry.group.docs.dependencies]
mkdocs = "*"
Expand Down
195 changes: 99 additions & 96 deletions requirements/requirements-dev.txt
View file
Open in desktop

Large diffs are not rendered by default.

112 changes: 56 additions & 56 deletions requirements/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ asciitree==0.3.3 ; python_version >= "3.9" and python_version < "4.0"
attrs==23.2.0 ; python_version >= "3.9" and python_version < "4.0"
beautifulsoup4==4.12.3 ; python_version < "3.10" and python_version >= "3.9"
bleach==6.1.0 ; python_version >= "3.9" and python_version < "4.0"
bokeh==3.3.4 ; python_version >= "3.9" and python_version < "4.0"
bottleneck==1.3.7 ; python_version >= "3.9" and python_version < "4.0"
cartopy==0.22.0 ; python_version >= "3.9" and python_version < "4.0"
bokeh==3.4.1 ; python_version >= "3.9" and python_version < "4.0"
bottleneck==1.3.8 ; python_version >= "3.9" and python_version < "4.0"
cartopy==0.23.0 ; python_version >= "3.9" and python_version < "4.0"
certifi==2024.2.2 ; python_version >= "3.9" and python_version < "4.0"
cftime==1.6.3 ; python_version >= "3.9" and python_version < "4.0"
charset-normalizer==3.3.2 ; python_version >= "3.9" and python_version < "4.0"
Expand All @@ -13,90 +13,90 @@ click==8.1.7 ; python_version >= "3.9" and python_version < "4.0"
cligj==0.7.2 ; python_version >= "3.9" and python_version < "4"
cloudpickle==3.0.0 ; python_version >= "3.9" and python_version < "4.0"
colorama==0.4.6 ; python_version >= "3.9" and python_version < "4.0" and platform_system == "Windows"
colorcet==3.0.1 ; python_version >= "3.9" and python_version < "4.0"
contourpy==1.2.0 ; python_version >= "3.9" and python_version < "4.0"
colorcet==3.1.0 ; python_version >= "3.9" and python_version < "4.0"
contourpy==1.2.1 ; python_version >= "3.9" and python_version < "4.0"
cycler==0.12.1 ; python_version >= "3.9" and python_version < "4.0"
dask==2024.1.1 ; python_version >= "3.9" and python_version < "4.0"
dask[array,complete,dataframe,diagnostics,distributed]==2024.1.1 ; python_version >= "3.9" and python_version < "4.0"
datashader==0.16.0 ; python_version >= "3.9" and python_version < "4.0"
distributed==2024.1.1 ; python_version >= "3.9" and python_version < "4.0"
dask-expr==1.1.1 ; python_version >= "3.9" and python_version < "4.0"
dask==2024.5.1 ; python_version >= "3.9" and python_version < "4.0"
dask[array,complete,dataframe,diagnostics,distributed]==2024.5.1 ; python_version >= "3.9" and python_version < "4.0"
datashader==0.16.1 ; python_version >= "3.9" and python_version < "4.0"
distributed==2024.5.1 ; python_version >= "3.9" and python_version < "4.0"
docopt==0.6.2 ; python_version < "3.10" and python_version >= "3.9"
fasteners==0.19 ; python_version >= "3.9" and python_version < "4.0"
fiona==1.9.5 ; python_version >= "3.9" and python_version < "4.0"
flox==0.9.0 ; python_version >= "3.9" and python_version < "4.0"
fonttools==4.48.1 ; python_version >= "3.9" and python_version < "4.0"
fsspec==2024.2.0 ; python_version >= "3.9" and python_version < "4.0"
future==0.18.3 ; python_version >= "3.9" and python_version < "4.0"
geopandas==0.14.3 ; python_version >= "3.9" and python_version < "4.0"
geoviews==1.11.0 ; python_version >= "3.9" and python_version < "4.0"
fasteners==0.19 ; python_version >= "3.9" and python_version < "4.0" and sys_platform != "emscripten"
fiona==1.9.6 ; python_version >= "3.9" and python_version < "4.0"
flox==0.9.7 ; python_version >= "3.9" and python_version < "4.0"
fonttools==4.51.0 ; python_version >= "3.9" and python_version < "4.0"
fsspec==2024.5.0 ; python_version >= "3.9" and python_version < "4.0"
future==1.0.0 ; python_version >= "3.9" and python_version < "4.0"
geopandas==0.14.4 ; python_version >= "3.9" and python_version < "4.0"
geoviews==1.12.0 ; python_version >= "3.9" and python_version < "4.0"
h5netcdf==1.3.0 ; python_version >= "3.9" and python_version < "4.0"
h5py==3.10.0 ; python_version >= "3.9" and python_version < "4.0"
holoviews==1.18.1 ; python_version >= "3.9" and python_version < "4.0"
idna==3.6 ; python_version >= "3.9" and python_version < "4.0"
importlib-metadata==7.0.1 ; python_version >= "3.9" and python_version < "4.0"
importlib-resources==6.1.1 ; python_version >= "3.9" and python_version < "3.10"
jinja2==3.1.3 ; python_version >= "3.9" and python_version < "4.0"
h5py==3.11.0 ; python_version >= "3.9" and python_version < "4.0"
holoviews==1.18.3 ; python_version >= "3.9" and python_version < "4.0"
idna==3.7 ; python_version >= "3.9" and python_version < "4.0"
importlib-metadata==7.1.0 ; python_version >= "3.9" and python_version < "3.12"
importlib-resources==6.4.0 ; python_version >= "3.9" and python_version < "3.10"
jinja2==3.1.4 ; python_version >= "3.9" and python_version < "4.0"
kiwisolver==1.4.5 ; python_version >= "3.9" and python_version < "4.0"
linkify-it-py==2.0.3 ; python_version >= "3.9" and python_version < "4.0"
llvmlite==0.42.0 ; python_version >= "3.9" and python_version < "4.0"
locket==1.0.0 ; python_version >= "3.9" and python_version < "4.0"
lz4==4.3.3 ; python_version >= "3.9" and python_version < "4.0"
markdown-it-py==3.0.0 ; python_version >= "3.9" and python_version < "4.0"
markdown==3.5.2 ; python_version >= "3.9" and python_version < "4.0"
markdown==3.6 ; python_version >= "3.9" and python_version < "4.0"
markupsafe==2.1.5 ; python_version >= "3.9" and python_version < "4.0"
matplotlib==3.8.2 ; python_version >= "3.9" and python_version < "4.0"
mdit-py-plugins==0.4.0 ; python_version >= "3.9" and python_version < "4.0"
matplotlib==3.9.0 ; python_version >= "3.9" and python_version < "4.0"
mdit-py-plugins==0.4.1 ; python_version >= "3.9" and python_version < "4.0"
mdurl==0.1.2 ; python_version >= "3.9" and python_version < "4.0"
msgpack==1.0.7 ; python_version >= "3.9" and python_version < "4.0"
msgpack==1.0.8 ; python_version >= "3.9" and python_version < "4.0"
multipledispatch==1.0.0 ; python_version >= "3.9" and python_version < "4.0"
netcdf4==1.6.5 ; python_version >= "3.9" and python_version < "4.0"
numba==0.59.0 ; python_version >= "3.9" and python_version < "4.0"
numbagg==0.8.0 ; python_version >= "3.9" and python_version < "4.0"
numba==0.59.1 ; python_version >= "3.9" and python_version < "4.0"
numbagg==0.8.1 ; python_version >= "3.9" and python_version < "4.0"
numcodecs==0.12.1 ; python_version >= "3.9" and python_version < "4.0"
numpy-groupies==0.10.2 ; python_version >= "3.9" and python_version < "4.0"
numpy-groupies==0.11.1 ; python_version >= "3.9" and python_version < "4.0"
numpy-indexed==0.3.7 ; python_version >= "3.9" and python_version < "4.0"
numpy==1.26.4 ; python_version >= "3.9" and python_version < "4.0"
opt-einsum==3.3.0 ; python_version >= "3.9" and python_version < "4.0"
packaging==23.2 ; python_version >= "3.9" and python_version < "4.0"
pandas==2.2.0 ; python_version >= "3.9" and python_version < "4.0"
panel==1.3.8 ; python_version >= "3.9" and python_version < "4.0"
param==2.0.2 ; python_version >= "3.9" and python_version < "4.0"
partd==1.4.1 ; python_version >= "3.9" and python_version < "4.0"
pillow==10.2.0 ; python_version >= "3.9" and python_version < "4.0"
platformdirs==4.2.0 ; python_version >= "3.9" and python_version < "4.0"
pooch==1.8.0 ; python_version >= "3.9" and python_version < "4.0"
packaging==24.0 ; python_version >= "3.9" and python_version < "4.0"
pandas==2.2.2 ; python_version >= "3.9" and python_version < "4.0"
panel==1.4.3 ; python_version >= "3.9" and python_version < "4.0"
param==2.1.0 ; python_version >= "3.9" and python_version < "4.0"
partd==1.4.2 ; python_version >= "3.9" and python_version < "4.0"
pillow==10.3.0 ; python_version >= "3.9" and python_version < "4.0"
platformdirs==4.2.2 ; python_version >= "3.9" and python_version < "4.0"
pooch==1.8.1 ; python_version >= "3.9" and python_version < "4.0"
psutil==5.9.8 ; python_version >= "3.9" and python_version < "4.0"
pyarrow-hotfix==0.6 ; python_version >= "3.9" and python_version < "4.0"
pyarrow==15.0.0 ; python_version >= "3.9" and python_version < "4.0"
pyarrow==16.1.0 ; python_version >= "3.9" and python_version < "4.0"
pyct==0.5.0 ; python_version >= "3.9" and python_version < "4.0"
pydap==3.4.1 ; python_version < "3.10" and python_version >= "3.9"
pyparsing==3.1.1 ; python_version >= "3.9" and python_version < "4.0"
pyparsing==3.1.2 ; python_version >= "3.9" and python_version < "4.0"
pyproj==3.6.1 ; python_version >= "3.9" and python_version < "4.0"
pyshp==2.3.1 ; python_version >= "3.9" and python_version < "4.0"
python-dateutil==2.8.2 ; python_version >= "3.9" and python_version < "4.0"
python-dateutil==2.9.0.post0 ; python_version >= "3.9" and python_version < "4.0"
pytz==2024.1 ; python_version >= "3.9" and python_version < "4.0"
pyviz-comms==3.0.1 ; python_version >= "3.9" and python_version < "4.0"
pyviz-comms==3.0.2 ; python_version >= "3.9" and python_version < "4.0"
pyyaml==6.0.1 ; python_version >= "3.9" and python_version < "4.0"
requests==2.31.0 ; python_version >= "3.9" and python_version < "4.0"
scipy==1.12.0 ; python_version >= "3.9" and python_version < "4.0"
setuptools==69.0.3 ; python_version >= "3.9" and python_version < "4.0"
shapely==2.0.2 ; python_version >= "3.9" and python_version < "4.0"
requests==2.32.2 ; python_version >= "3.9" and python_version < "4.0"
scipy==1.13.1 ; python_version >= "3.9" and python_version < "4.0"
shapely==2.0.4 ; python_version >= "3.9" and python_version < "4.0"
six==1.16.0 ; python_version >= "3.9" and python_version < "4.0"
sortedcontainers==2.4.0 ; python_version >= "3.9" and python_version < "4.0"
soupsieve==2.5 ; python_version < "3.10" and python_version >= "3.9"
tblib==3.0.0 ; python_version >= "3.9" and python_version < "4.0"
toolz==0.12.1 ; python_version >= "3.9" and python_version < "4.0"
tornado==6.4 ; python_version >= "3.9" and python_version < "4.0"
tqdm==4.66.1 ; python_version >= "3.9" and python_version < "4.0"
typing-extensions==4.9.0 ; python_version >= "3.9" and python_version < "4.0"
tzdata==2023.4 ; python_version >= "3.9" and python_version < "4.0"
uc-micro-py==1.0.2 ; python_version >= "3.9" and python_version < "4.0"
urllib3==2.2.0 ; python_version >= "3.9" and python_version < "4.0"
tqdm==4.66.4 ; python_version >= "3.9" and python_version < "4.0"
typing-extensions==4.12.0 ; python_version >= "3.9" and python_version < "4.0"
tzdata==2024.1 ; python_version >= "3.9" and python_version < "4.0"
uc-micro-py==1.0.3 ; python_version >= "3.9" and python_version < "4.0"
urllib3==2.2.1 ; python_version >= "3.9" and python_version < "4.0"
webencodings==0.5.1 ; python_version >= "3.9" and python_version < "4.0"
webob==1.8.7 ; python_version >= "3.9" and python_version < "3.10"
xarray==2024.1.1 ; python_version >= "3.9" and python_version < "4.0"
xarray[accel,io]==2024.1.1 ; python_version >= "3.9" and python_version < "4.0"
xyzservices==2023.10.1 ; python_version >= "3.9" and python_version < "4.0"
zarr==2.16.1 ; python_version >= "3.9" and python_version < "4.0"
xarray==2024.5.0 ; python_version >= "3.9" and python_version < "4.0"
xarray[accel,io]==2024.5.0 ; python_version >= "3.9" and python_version < "4.0"
xyzservices==2024.4.0 ; python_version >= "3.9" and python_version < "4.0"
zarr==2.18.1 ; python_version >= "3.9" and python_version < "4.0"
zict==3.0.0 ; python_version >= "3.9" and python_version < "4.0"
zipp==3.17.0 ; python_version >= "3.9" and python_version < "4.0"
zipp==3.18.2 ; python_version >= "3.9" and python_version < "3.12"
28 changes: 20 additions & 8 deletions tests/api_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,17 @@
from thalassa import normalization

ADCIRC_NC = DATA_DIR / "fort.63.nc"


def test_main_api():
variable = "zeta"
ds = api.open_dataset(ADCIRC_NC)
SELAFIN = DATA_DIR / "iceland.slf"

@pytest.mark.parametrize(
"file,variable",
[
pytest.param(ADCIRC_NC, "zeta"),
pytest.param(SELAFIN, "S"),
],
)
def test_main_api(file, variable):
ds = api.open_dataset(file)
assert normalization.is_generic(ds)

# Create objects
Expand All @@ -34,9 +40,15 @@ def test_main_api():
hv.render(pointer_ts, backend="bokeh")


@pytest.mark.parametrize("variable", [None, "zeta"])
def test_create_trimesh(variable):
ds = api.open_dataset(ADCIRC_NC)
@pytest.mark.parametrize(
"file,variable",
[
pytest.param(ADCIRC_NC, "zeta"),
pytest.param(SELAFIN, "S"),
],
)
def test_create_trimesh(file, variable):
ds = api.open_dataset(file)
trimesh = api.create_trimesh(ds, variable=variable)
assert isinstance(trimesh, gv.TriMesh)

Expand Down
Binary file added tests/data/iceland.slf
View file
Open in desktop
Binary file not shown.
2 changes: 2 additions & 0 deletions tests/normalization_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"ds,expected_fmt",
[
pytest.param(api.open_dataset(DATA_DIR / "fort.63.nc", normalize=False), THALASSA_FORMATS.ADCIRC, id="ADCIRC"),
pytest.param(api.open_dataset(DATA_DIR / "iceland.slf", normalize=False), THALASSA_FORMATS.TELEMAC, id="TELEMAC"),
pytest.param(xr.Dataset(), THALASSA_FORMATS.UNKNOWN, id="Unknown"),
],
)
Expand All @@ -24,6 +25,7 @@ def test_infer_format(ds, expected_fmt):
"path,expected",
[
pytest.param(DATA_DIR / "fort.63.nc", True, id="ADCIRC"),
pytest.param(DATA_DIR / "iceland.slf", True, id="TELEMAC"),
pytest.param(__file__, False, id="Unknown"),
],
)
Expand Down
4 changes: 2 additions & 2 deletions thalassa/api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,14 +67,14 @@ def open_dataset(
Parameters:
path: The path to the dataset file (netCDF, zarr, grib)
normalize: Boolean flag indicating whether the dataset should be converted/normalized to the "Thalassa schema".
Normalization is currently only supported for ``SCHISM`` and ``ADCIRC`` netcdf files.
Normalization is currently only supported for ``SCHISM``, ``TELEMAC``, and ``ADCIRC`` netcdf files.
kwargs: The ``kwargs`` are being passed through to ``xarray.open_dataset``.

"""
import xarray as xr

default_kwargs: dict[str, T.Any] = dict(
mask_and_scale=True,
# mask_and_scale=True,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why comment out?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good question, I forgot to mention this:
mask_and_scale is an argument that is not available in the xarray-selafin backend but that seems to exists for all other formats (zarr, grib, netcdf). I don't know how to solve this

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mask_and_scale is about applying the add_offset and the scale_factor. It's a netcdf thing, but xarray also applies it to zarr. I don't know if it makes sense in selafin, but even if it doesn't, I would add it as a do-nothing argument. Either that or I would add a **kwargs so that it can be consumed without throwing an exception

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you mean directly in xarray-selafin, correct?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the docs a **kwargs is not allowed:
https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html#open-dataset

Furthermore, mask_and_scale may not be supported by all backends: https://docs.xarray.dev/en/stable/generated/xarray.open_dataset.html

Therefore, I guess it's OK to remove it. That being said, if the selafin files can be quantized it might make sense to add support for mask_and_scale in xarray-selafin but that's a different issue.

cache=False,
drop_variables=ADCIRC_VARIABLES_TO_BE_DROPPED,
)
Expand Down
Loading
Loading