Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

__dask_tokenize__ #3446

Merged
merged 6 commits into from
Oct 31, 2019
Merged

__dask_tokenize__ #3446

merged 6 commits into from
Oct 31, 2019

Conversation

dcherian
Copy link
Contributor

@dcherian dcherian commented Oct 25, 2019 •
edited
Loading

  • Closes implement normalize_token #3378
  • Tests added
  • Passes black . && mypy . && flake8
  • Fully documented, including whats-new.rst for all changes and api.rst for new API

@crusaderky here's your testing checklist. I wasn't entirely sure what you meant by the last two but I took a stab at it. feel free to push to this branch so that we can finish quickly

  • running tokenize() twice on the same object returns the same result
  • changing the content of a data_var (or the variable, for DataArray) changes the output
  • changing the content of a coord changes the output
  • changing attrs, name, or dimension names change the output
  • whether a variable is a data_var or a coord changes the output
  • dask arrays aren't computed
  • non-numpy, non-dask NEP18 data is not converted to numpy
  • works with xarray's fancy wrappers around NetCDF and the like

@dcherian
Implement __dask_tokenize__
4ab6a66
@dcherian
Fix window test
507b1f6
@dcherian
Merge remote-tracking branch 'upstream/master' into dask-tokenize
bc39877 
* upstream/master:
  upgrade black verison to 19.10b0 (pydata#3456)
  Remove outdated code related to compatibility with netcdftime (pydata#3450)
  Remove deprecated behavior from dataset.drop docstring (pydata#3451)
  jupyterlab dark theme (pydata#3443)
  Drop groups associated with nans in group variable (pydata#3406)
  Allow ellipsis (...) in transpose (pydata#3421)
  Another groupby.reduce bugfix. (pydata#3403)
  add icomoon license (pydata#3448)
  change ALL_DIMS to equal ellipsis (pydata#3418)
  Escaping dtypes (pydata#3444)
  Html repr (pydata#3425)
@crusaderky crusaderky mentioned this pull request Oct 29, 2019
Closed
12 tasks
@crusaderky
Copy link
Contributor

@dcherian Please have a look through my changes; after that feel free to merge

dcherian
dcherian commented Oct 31, 2019
View reviewed changes
Copy link
Contributor Author

@dcherian dcherian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left one minor comment on tests.

Should we add support for DataArrayCoordinates and DatasetCoordinates as

    def __dask_tokenize__(self):
        return (type(self), self.variables, self._names, self.dims)

xarray/tests/test_dask.py Show resolved Hide resolved
@crusaderky
Copy link
Contributor

Should we add support for DataArrayCoordinates and DatasetCoordinates

Why would anybody push them to dask?

@crusaderky crusaderky merged commit 53c5199 into pydata:master Oct 31, 2019
@dcherian dcherian deleted the dask-tokenize branch October 31, 2019 16:53
dcherian added a commit to dcherian/xarray that referenced this pull request Nov 4, 2019
@dcherian
Merge remote-tracking branch 'upstream/master' into interp-na-maxgap
ac7aeeb 
* upstream/master:
  __dask_tokenize__ (pydata#3446)
  Type check sentinel values (pydata#3472)
  Fix typo in docstring (pydata#3474)
  fix test suite warnings re `drop` (pydata#3460)
  Fix integrate docs (pydata#3469)
  Fix leap year condition in monthly means example (pydata#3464)
  Hypothesis tests for roundtrip to & from pandas (pydata#3285)
  unpin cftime (pydata#3463)
  Cleanup whatsnew (pydata#3462)
  enable xr.ALL_DIMS in xr.dot (pydata#3424)
  Merge stable into master (pydata#3457)
  upgrade black verison to 19.10b0 (pydata#3456)
  Remove outdated code related to compatibility with netcdftime (pydata#3450)
  Remove deprecated behavior from dataset.drop docstring (pydata#3451)
  jupyterlab dark theme (pydata#3443)
  Drop groups associated with nans in group variable (pydata#3406)
  Allow ellipsis (...) in transpose (pydata#3421)
  Another groupby.reduce bugfix. (pydata#3403)
  add icomoon license (pydata#3448)
dcherian added a commit to dcherian/xarray that referenced this pull request Nov 8, 2019
@dcherian
Merge remote-tracking branch 'upstream/master' into fix/allow_lazy
6a6ba1b 
* upstream/master: (27 commits)
  drop_vars; deprecate drop for variables (pydata#3475)
  uamiv test using only raw uamiv variables (pydata#3485)
  Optimize dask array equality checks. (pydata#3453)
  Propagate indexes in DataArray binary operations. (pydata#3481)
  python 3.8 tests (pydata#3477)
  __dask_tokenize__ (pydata#3446)
  Type check sentinel values (pydata#3472)
  Fix typo in docstring (pydata#3474)
  fix test suite warnings re `drop` (pydata#3460)
  Fix integrate docs (pydata#3469)
  Fix leap year condition in monthly means example (pydata#3464)
  Hypothesis tests for roundtrip to & from pandas (pydata#3285)
  unpin cftime (pydata#3463)
  Cleanup whatsnew (pydata#3462)
  enable xr.ALL_DIMS in xr.dot (pydata#3424)
  Merge stable into master (pydata#3457)
  upgrade black verison to 19.10b0 (pydata#3456)
  Remove outdated code related to compatibility with netcdftime (pydata#3450)
  Remove deprecated behavior from dataset.drop docstring (pydata#3451)
  jupyterlab dark theme (pydata#3443)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@crusaderky crusaderky crusaderky approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

implement normalize_token
2 participants
@dcherian @crusaderky