Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Respect user-specified coordinates attribute. #3487

Merged
merged 23 commits into from
Dec 10, 2019
Merged

Respect user-specified coordinates attribute. #3487

merged 23 commits into from
Dec 10, 2019

Conversation

dcherian
Copy link
Contributor

@dcherian dcherian commented Nov 5, 2019 •
edited
Loading

A minimally invasive solution to #3351. If variable.encoding["coordinates"] is specified, we write that attribute to disk and warn the user that roundtripping may not work.

@dcherian
Copy link
Contributor Author

dcherian commented Nov 5, 2019

@jhamman your input would be appreciated here if you have the time.

jhamman
jhamman reviewed Nov 5, 2019
View reviewed changes
xarray/tests/test_backends.py Outdated Show resolved Hide resolved
doc/io.rst Outdated Show resolved Hide resolved
@dcherian
Copy link
Contributor Author

dcherian commented Nov 6, 2019

updated tests and fixed whats-new

@dcherian dcherian changed the title Respect user-specified coordinates attribute. [WIP] Respect user-specified coordinates attribute. Nov 8, 2019
@dcherian
Merge remote-tracking branch 'upstream/master' into fix/user-coordinates
d49ceef 
* upstream/master:
  add missing pint integration tests (pydata#3508)
  DOC: update bottleneck repo url (pydata#3507)
  add drop_sel, drop_vars, map to api.rst (pydata#3506)
  remove syntax warning (pydata#3505)
  Dataset.map, GroupBy.map, Resample.map (pydata#3459)
  tests for datasets with units (pydata#3447)
  fix pandas-dev tests (pydata#3491)
  unpin pseudonetcdf (pydata#3496)
  whatsnew corrections (pydata#3494)
  drop_vars; deprecate drop for variables (pydata#3475)
  uamiv test using only raw uamiv variables (pydata#3485)
  Optimize dask array equality checks. (pydata#3453)
@dcherian
Merge branch 'fix/user-coordinates' of github.com:dcherian/xarray int…
bb62590 
…o fix/user-coordinates

* 'fix/user-coordinates' of github.com:dcherian/xarray:
@dcherian
fix whats-new
0abe242
@max-sixty
Copy link
Collaborator

Shall we merge?

@dcherian
[minor] add comments
1a42c6c
@dcherian
Merge remote-tracking branch 'upstream/master' into fix/user-coordinates
e1a3823 
* upstream/master:
  Allow appending datetime & boolean variables to zarr stores (pydata#3504)
  warn if dim is passed to rolling operations. (pydata#3513)
  Deprecate allow_lazy (pydata#3435)
  Recursive tokenization (pydata#3515)
  format indexing.rst code with black (pydata#3511)
@dcherian
fix whats-new
1c97c1e
@dcherian
Actually test global "coordinates" handling.
5773e5e
@dcherian dcherian changed the title [WIP] Respect user-specified coordinates attribute. Respect user-specified coordinates attribute. Nov 14, 2019
@dcherian
Copy link
Contributor Author

The warnings I added earlier were very annoying so I reworked it.

Now we keep track of all variable names written to a "coordinates" attribute. Any names that are not written are added to a global "coordinates" attribute. I've also fixed the global attribute test to actually test that behaviour :)

dcherian and others added 5 commits November 14, 2019 08:45
@dcherian
filerwarning not necessary.
670f97f
@dcherian
Merge branch 'master' into fix/user-coordinates
e1c48c9
@dcherian
Merge remote-tracking branch 'upstream/master' into fix/user-coordinates
77c7c0f 
* upstream/master: (35 commits)
  fix plotting with transposed nondim coords. (pydata#3441)
  make coarsen reductions consistent with reductions on other classes (pydata#3500)
  Resolve the version issues on RTD (pydata#3589)
  Add bottleneck & rasterio git tip to upstream-dev CI (pydata#3585)
  update whats-new.rst (pydata#3581)
  Examples for quantile (pydata#3576)
  add cftime intersphinx entries (pydata#3577)
  Add pyXpcm to Related Projects doc page (pydata#3578)
  Reimplement quantile with apply_ufunc (pydata#3559)
  add environment file for binderized examples (pydata#3568)
  Add drop to api.rst under pending deprecations (pydata#3561)
  replace duplicate method _from_vars_and_coord_names (pydata#3565)
  propagate indexes in to_dataset, from_dataset (pydata#3519)
  Switch examples to notebooks + scipy19 docs improvements (pydata#3557)
  fix whats-new.rst (pydata#3554)
  Tweaks to release instructions (pydata#3555)
  Clarify conda environments for new contributors (pydata#3551)
  Revert to dev version
  0.14.1 whatsnew (pydata#3547)
  sparse option to reindex and unstack (pydata#3542)
  ...
@dcherian
Add comment
3ca669b
@dcherian
Merge branch 'fix/user-coordinates' of github.com:dcherian/xarray int…
d5d3734 
…o fix/user-coordinates

* 'fix/user-coordinates' of github.com:dcherian/xarray:
@dcherian
Copy link
Contributor Author

dcherian commented Dec 10, 2019 •
edited
Loading

I'm going to merge this. I'm happy to address any comments in future PRs.

This is a step forward because:

  1. This change now lets xarray roundtrip datasets that have "coordinates" attributes specified.
  2. It also lets users specify a coordinates attribute and writes it to disk (this raised an error earlier).
  3. Xarray should now set the not-CF-compliant "global" coordinates attribute in fewer cases than earlier.

@dcherian dcherian merged commit 2ee89c3 into pydata:master Dec 10, 2019
@dcherian dcherian deleted the fix/user-coordinates branch December 10, 2019 16:02
dcherian added a commit to dcherian/xarray that referenced this pull request Dec 30, 2019
@dcherian
Merge remote-tracking branch 'upstream/master' into dask-median
42a7ed3 
* upstream/master:
  added pyinterp to related projects (pydata#3655)
  Allow incomplete hypercubes in combine_by_coords (pydata#3649)
  concat keeps attrs from first variable. (pydata#3637)
  Extend DatetimeAccessor properties and support `.dt` accessor for Timedelta (pydata#3612)
  update readthedocs.yml (pydata#3639)
  silence sphinx warnings round 3 (pydata#3602)
  Fix/quantile wrong errmsg (pydata#3635)
  Provide shape info in shape mismatch error. (pydata#3619)
  Minor doc fixes (pydata#3615)
  Respect user-specified coordinates attribute. (pydata#3487)
  Add Facetgrid.row_labels & Facetgrid.col_labels (pydata#3597)
  Fix pint integration tests (pydata#3600)
  Minor fix to combine_by_coords to allow for the combination of CFTimeIndexes separated by large time intervals (pydata#3543)
dcherian added a commit to dcherian/xarray that referenced this pull request Dec 30, 2019
@dcherian
Merge branch 'master' of github.com:pydata/xarray into chunk-unique-t…
d40a88d 
…oken

* 'master' of github.com:pydata/xarray:
  Add nanmedian for dask arrays (pydata#3604)
  added pyinterp to related projects (pydata#3655)
  Allow incomplete hypercubes in combine_by_coords (pydata#3649)
  concat keeps attrs from first variable. (pydata#3637)
  Extend DatetimeAccessor properties and support `.dt` accessor for Timedelta (pydata#3612)
  update readthedocs.yml (pydata#3639)
  silence sphinx warnings round 3 (pydata#3602)
  Fix/quantile wrong errmsg (pydata#3635)
  Provide shape info in shape mismatch error. (pydata#3619)
  Minor doc fixes (pydata#3615)
  Respect user-specified coordinates attribute. (pydata#3487)
  Add Facetgrid.row_labels & Facetgrid.col_labels (pydata#3597)
  Fix pint integration tests (pydata#3600)
  Minor fix to combine_by_coords to allow for the combination of CFTimeIndexes separated by large time intervals (pydata#3543)
dcherian added a commit to dcherian/xarray that referenced this pull request Dec 30, 2019
@dcherian
Merge branch 'master' of github.com:pydata/xarray into optimize-lazy-…
9291b91 
…equiv

* 'master' of github.com:pydata/xarray: (28 commits)
  Add nanmedian for dask arrays (pydata#3604)
  added pyinterp to related projects (pydata#3655)
  Allow incomplete hypercubes in combine_by_coords (pydata#3649)
  concat keeps attrs from first variable. (pydata#3637)
  Extend DatetimeAccessor properties and support `.dt` accessor for Timedelta (pydata#3612)
  update readthedocs.yml (pydata#3639)
  silence sphinx warnings round 3 (pydata#3602)
  Fix/quantile wrong errmsg (pydata#3635)
  Provide shape info in shape mismatch error. (pydata#3619)
  Minor doc fixes (pydata#3615)
  Respect user-specified coordinates attribute. (pydata#3487)
  Add Facetgrid.row_labels & Facetgrid.col_labels (pydata#3597)
  Fix pint integration tests (pydata#3600)
  Minor fix to combine_by_coords to allow for the combination of CFTimeIndexes separated by large time intervals (pydata#3543)
  Fix map_blocks HLG layering (pydata#3598)
  Silence sphinx warnings: Round 2 (pydata#3592)
  2x~5x speed up for isel() in most cases (pydata#3533)
  remove xarray again (pydata#3591)
  fix plotting with transposed nondim coords. (pydata#3441)
  make coarsen reductions consistent with reductions on other classes (pydata#3500)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jhamman jhamman jhamman approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

custom "coordinates" attribute
3 participants
@dcherian @max-sixty @jhamman