support for units with pint

[keewis]

pint's implementation of NEP-18 (see hgrecco/pint#905) is close enough so we can finally start working on the pint support (i.e. make the integration tests pass). This would be the list of tasks to get there:

• integration tests:
  • implement integration tests for DataArray, Dataset and top-level functions (tests for arrays with units #3238, tests for datasets with units #3447, Tests for module-level functions with units #3493)
  • add tests for Variable as discussed in Tests for module-level functions with units #3493 (Tests for variables with units #3654)
  • clean up the current tests (Fix pint integration tests #3600)
  • use the standard assert_identical and assert_allclose functions (Pint support for top-level functions #3611, Pint support for DataArray #3643, Tests for variables with units #3654, Pint support for variables #3706, pint support for Dataset #3975)
  • clean up the TestVariable.test_pad tests
• actually get xarray to support units:
  • top-level functions (Pint support for top-level functions #3611)
  • Variable (Pint support for variables #3706)
    • rolling_window and identical need larger modifications
  • DataArray (Pint support for DataArray #3643)
  • Dataset
  • silence all the UnitStrippedWarnings in the testsuite (silence UnitStrippedWarnings #4163)
  • try to get nanprod to work with quantities
  • add support for per variable fill values (allow specifying a fill value per variable #4165)
  • repr with units (Feature request: show units in dataset overview #2773)
  • type hierarchy (e.g. for np.maximum(data_array, quantity) vs np.maximum(quantity, data_array)) (Consistent Handling of Type Casting Hierarchy #3950)
• update the documentation
  • point to pint-xarray (see document the duck array integration status #4530)
  • mention the requirement for UnitRegistry(force_ndarray=True) or UnitRegistry(force_ndarray_like=True) (see https://pint-xarray.readthedocs.io/en/stable/creation.html#attaching-units)
  • list the known issues (see Pint support for DataArray #3643 (comment) and Pint support for DataArray #3643 (comment)) (document the duck array integration status #4530):
    • pandas (indexing)
    • bottleneck (bfill, ffill)
    • scipy (interp)
    • numbagg (rolling_exp)
    • numpy.lib.stride_tricks.as_strided: rolling
    • numpy.vectorize: interpolate_na
  • update the install instructions (we can use standard conda / pip now) this should be done by pint-xarray

[dcherian]

Thanks for leading this effort @keewis.

I would start with the lowest-level operations like the constructors and align, concat, merge. These are called in many of the other functions so fixing these is a prerequisite for getting the rest working. I've looked at align, concat & merge recently so can help if you need to chat about confusing error messages.

indexes strip units

What does this mean? You can't have units in a IndexVariable?

[keewis]

What does this mean? You can't have units in a IndexVariable?

yes, we had that discussion from #525 (comment) on. Short version: pd.Index converts using np.asarray and support for units probably requires #1603.

[amcnicho]

I will try to figure out the reason for each of these test failures, but I'd appreciate help.

Would #3643 be the best place to offer contributions at this point, or somewhere else?

[keewis]

I think issues related to DataArray + pint should be in #3643, for everything else you can use this issue or new issues / pull requests.

If you want to, I'd appreciate someone reviewing the tests in test_units.py since I don't think anyone other than me thoroughly looked at all of them. You could also investigate / fix the Dataset issues, investigate the reason for the UnitStrippedWarnings or start writing documentation on how to use pint in combination with xarray.

[keewis]

so, except from the major issues mentioned above which we won't be able to fix in the near future (but there will probably be workarounds in pint-xarray) we only have three minor issues: nanprod, support for per variable fill values and the repr (#2773).

I don't think nanprod and the fill values are particularly urgent, so if we get support for the repr (maybe using some sort of hook that a library like pint-xarray can then use to properly format the duckarray) and put together the documentation page, we could include this in the 0.16 release.

Edit: I guess the release is already big enough so I don't really mind waiting on the next release, but this is really close.

[dcherian]

@keewis Shall we close this? It seems the only outstanding one is nanprod with quantities. Which sort of indicates that we've made the necessary big changes.

[keewis]

#6873 might fix the nanprod issue, and we have a separate issue for the last big change left (#3950, which is not really limited to quantities) so I agree that we should be able to close this with #6873.

We might want to open a new issue to get the known issues to work, though.