xarray improperly decodes times from a NetCDF when it is a uint

[sappjw]

What happened?

xarray improperly decodes times from a NetCDF when it is a uint. The attached CDL file generates a NetCDF file with the right time ('good_time') and the wrong time ('time') (use ncgen -o both_times.nc -k nc4 both_times.txt)

What did you expect to happen?

time to be properly decoded (see good_time).

Minimal Complete Verifiable Example

import xarray as xr

xr.open_dataset('both_times.nc').good_time
xr.open_dataset('both_times.nc').time

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

In [1]: xr.open_dataset('both_times.nc').good_time
<xarray.DataArray 'good_time' (trajectory: 284)>
array(['2018-08-22T03:23:03.000000000', '2018-08-22T03:23:53.000000000',
       '2018-08-22T03:25:55.000000000', ..., '2018-08-22T08:18:10.000000000',
       '2018-08-22T08:19:00.000000000', '2018-08-22T08:19:50.000000000'],
      dtype='datetime64[ns]')
Coordinates:
  * trajectory  (trajectory) uint32 0 1 2 3 4 5 6 ... 278 279 280 281 282 283
Attributes:
    axis:           T
    long_name:      Time of observation
    standard_name:  time

In [2]: xr.open_dataset('both_times.nc').time
<xarray.DataArray 'time' (trajectory: 284)>
array(['2018-08-22T03:23:03.000000000', '2018-08-22T03:23:05.755359744',
       '2018-08-22T03:23:03.201308160', ..., '2018-08-22T03:23:06.144805888',
       '2018-08-22T03:23:04.605198336', '2018-08-22T03:23:03.065590784'],
      dtype='datetime64[ns]')
Coordinates:
  * trajectory  (trajectory) uint32 0 1 2 3 4 5 6 ... 278 279 280 281 282 283
Attributes:
    standard_name:  time
    long_name:      Time of observation
    axis:           T

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS

commit: None
python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10) [GCC 10.3.0]
python-bits: 64
OS: Linux
OS-release: 3.10.0-1160.62.1.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.10.6
libnetcdf: 4.8.0

xarray: 2022.3.0
pandas: 1.4.2
numpy: 1.22.3
scipy: 1.7.0
netCDF4: 1.5.7
pydap: None
h5netcdf: 1.0.0
h5py: 3.3.0
Nio: None
zarr: 2.11.3
cftime: 1.6.0
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: 0.9.10.1
iris: None
bottleneck: 1.3.4
dask: 2022.04.1
distributed: 2022.4.1
matplotlib: 3.5.1
cartopy: None
seaborn: 0.11.2
numbagg: None
fsspec: 2021.06.1
cupy: None
pint: 0.19.1
sparse: None
setuptools: 62.1.0
pip: 22.0.4
conda: 4.12.0
pytest: 7.1.1
IPython: 8.2.0
sphinx: 4.5.0

[spencerkclark]

Thanks @sappjw -- this is a distillation of the bug derived from your example:

>>> import numpy as np
>>> import xarray as xr
>>> xr.coding.times.decode_cf_datetime(np.uint32(50), "seconds since 2018-08-22T03:23:03Z")
array('2018-08-22T03:23:05.755359744', dtype='datetime64[ns]')

I believe the solution is to also cast all unsigned integer values -- anything with dtype.kind == "u" -- to np.int64 values here:

xarray/xarray/coding/times.py

Lines 220 to 224 in 770e878

	# To avoid integer overflow when converting to nanosecond units for integer
	# dtypes smaller than np.int64 cast all integer-dtype arrays to np.int64
	# (GH 2002).
	if flat_num_dates.dtype.kind == "i":
	flat_num_dates = flat_num_dates.astype(np.int64)

Ordinarily we might worry about overflow in this context -- i.e. some np.uint64 values cannot be represented by np.int64 values -- but I believe since we already verify that the minimum and maximum value of the input array can be represented by nanosecond-precision timedelta values, we can safely do this.