pandas-0.23 breaks stack with duplicated indices

[crusaderky]

In this script:

import pandas
import xarray

df = pandas.DataFrame(
    [[1, 2], [3, 4]],
    index=['foo', 'foo'],
    columns=['bar', 'baz'])
print(df.stack())

a = xarray.DataArray(df)
print(a.stack(s=a.dims))

The first part works both with pandas 0.22 and 0.23.
The second part works in xarray 0.10. 4 + pandas 0.22, and crashes with pandas 0.23:

  File "/mnt/resource/tmp/anaconda_guido/lib/python3.6/site-packages/xarray/core/dataarray.py", line 1115, in stack
    ds = self._to_temp_dataset().stack(**dimensions)
  File "/mnt/resource/tmp/anaconda_guido/lib/python3.6/site-packages/xarray/core/dataset.py", line 2123, in stack
    result = result._stack_once(dims, new_dim)
  File "/mnt/resource/tmp/anaconda_guido/lib/python3.6/site-packages/xarray/core/dataset.py", line 2092, in _stack_once
    idx = utils.multiindex_from_product_levels(levels, names=dims)
  File "/mnt/resource/tmp/anaconda_guido/lib/python3.6/site-packages/xarray/core/utils.py", line 96, in multiindex_from_product_levels
    return pd.MultiIndex(levels, labels, sortorder=0, names=names)
  File "/mnt/resource/tmp/anaconda_guido/lib/python3.6/site-packages/pandas/core/indexes/multi.py", line 240, in __new__
    result._verify_integrity()
  File "/mnt/resource/tmp/anaconda_guido/lib/python3.6/site-packages/pandas/core/indexes/multi.py", line 283, in _verify_integrity
    level=i))
ValueError: Level values must be unique: ['foo', 'foo'] on level 0

[jhamman]

See also #1499. I don't think we have any tests for duplicate dimensions in xarray so I'm not surprised this has stopped working.

[shoyer]

It looks like we were indeed creating a MultiIndex with duplicate level values in this case, which is probably not a great idea -- it's a slightly weird edge case in pandas' data model.

We should probably update the logic in our multiindex_from_product_levels() function to factorize levels if they aren't already unique.

[shoyer]

See #2168