[BUG] DataFrame.reindex behavior differs from pandas

[bdice]

Describe the bug
The behavior of DataFrame.reindex fails doctests and differs from pandas. From a preliminary discussion in our dev meeting, we may have the behavior that we intend already in place -- this issue can be closed if we choose to keep the current implementation and update the doctests in #9815 to match.

Steps/Code to reproduce bug
This doctest (#9815) fails:

cudf/python/cudf/cudf/core/dataframe.py

Lines 2254 to 2272 in 582cc6e

	>>> import cudf
	>>> df = cudf.DataFrame()
	>>> df['key'] = [0, 1, 2, 3, 4]
	>>> df['val'] = [float(i + 10) for i in range(5)]
	>>> df_new = df.reindex(index=[0, 3, 4, 5],
	... columns=['key', 'val', 'sum'])
	>>> df
	key val
	0 0 10.0
	1 1 11.0
	2 2 12.0
	3 3 13.0
	4 4 14.0
	>>> df_new
	key val sum
	0 0 10.0 NaN
	3 3 13.0 NaN
	4 4 14.0 NaN
	5 -1 NaN NaN

Below is the doctest output:

File "/home/bdice/code/cudf/python/cudf/cudf/core/dataframe.py", line 2289, in DataFrame.reindex
Failed example:
    df_new
Expected:
       key   val  sum
    0    0  10.0  NaN
    3    3  13.0  NaN
    4    4  14.0  NaN
    5   -1   NaN  NaN
Got:
        key   val   sum
    0     0  10.0  <NA>
    3     3  13.0  <NA>
    4     4  14.0  <NA>
    5  <NA>  <NA>  <NA>

Expected behavior

Pandas gives the following output (uses NaN instead of <NA>, and casts the column key to type float64):

>>> pd.DataFrame({'key': [0, 1, 2, 3, 4], 'value': [10.0, 11.0, 12.0, 13.0, 14.0]}).reindex(index=[0, 3, 4, 5], columns=['key', 'value', 'sum'])
   key  value  sum
0  0.0   10.0  NaN
3  3.0   13.0  NaN
4  4.0   14.0  NaN
5  NaN    NaN  NaN

[shwina]

I think here we have the right behaviour. All our types are nullable, so we can afford to use <NA> as a null value and avoid casting things to float. We may need a way of telling doctests to not check the result dtypes when necessary (similar to the check_dtype=False argument to assert_frame_equal in pytests).

[bdice]

@shwina Awesome, thanks for confirming. I have updated the doctest and will close this issue.

We may need a way of telling doctests to not check the result dtypes when necessary (similar to the check_dtype=False argument to assert_frame_equal in pytests).

No additional changes are necessary beyond updating the output of the example in 699c21a. Doctests execute the commands and compare the printed output. In some sense, doctests are a test of the REPL environment and validate the workings of repr / str methods.