API: Expand DataFrame.astype to allow Categorical(categories, ordered)

[TomAugspurger]

A small, complete example of the issue

This is a proposal to allow something like

df.astype({'A': pd.CategoricalDtype(['a', 'b', 'c', 'd'], ordered=True})

Currently, it can be awkward to convert many columns in a DataFrame to a Categorical with control over the categories and orderedness. If you just want to use the defaults, it's not so bad with .astype:

In [5]: df = pd.DataFrame({"A": list('abc'), 'B': list('def')})

In [6]: df
Out[6]:
   A  B
0  a  d
1  b  e
2  c  f

In [8]: df.astype({"A": 'category', 'B': 'category'}).dtypes
Out[8]:
A    category
B    category
dtype: object

If you need to control categories or ordered, your best off with

In [20]: mapping = {'A': lambda x: x.A.astype('category').cat.set_categories(['a', 'b'], ordered=True),
    ...:            'B': lambda x: x.B.astype('category').cat.set_categories(['d', 'f', 'e'], ordered=False)}

In [21]: df.assign(**mapping)
Out[21]:
     A  B
0    a  d
1    b  e
2  NaN  f

By expanding astype to accept instances of Categorical, you remove the need for the lambdas and you can do conversions of other types at the same time.

This would mirror the semantics in #14503

Updated to change pd.Categorical(...) to a new/modified pd.CategoricalDtype(...) based on the discussion below.

[jreback]

so our current semantics are wrong on CategoricalDtype, IOW, we really should actually have the categories (and ordered) as part of the actual dtype, then that would be what you would pass here.

But since we don't support that ATM, passing an 'empty' Categorical is prob reasonable.

[TomAugspurger]

But since we don't support that ATM, passing an 'empty' Categorical is prob reasonable.

Can you clarify that? Are you saying anywhere we currently allow dtype='category', we really mean dtype=pd.Categorical()? Because I don't think those are the same. The dtype='category' is on a higher level than any instance of the type, including an empty Categorical. I could be misunderstanding though.

[jreback]

what I am saying is that in reality categorical dtype is actually mis-defined (has always been); we have a single instance of it. In reality it should have the categories IN the dtype (and ordered). But that's not how its actually setup. This is more theoretical, because its sort of hard to change it now.

[TomAugspurger]

Mmm, I think I see what you're saying now. In other words, currently pd.Categorical(values, categories, ordered) is a value constructor. We want something like pd.CategoricalType(categories, ordered) as a type constructor (which is what I had in mind when writing up the initial issue). I'll do some thinking / researching on how other languages handle this.

[jreback]

right. It is possible to add a constructor like that to CategoricalDtype.