BUG: pd.concat doesn't preserve categorical dtypes

[YarShev]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

df = pd.DataFrame({"a": [0, 1]})
df2 = pd.DataFrame({"a": [2, 3]})

print(df.dtypes)
# a    int64
# dtype: object
print(df2.dtypes)
# a    int64
# dtype: object

df3 = pd.concat([df, df2], axis=0)
print(df3.dtypes)
# a    int64
# dtype: object

df = df.astype('category')
df2 = df2.astype('category')

print(df.dtypes)
# a    category
# dtype: object
print(df2.dtypes)
# a    category
# dtype: object

df3 = pd.concat([df, df2], axis=0)
print(df3.dtypes)
a    int64
dtype: object

Issue Description

pd.concat doesn't preserve categorical dtype if the dfs have categorical columns, whereas pd.concat preserves int dtype.

Expected Behavior

The result dtype should be categorical, shouldn't it?

Installed Versions

pandas : 1.5.3
numpy : 1.24.1
pytz : 2022.7.1
dateutil : 2.8.2
setuptools : 66.1.1
pip : 22.3.1
Cython : None
pytest : 7.2.1
hypothesis : None
sphinx : 6.1.3
blosc : None
feather : 0.4.1
xlsxwriter : None
lxml.etree : 4.9.2
html5lib : None
pymysql : None
psycopg2 : 2.9.3
jinja2 : 3.1.2
IPython : 8.8.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli :
fastparquet : 2022.12.0
fsspec : 2023.1.0
gcsfs : None
matplotlib : 3.6.3
numba : None
numexpr : 2.8.3
odfpy : None
openpyxl : 3.0.10
pandas_gbq : 0.15.0
pyarrow : 10.0.1
pyreadstat : None
pyxlsb : None
s3fs : 2023.1.0
scipy : 1.10.0
snappy : None
sqlalchemy : 1.4.46
tables : 3.7.0
tabulate : None
xarray : 2023.1.0
xlrd : 2.0.1
xlwt : None
zstandard : None
tzdata : None

[jbrockmendel]

The dtypes are both categorical, but not matching categoricals. With different dtypes, you get casting to a common dtype, in this case int64. (A reasonable case could be made for casting to a union_categorical dtype, i think there are issues about this). If you cast to a common dtype first, you retain the categorical-ness.

df['a'] = df['a'].cat.set_categories([0, 1, 2, 3])
df2['a'] = df2['a'].cat.set_categories([0, 1, 2, 3])

df3 = pd.concat([df, df2], axis=0)

>>> df3.dtypes
a    category
dtype: object

[ddrinka]

I just ran into this as well. I expected concatenating two dataframes with different sets of categorical values to result in a new dataframe with a common set of categorical values. Instead the column was converted to a string.

Here's some history: #25412

[YarShev]

A reasonable case could be made for casting to a union_categorical dtype

I would like this to get fixed in 1.5.4.

[jbrockmendel]

xref #42840

[jbrockmendel]

Looks like #14177 is the API discussion. Mostly seems in favor.

[phofl]

I would like this to get fixed in 1.5.4.

While I'd be in favour of combining the categories, this does not qualify as a regression fix and hence is out of scope for 1.5.4

[YarShev]

This should probably be included in 1.6.0 then.

[jbrockmendel]

There is no 1.6. 2.0rc is expected this week

[jbrockmendel]

Discussed this on the dev call today, agreed to add a keyword to pd.concat to enable this behavior.

[YarShev]

Cool! Will that feature be included in 2.0?

[jbrockmendel]

no, a 2.0rc has already been released. it'll get into 2.1 if someone implements it

[YarShev]

Good, thanks! Hope someone will pick this up to add into 2.1.