Performance regression uint64 groupby().min() in 0.25.1 vs. 0.24.2

[deeplycloudy]

Code Sample, a copy-pastable example if possible

import pandas as pd
dft = pd.read_csv('pandas_gb_min_test.csv.gz', parse_dates=['split_lutevent_time_offset'])

# An unsigned dtype is slow (23 s) on 0.25.1, but fast on 0.24.2.
# Commenting out this line on 0.25.1 restores performance
dft['split_event_parent_event_id'] = dft.split_event_parent_event_id.astype('uint64')

for mdvn in dft.columns: print(mdvn, dft[mdvn].dtype, type(dft[mdvn]))
mindata = dft.groupby('xyidx').min()

Problem description

In pandas 0.25.1, the performance of df.groupby().min() is 30x slower than in 0.24.2 when one of the columns has a dtype of uint64.

I saved the exact dataframe min_data_in from my original code with min_data_in.to_csv('pandas_gb_min_test.csv'). The columns in the original code had the following data types:

>>> for mdvn in min_data_in.columns: print(mdvn, min_data_in[mdvn].dtype, type(min_data_in[mdvn]))

split_event_parent_event_id uint64 <class 'pandas.core.series.Series'>
split_lutevent_time_offset datetime64[ns] <class 'pandas.core.series.Series'>
split_event_mesh_x_idx int64 <class 'pandas.core.series.Series'>
split_event_mesh_y_idx int64 <class 'pandas.core.series.Series'>
split_lutevent_min_flash_area float64 <class 'pandas.core.series.Series'>
xyidx int64 <class 'pandas.core.series.Series'>

After loading with read_csv the dtype of split_event_parent_event_id was signed.

If I restore the uint64 dtype, v0.25.1 runtime is 23 s. Runtime for v0.24.2 is 0.7 s.

A signed dtype in v0.25.1 has the same performance as 0.24.2.

The data file, cProfile capture, and snakeviz output are attached.

Output of pd.show_versions()

0.24.2
and
0.25.1

pd0242.pdf
pd0251.pdf
pd0251.prof.zip
pd0242.prof.zip
pandas_gb_min_test.csv.gz

[jbrockmendel]

The data file, cProfile capture, and snakeviz output are attached.

This is better than nothing, but can you provide something we can copy/paste directly?

[deeplycloudy]

The revised code, below, is sufficient to demonstrate the delta.
0.24.2 = 2 seconds
0.25.1 = 14 seconds

n=10000000
x = np.random.randint(0, 100000, size=n).astype('int64')
a = np.random.randint(0, 1000, size=n).astype('uint64')
df = pd.DataFrame({'a':a, 'x':x})
df.groupby('x').min() 

The delta is even worse on the original dataset above, suggesting a dependence on the number of groups in the groupby.

Complete profile dump and visualization in the attached.

pd0251synthetic.pdf
pd0242synthetic.pdf

[jbrockmendel]

Thanks.

It looks like in groupby.generic there is a cyfunc = self._get_cython_func(func_or_funcs) that is returning None. I'll have to look closer to be sure, but I expect it should be finding a cythonized implementation.

[jbrockmendel]

Yep, groupby.group L 1372 there is a try/except with self._cython_agg_general(alias, alt=npfunc, **kwargs) that raises on master and succeeds in 0.24.2

PR would be welcome.

[jbrockmendel]

In _libs.groubpy_helper groupby_t needs to be amended to include uint64_t.

[davidkimble]

I too am noticing this issue, but not exclusive to uint64. I'm also seeing the performance regression on bool and float32 column types. ~25x slower in 0.25.1+ than in 0.24.2

[oscar6echo]

I have ran into this problem too: Any groupby on more than 7/8 columns takes forever. This performance regression is a wall.
Practically, I stick to version 0.24.2 in scripts that rely on groupby operations.

[smithto1]

This issue is fixed in the latest release.

@WillAyd @jbrockmendel should we add an ASV check to monitor this?

0.24.2
806 ms ± 160 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

1.2.4
727 ms ± 12.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Using minimal example below:

import numpy as np
import pandas as pd

n=10000000
x = np.random.randint(0, 100000, size=n).astype('int64')
a = np.random.randint(0, 1000, size=n).astype('uint64')
df = pd.DataFrame({'a':a, 'x':x})

print(pd.__version__)

%timeit df.groupby('x').min() 

[jbrockmendel]

should we add an ASV check to monitor this?

go for it

[smithto1]

take