Segfault with black 22.1.0

[evilrhinohater]

Describe the bug

When running black with a self-built Python 3.10 I get a segfault.

To Reproduce

Can be reproduced in Docker:

#Dockerfile
FROM ubuntu:20.04

RUN sed -i '/deb-src/s/^# //' /etc/apt/sources.list
RUN apt-get update

ARG DEBIAN_FRONTEND=noninteractive 
ARG TZ=UTC

RUN apt-get build-dep -y python3

# https://devguide.python.org/setup/#install-dependencies
RUN apt-get install -y build-essential gdb lcov pkg-config \
      libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev \
      libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev \
      lzma lzma-dev tk-dev uuid-dev zlib1g-dev

RUN apt-get -y install curl

RUN curl -o Python-3.10.2.tar.xz https://www.python.org/ftp/python/3.10.2/Python-3.10.2.tar.xz
RUN tar xf Python-3.10.2.tar.xz
RUN cd Python-3.10.2 \
  && ./configure --with-pydebug --with-ensurepip \
  && make \
  && make altinstall

RUN adduser myuser
RUN mkdir -p /var/blacktest
RUN chown myuser:myuser /var/blacktest

ENV PATH /usr/local/bin:${PATH}

USER myuser
WORKDIR /var/blacktest
COPY Pipfile Pipfile
RUN pip3.10 install pipenv
RUN python3.10 -m pipenv install --dev

Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]
black = "*"

[requires]
python_version = "3.10"

Then running black leads to a segfault error

$ docker build -t blacktest .
$ docker run --rm -it blacktest
myuser@c19751e02d87: python3.10 -m pipenv shell
(blacktest) myuser@c19751e02d87:/var/blacktest$ black
Objects/typeobject.c:5970: type_ready_checks: Assertion "type->tp_call != ((void *)0)" failed
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x559372c83ca0
object refcount : 1
object type     : 0x55937285cea0
object type name: type
object repr     : Segmentation fault (core dumped)

Expected behavior

Black should not lead to a segfault.

Environment

• Black's version: 22.1.0
• OS and Python version: Ubuntu 20.04, Python 3.10 (built from source, see above)

** Further remarks **

Curiously, I could run the tests as instructed without errors.

[JelleZijlstra]

Thanks for reporting this!

I don't have time to investigate in detail right now, but here's a few things that anyone reading this could do to help:

• The crash is almost certainly related to our use of mypyc. Thus, a workaround could be to install from source (I think --no-binary :all: is the right pip option). Also, maybe there's already a known issue over at https://github.com/mypyc/mypyc that explains what we're seeing.
• It would be helpful to get a full stack trace. Is the crash reproducible in gdb?
• Similarly, is it possible to figure out which code in Black is triggering the crash?

[ichard26]

The crash is almost certainly related to our use of mypyc. Thus, a workaround could be to install from source (I think --no-binary :all: is the right pip option). Also, maybe there's already a known issue over at https://github.com/mypyc/mypyc that explains what we're seeing.

I recommend using --no-binary black instead of blocklisting all wheels from deps to minimize build time issues even from Pure Python packages.

[evilrhinohater]

Thanks for the suggestion. I can confirm that --no-binary black solves this issue for me.

[godlygeek]

python/mypy#12605 reports this bug upstream - mypyc is generating code that provokes a Python assertion error when a class inherits __call__ from its superclass, and the assertion handler is crashing in the middle of trying to report that assertion violation.

[JulienPalard]

Can reproduce with black==22.8.0.

[ichard26]

python/mypy#13152 fixes this, but it hasn't made its way into a mypy release yet. The next release is being tracked in python/mypy#13385 but it seems to be behind schedule (as usual for mypy). Once mypy cuts a new release I'll upgrade our mypy[c] cibuildwheel requirement, resolving this.

[uweschmitt]

Any updates on this? mypy published python/mypy#13385 mean while. I reinstalled black and got version 22.10.0 which still crashes on my Mac.

[JelleZijlstra]

The latest release, 22.12.0, was made with mypy 0.991, but for unrelated reasons we don't have MacOS mypyc wheels.

[ichard26]

Black 23.1.0 no longer runs into this segfault as it was built using mypy(c) 0.991. Interestingly Black 22.12.0 still segfaulted... I'm on Linux with a CPython 3.11 debug build for what it's worth.

[ichard26]

Ah, we bumped mypy(c) the day after 22.12.0 was cut.

commit 80de2372e4ec385c082c38f365414ac3622a4010
Author: Richard Si <[email protected]>
Date:   Sat Dec 10 15:56:34 2022 -0500

    Bump mypy[c] from 0.971 to 0.991 (#3360)