Crash on latest Django Project with dmypy #11456

Thorbenl · 2021-11-04T10:27:57Z

Bug Report

Canot run mypy on latest django, mypy and django-stubs version. started crashin after updating django from 3.2.8 to 3.2.9

Expected Behavior

normal typecheck run

Actual Behavior
crashes with:

Daemon started
Daemon crashed!
Traceback (most recent call last):
  File "mypy/dmypy_server.py", line 221, in serve
  File "mypy/dmypy_server.py", line 264, in run_command
  File "mypy/dmypy_server.py", line 323, in cmd_run
  File "mypy/dmypy_server.py", line 380, in check
  File "mypy/dmypy_server.py", line 410, in initialize_fine_grained
  File "mypy/build.py", line 179, in build
  File "mypy/build.py", line 253, in _build
  File "mypy/build.py", line 2638, in dispatch
  File "mypy/build.py", line 2962, in process_graph
  File "mypy/build.py", line 3071, in process_stale_scc
  File "mypy/build.py", line 2172, in finish_passes
  File "mypy/build.py", line 2220, in update_fine_grained_deps
  File "mypy/build.py", line 2211, in compute_fine_grained_deps
  File "mypy/server/deps.py", line 114, in get_dependencies
  File "mypy/nodes.py", line 305, in accept
  File "mypy/server/deps.py", line 176, in visit_mypy_file
  File "mypy/traverser.py", line 35, in visit_mypy_file
  File "mypy/nodes.py", line 950, in accept
  File "mypy/server/deps.py", line 239, in visit_class_def
  File "mypy/traverser.py", line 71, in visit_class_def
  File "mypy/nodes.py", line 1015, in accept
  File "mypy/server/deps.py", line 351, in visit_block
  File "mypy/traverser.py", line 39, in visit_block
  File "mypy/nodes.py", line 687, in accept
  File "mypy/server/deps.py", line 201, in visit_func_def
  File "mypy/traverser.py", line 54, in visit_func_def
  File "mypy/traverser.py", line 51, in visit_func
  File "mypy/nodes.py", line 1015, in accept
  File "mypy/server/deps.py", line 351, in visit_block
  File "mypy/traverser.py", line 39, in visit_block
  File "mypy/nodes.py", line 1073, in accept
  File "mypy/server/deps.py", line 403, in visit_assignment_stmt
  File "mypy/server/deps.py", line 463, in process_lvalue
  File "mypy/server/deps.py", line 495, in get_non_partial_lvalue_type
AssertionError

(Write what happened.)

Your Environment

mypy==0.812
django-stubs==1.8.0
Python version used: 3.9.8
Operating system and version: MacOS 11

sobolevn · 2021-11-04T13:19:30Z

Please, report it in our repo: https://github.com/typeddjango/django-stubs/

Also:

Minimal repro will help a lot!
Try 1.9.0, it might help!

Thanks!

Thorbenl · 2021-11-08T07:43:30Z

@sobolevn I will try. Sadly, could only reproduce it on a fresh repo ones, and then it went away. For our project tho, it never goes away, does not matter which version combination i would take with stubs and mypy :( Ima try to pick it out of our giant repo.

christianbundy · 2021-11-16T23:17:20Z

Having the same problem here, it started with a Pipfile.lock update recently and only affects dmypy. We're running [email protected].

christianbundy · 2021-11-17T15:48:24Z

@Thorbenl Any chance you have aiohttp as a dependency? I dropped into pdb and here's what I saw:

(Pdb) lvalue.node.info
<TypeInfo aiohttp.http_parser.HttpParser>

After bisecting the project history, it looks like aio-libs/aiohttp@a6c7f15 as the problematic commit.

You can repro by creating mypy.ini in aiohttp with the contents:

[mypy]
cache_fine_grained = True
ignore_errors = True

And running mypy ..

(Technically you don't have to run errors, but this lets you bisect automatically.)

cc: @asvetlov

christianbundy · 2021-11-17T16:07:33Z

Found the problem, although I'm not completely sure why it's a problem:

- class HttpParser(abc.ABC):
+ class HttpParser(abc.ABC, Generic[_MsgT]):

christianbundy · 2021-11-17T16:14:05Z

Full traceback:

aiohttp/http_parser.py: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.920+dev.43eecdb0969c8cb8777fa802ba53e11075b4c802
Traceback (most recent call last):
  File "/usr/local/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/usr/local/lib/python3.9/site-packages/mypy/__main__.py", line 11, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "/usr/local/lib/python3.9/site-packages/mypy/main.py", line 92, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/usr/local/lib/python3.9/site-packages/mypy/main.py", line 170, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 180, in build
    result = _build(
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 256, in _build
    graph = dispatch(sources, manager, stdout)
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 2712, in dispatch
    process_graph(graph, manager)
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 3043, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 3152, in process_stale_scc
    graph[id].finish_passes()
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 2225, in finish_passes
    free_tree(self.tree)
  File "/usr/local/Cellar/[email protected]/3.9.8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 1964, in wrap_context
    yield
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 2222, in finish_passes
    self.update_fine_grained_deps(self.manager.fg_deps)
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 2272, in update_fine_grained_deps
    merge_dependencies(self.compute_fine_grained_deps(), deps)
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 2263, in compute_fine_grained_deps
    return get_dependencies(target=self.tree,
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 117, in get_dependencies
    target.accept(visitor)
  File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 326, in accept
    return visitor.visit_mypy_file(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 177, in visit_mypy_file
    super().visit_mypy_file(o)
  File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 37, in visit_mypy_file
    d.accept(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 1000, in accept
    return visitor.visit_class_def(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 238, in visit_class_def
    super().visit_class_def(o)
  File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 73, in visit_class_def
    o.defs.accept(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 1069, in accept
    return visitor.visit_block(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 348, in visit_block
    super().visit_block(o)
  File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 41, in visit_block
    s.accept(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 727, in accept
    return visitor.visit_func_def(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 201, in visit_func_def
    super().visit_func_def(ex)
  File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 56, in visit_func_def
    self.visit_func(o)
  File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 53, in visit_func
    o.body.accept(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 1069, in accept
    return visitor.visit_block(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 348, in visit_block
    super().visit_block(o)
  File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 41, in visit_block
    s.accept(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 1135, in accept
    return visitor.visit_assignment_stmt(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 400, in visit_assignment_stmt
    self.process_lvalue(lvalue)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 460, in process_lvalue
    lvalue_type = self.get_non_partial_lvalue_type(lvalue)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 492, in get_non_partial_lvalue_type
    assert not lvalue.is_new_def
AssertionError:

christianbundy · 2021-11-17T16:27:53Z

Minimal repro!

from typing import Generic, TypeVar

T = TypeVar('T', int, str)

class ExampleClass(Generic[T]):
    def __init__(
        self
    ) -> None:
        self.example_attribute = None

This seems to happen when:

Fine-grained cache is enabled (e.g. manually in config, or mandatory if using dmypy)
Some TypeVar (T) has 2+ types.
Some class subclasses Generic[T]
The init method on that class sets an attribute to None.

🤯

sobolevn · 2021-11-17T17:00:56Z

Btw, this works fine:

from typing import Generic, TypeVar

T = TypeVar('T', int, str)

class ExampleClass(Generic[T]):
    def __init__(
        self
    ) -> None:
        ...

The problem reveals itself only if self.example_attribute = None is in place.

Great insight, thanks!

christianbundy · 2021-11-17T17:03:16Z

Any ideas on how we might solve it? I haven't contributed to mypy, and honestly I'm not completely sure what purpose this assert has in the first place.

sobolevn · 2021-11-17T17:08:02Z

It happens here:

 File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 400, in visit_assignment_stmt
    self.process_lvalue(lvalue)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 460, in process_lvalue
    lvalue_type = self.get_non_partial_lvalue_type(lvalue)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 492, in get_non_partial_lvalue_type
    assert not lvalue.is_new_def

I've never touched this part of mypy before. And these lines of code are quite old (4 years):

mypy/mypy/server/deps.py

Lines 487 to 493 in 43eecdb

    
           # Probably a secondary, non-definition assignment that doesn't 
        
           # result in a non-partial type. We won't be able to infer any 
        
           # dependencies from this so just return something. (The first, 
        
           # definition assignment with a partial type is handled 
        
           # differently, in the semantic analyzer.) 
        
           assert not lvalue.is_new_def 
        
           return UninhabitedType()

Git Blame Link

This can either be:

A trivial if bug, where we just forgot to add some condition
Quite complex one, due to our way-overly-compicated-plugin

sobolevn · 2021-11-17T17:17:06Z

@christianbundy this seems to remove the crash:

            if isinstance(lvalue.node, Var):
                if lvalue.node.type:
                    lvalue_type = get_proper_type(lvalue.node.type)
                else:
                    lvalue_type = NoneType()

But, I am not sure it is correct though. Do you want to send a PR? We can work from there!

Problem: When using the fine-grained cache, we raise an assertion in some situations when an attribute is defined as `None`. Solution: Remove truthiness check to ensure that `None` is an allowed type. Fixes: python#8682 Fixes: python#11456

christianbundy · 2021-11-17T17:44:30Z

Good idea, thank you!

Problem: When using the fine-grained cache, we raise an assertion in some situations when an attribute is defined as `None`. Solution: Remove truthiness check to ensure that `None` is an allowed type. Fixes: #8682 Fixes: #11456

python/mypy#11456

Problem: When using the fine-grained cache, we raise an assertion in some situations when an attribute is defined as `None`. Solution: Remove truthiness check to ensure that `None` is an allowed type. Fixes: python#8682 Fixes: python#11456

python/mypy#11456

Thorbenl added the bug mypy got something wrong label Nov 4, 2021

christianbundy mentioned this issue Nov 16, 2021

AssertionError mypy/server/deps.py:495 #8682

Closed

christianbundy mentioned this issue Nov 17, 2021

Fix None assignments with fine-grained cache #11574

Merged

hauntsaninja closed this as completed in #11574 Dec 8, 2021

severo added a commit to huggingface/dataset-viewer that referenced this issue Dec 14, 2021

chore: 🤖 avoid an error in vsscode with mypy daemon

348a6ae

python/mypy#11456

RJPercival mentioned this issue Jan 11, 2022

0.930 still gives me "inheriting final" error with an enum.Flag subclass #11850

Closed

mattstern31 added a commit to mattstern31/datasets-server-storage-admin that referenced this issue Nov 11, 2023

chore: 🤖 avoid an error in vsscode with mypy daemon

0d0e4a5

python/mypy#11456

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash on latest Django Project with dmypy #11456

Crash on latest Django Project with dmypy #11456

Thorbenl commented Nov 4, 2021

sobolevn commented Nov 4, 2021 •

edited

Loading

Thorbenl commented Nov 8, 2021

christianbundy commented Nov 16, 2021

christianbundy commented Nov 17, 2021

christianbundy commented Nov 17, 2021

christianbundy commented Nov 17, 2021

christianbundy commented Nov 17, 2021

sobolevn commented Nov 17, 2021

christianbundy commented Nov 17, 2021

sobolevn commented Nov 17, 2021

sobolevn commented Nov 17, 2021

christianbundy commented Nov 17, 2021

Crash on latest Django Project with dmypy #11456

Crash on latest Django Project with dmypy #11456

Comments

Thorbenl commented Nov 4, 2021

sobolevn commented Nov 4, 2021 • edited Loading

Thorbenl commented Nov 8, 2021

christianbundy commented Nov 16, 2021

christianbundy commented Nov 17, 2021

christianbundy commented Nov 17, 2021

christianbundy commented Nov 17, 2021

christianbundy commented Nov 17, 2021

sobolevn commented Nov 17, 2021

christianbundy commented Nov 17, 2021

sobolevn commented Nov 17, 2021

sobolevn commented Nov 17, 2021

christianbundy commented Nov 17, 2021

sobolevn commented Nov 4, 2021 •

edited

Loading