[django-filter] Absorb django-filter-stubs into typeshed

[intgr]

As described in #10918, I have added stubs from django-filter-stubs project. Upstream issue: DavisRayM/django-filter-stubs#13

Please don't use squash merge to merge this (unless there's explicit decision to reconsider #10918). I have preserved original commit authors. There really weren't many changes in stubs files.

• Closes Proposal: absorb django-filter-stubs into typeshed #10918
• Closes Consider move the repo to typeshed? DavisRayM/django-filter-stubs#13

[AlexWaygood]

pre-commit.ci autofix

[intgr]

This should be more or less ready for initial merge. If we can get this merged, I can look into the stubtest allowlist issues separately. Unless you think I should fix them immediately here?

Some questions:

1. The stubs depend on django-stubs and djangorestframework-stubs. If we continue with this merge, these need to be added to stub_uploader allowlist. Nikita Sobolev can vouch for the fact that these dependencies aren't evil 😈

   Opened PR: Add django-stubs, djangorestframework-stubs to allowlist typeshed-internal/stub_uploader#111

2. stubtest gets exception django.core.exceptions.ImproperlyConfigured from poking around in some django-filter internals. I could just allowlist these.

   But if we could somehow set DJANGO_SETTINGS_MODULE env variable for the stubtest run (to some empty Python module even), then we wouldn't need to suppress them. Is there any precedent to this?

3. Upstream used Any for incomplete types. Should I convert all of these to _typeshed.Incomplete or not bother with this?

[intgr]

Some further questions. Personally I'd prefer to get this merged first and then do further improvements in follow-up PRs.

[intgr]

There are lots of re-exports that shouldn't be re-exported. Should I clean these up before initial merge, or in follow-up?

[AlexWaygood]

follow-up is fine imo

[intgr]

Should I clean up attribute assignments?

Suggested change

	widget: Any = ...
	widget: Any

[AlexWaygood]

Fine to do this in a follow-up imo

[AlexWaygood]

(Writing from my phone, I have limited access to a laptop with Internet until Tuesday)

In general, I think it's fine to tackle issues with these stubs in followup PRs, now you've got CI passing (thanks!). Most things in stubs/peewee/ are still unannotated; we still have many allowlist entries for types-redis — django-filter will hardly be unique in its stubs not being quite up to the standard of, say, our stdlib stubs.

For your question (1) — I can't remember off the top of my head what the exact rules are, but unfortunately adding django-stubs and djangorestframework-stubs to the stub_uploader allowlist might not be sufficient to get CI passing here. I think the stub_uploader might also check that all external dependencies listed in the requires field are actually dependencies of the runtime django-filter package, as an additional security check. I might be misremembering, though.

For your question (2), I think we're reasonably open to adding special-casing for specific stubs packages if it means we can run stubtest on them in CI. If you look at tests/stubtest_third_party.py, we already have some special-casing hardcoded into the script so that we can run stubtest on types-uwisgi in CI

[intgr]

For your question (1) [...] unfortunately adding django-stubs and djangorestframework-stubs to the stub_uploader allowlist might not be sufficient to get CI passinghere. I think the stub_uploader might also check that all external dependencies listed in the requires field are actually dependencies of the runtime django-filter package

Yep, that's also mentioned in typeshed-internal/stub_uploader#90 (comment)

Ideally these wouldn't be hard dependencies of the pacakge, but would only be installed in CI when testing. But there seems to be no such option currently?

Users of django who want to use types-django-filter are likely to have Django's stubs installed anyway. And if they don't, then types-django-filter is just less useful, but still better than nothing.

There are two different stubs packages for Django: django-stubs and django-types and we shouldn't force that choice.

[intgr]

I think anything else we can fix/improve afterwards, but the stub_uploader error is a blocker: #10919 (comment)

One idea would be to create a new METADATA.toml option e.g. test_requires = [...] that's only used in CI and not subject to the strict checks as requires = [...] is. What do you think?

[AlexWaygood]

One idea would be to create a new METADATA.toml option e.g. test_requires = [...] that's only used in CI and not subject to the strict checks as requires = [...] is. What do you think?

I'd prefer to avoid doing that, if at all possible. In 99% of cases in typeshed, if an external package is needed to make our tests pass, it's also needed for the stubs to make sense when they're installed by a user. I worry that by adding this option to our METADATA.toml files, we'd encourage people to add "test-only" dependencies which would mean our stubs would start behaving very differently under the conditions of our tests than they actually do when installed by users. We want to avoid that -- we want our tests to represent as closely as possible how our stubs packages behave when they're installed by users.

I believe that if a stubs package from typeshed imports a type that mypy can't resolve (because a dependency is missing), the imported type will just get silently resolved as Any -- mypy won't complain about it! So the user will get no warning that the stubs they installed aren't working as intended.

For the specific case of types-django-filter, maybe it's extremely unlikely that people will have types-django-filter installed without having django-stubs or django-types installed. But... that still feels like a risk I'm not willing to take -- people do crazy things on the internet :)

I'm going to take a look now and see if there are any other ways of solving this problem.

[intgr]

Many months have gone past. Has anything changed in terms of introducing a dependency on django-stubs? Or maybe some fresh ideas to solve this blocker?

There are a few other Django-related or djangorestframework-related projects I might want to provide stubs for, but no point in undertaking that work until this blocker has been cleared. And maintaining a 3rd party stubs project outside of typeshed is a bigger commitment than I want to undertake.

[srittau]

@intgr Sorry for the long process. I've added the Django stubs to the stub uploader allowlist. I see this as uncritical. There are some type errors now, though.

[intgr]

Thanks for helping push this forward.

Note that stub_uploader tests are giving the following error:

E stub_uploader.metadata.InvalidRequires: Expected dependency django-stubs to be listed in django-filter's requires_dist or the sdist's *.egg-info/requires.txt

This blocker was described in typeshed-internal/stub_uploader#111 (comment) / typeshed-internal/stub_uploader#90 (comment)

Do you have an idea what to do about this error? I suspect django-filter upstream will not want to add django-stubs to its dependencies.

[Ganji00]

@intgr maybe we can try try to make a PR with django-stubs as a dependency to django-filter?

[JelleZijlstra]

Sorry for the inaction here. I feel we should be able to address this with changes in stub-uploader. (Maybe a rule that if you depend on X at runtime, the stubs may depends on X-stubs, for some trusted values of X?)

[github-actions]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉