Importing pkg_resources is slow. Work is done at import time.

[dsully]

Any "import pkg_resources" by a module is slow, in the 100-150ms range, depending on the system.

This is due to the number of modules imported by pkg_resources itself (email.parser is also slow), and side-effect work done at import time, notably the _initialize and _initialize_master_working_set functions at the bottom of pkg_resources/__init__.py.

As wall clock time matters to humans running CLIs, every millisecond counts.

The work done at import time should be deferred until needed, and the imports themselves also deferred if possible.

[rawouter]

Updating this bug, I traced it down from a very slow import OpenSSL on our system due to this issue. We do have a pretty large sys.path with lots of user generated 3rd party libraries, it takes more than 0.800ms to load and we aim to run scripts subseconds which is impossible:

$ time python -c "import pkg_resources"
real	0m0.896s
user	0m0.324s
sys	0m0.136s

$ python -c "import sys; print(len(sys.path))"
40

[akx]

I'm seeing ~200 msec for a click CLI app I'm developing (printing the usage):

With:     0.43 real         0.38 user         0.04 sys
Without:  0.25 real         0.22 user         0.02 sys

[cboylan]

Ran into this and poked at it briefly. It looks like a significant portion of the cost here is related to the sorting in https://github.com/pypa/setuptools/blob/master/pkg_resources/__init__.py#L1977-L2000. Notice the number of calls to _by_version and total cumulative time there in the attached file. This file was generated by running python -m cProfile -s cumulative foo.py where foo.py is only import pkg_resources. Digging deeper the biggest cost in sorting appears to be parsing all of the versions using packaging.version.parse.

pkg_resources_import_profile.txt

The other big cost here appears to be https://github.com/pypa/setuptools/blob/master/pkg_resources/__init__.py#L640-L658.

I don't know enough about why this state is generated on import, but it seems like maybe things like sorting or even reading through the entire list of packages available could potentially be deferred until the info is necessary (also why is sorting important at all?).

And finally pip list reports setuptools (34.2.0) is the version of setuptools in use here.

[socketpair]

https://github.com/ninjaaron/fast-entry_points

[mgedmin]

Duplicate of #510?

[jaraco]

Yes, thanks.