Improve start-up time by deferring slow imports. #6
Conversation
The `configurator` package takes 200-300 ms to import, which is enough to cause a noticeable pause when a program is starting up. Almost all of this time is spent importing `pkg_resources`: $ python -X importtime -c 'import configurator' $ python -X importtime -c 'import pkg_resources' By defering this import until it is needed, `configurator` can start up in about 7 ms---much faster. This is really nice if the command being executed doesn't actually need to read a config file, e.g. if it's just displaying usage help or something.
| @@ -12,6 +10,9 @@ class Parsers(dict): | |||
|
|
|||
| @classmethod | |||
| def from_entrypoints(cls): | |||
| # `pkg_resources` is slow to import, so defer until we need it. | |||
| from pkg_resources import iter_entry_points | |||
There was a problem hiding this comment.
This is a bit ugly. Why is important pkg_resources so slow? Have you reported the issue upstream at https://github.com/pypa/setuptools?
There was a problem hiding this comment.
It is a bit ugly. I don't know why it's so slow, but there are already a couple issues open relating to this: pypa/setuptools#510 pypa/setuptools#926.
|
I added another commit to defer a few more imports, specifically I don't think it's worth doing anything more thorough than this, since there's really not much time left to save. Go ahead and merge if everything looks good to you. Thanks for writing this library, by the way. I just found it a few weeks ago. In the past I'd write my own half-baked code every time I need to merge config files, but this is much nicer. |
|
I have to be honest, I'm having second thoughts about this PR. I'm really not a fan local imports, particularly those in a9f8e04. I'm also curious about the actual win here: if a How about having the import of configurator in your library be moved to where it's instantiated? Apologies for all the questions, trying to get to the right solution :-) |
|
No worries. My application is a command line tool with a number of different subcommands, many of which don't read any configuration files. I was trying to make the commands a little more snappy, so I profiled the imports and noticed that almost all of the import time was devoted to None of this is a big problem for me; as you point out, I can import Feel free to revert a9f8e04 (or I can do it, if that would be easier). I was just trying to address your comment about loading each library when it's used, but the performance improvement is negligible. I do think that |
|
I'm starting to think that allowing parsers to be provided by setuptools entrypoints might just be a bad idea :-/ |
|
Can you test 2.0.0? My hope is this will have much better performance, even when you use it! |
|
2.0.0 seems much faster. Thanks for taking the time to work on this! |
The
configuratorpackage takes 200-300 ms to import, which is enough to cause a noticeable pause when a program is starting up. Almost all of this time is spent importingpkg_resources:By deferring this import until it is needed,
configuratorcan start up in about 7 ms—much faster. This is really nice if the command being executed doesn't actually need to read a config file, e.g. if it's just displaying usage help or something.