Delay import of sympy until actually needed. #194

anntzer · 2020-01-27T07:13:06Z

sympy is a rather slow import (~330ms for me) which contributes nearly
half of bezier's import time (~750ms). Delay the import until actually
needed.

As a typical use case where this is visible: I have a command-line
utility which does some computation using bezier. Right now invoking
something as simple as my-program --help takes over a second to
complete (certainly a noticeable delay), in particular due to such slow
imports -- even if I do not use the symbolic computation part at all.

Obviously there are other ways in which this can be implemented, e.g. by
duplicating the try... except ImportError in each function, or with
another helper function, but given that you already have a
requires_sympy decorator, I thought I may as well reuse it for this
purpose.

Edit: The decorator magic apparently makes pylint/flake8 extremely unhappy. I can fix that with whichever approach you prefer -- skip linting on the offending lines, or just duplicate the try: except import everywhere, or a helper function.

dhermes · 2020-01-27T16:40:34Z

src/python/bezier/_symbolic.py


    return ensure_sympy


 @require_sympy
-def to_symbolic(nodes):
+def to_symbolic(sympy, nodes):


This would mean Args needs to be updated for this function. (Once upon a time Pylint would've flagged the mismatch, not sure what ~~the difference is~~ has changed.)

Ah I spoke too soon, lint failure from this PR:

************* Module bezier._symbolic src/python/bezier/_symbolic.py:52:0: W9015: "sympy" missing in parameter documentation (missing-param-doc) src/python/bezier/_symbolic.py:52:0: W9016: "sympy" missing in parameter type documentation (missing-type-doc) src/python/bezier/_symbolic.py:73:0: W9015: "sympy" missing in parameter documentation (missing-param-doc) src/python/bezier/_symbolic.py:73:0: W9016: "sympy" missing in parameter type documentation (missing-type-doc) src/python/bezier/_symbolic.py:98:0: W9015: "sympy" missing in parameter documentation (missing-param-doc) src/python/bezier/_symbolic.py:98:0: W9016: "sympy" missing in parameter type documentation (missing-type-doc) src/python/bezier/_symbolic.py:111:16: E1120: No value for argument 'nodes' in function call (no-value-for-parameter) src/python/bezier/_symbolic.py:114:31: E1120: No value for argument 's' in function call (no-value-for-parameter) src/python/bezier/_symbolic.py:122:0: W9015: "sympy" missing in parameter documentation (missing-param-doc) src/python/bezier/_symbolic.py:122:0: W9016: "sympy" missing in parameter type documentation (missing-type-doc) src/python/bezier/_symbolic.py:139:0: W9015: "sympy" missing in parameter documentation (missing-param-doc) src/python/bezier/_symbolic.py:139:0: W9016: "sympy" missing in parameter type documentation (missing-type-doc) src/python/bezier/_symbolic.py:157:22: E1120: No value for argument 'degree' in function call (no-value-for-parameter) src/python/bezier/_symbolic.py:159:11: E1120: No value for argument 's' in function call (no-value-for-parameter) src/python/bezier/_symbolic.py:139:22: W0613: Unused argument 'sympy' (unused-argument) src/python/bezier/_symbolic.py:163:0: W9015: "sympy" missing in parameter documentation (missing-param-doc) src/python/bezier/_symbolic.py:163:0: W9016: "sympy" missing in parameter type documentation (missing-type-doc) src/python/bezier/_symbolic.py:197:0: W9015: "sympy" missing in parameter documentation (missing-param-doc) src/python/bezier/_symbolic.py:197:0: W9016: "sympy" missing in parameter type documentation (missing-type-doc) src/python/bezier/_symbolic.py:211:16: E1120: No value for argument 'nodes' in function call (no-value-for-parameter) src/python/bezier/_symbolic.py:214:31: E1120: No value for argument 't' in function call (no-value-for-parameter) src/python/bezier/_symbolic.py:222:0: W9015: "sympy" missing in parameter documentation (missing-param-doc) src/python/bezier/_symbolic.py:222:0: W9016: "sympy" missing in parameter type documentation (missing-type-doc) src/python/bezier/_symbolic.py:246:0: W9015: "sympy" missing in parameter documentation (missing-param-doc) src/python/bezier/_symbolic.py:246:0: W9016: "sympy" missing in parameter type documentation (missing-type-doc) src/python/bezier/_symbolic.py:264:25: E1120: No value for argument 'degree' in function call (no-value-for-parameter) src/python/bezier/_symbolic.py:266:11: E1120: No value for argument 't' in function call (no-value-for-parameter) src/python/bezier/_symbolic.py:246:25: W0613: Unused argument 'sympy' (unused-argument) ************* Module bezier.triangle src/python/bezier/triangle.py:1180:29: E1120: No value for argument 'degree' in function call (no-value-for-parameter) src/python/bezier/triangle.py:1218:15: E1120: No value for argument 'degree' in function call (no-value-for-parameter) ************* Module bezier.curve src/python/bezier/curve.py:764:26: E1120: No value for argument 'degree' in function call (no-value-for-parameter) src/python/bezier/curve.py:800:15: E1120: No value for argument 'degree' in function call (no-value-for-parameter)

Well, it's not really an argument -- callers don't need to pass it, the decorator handles it.

I guess getting rid of requires_sympy and having something like

def _load_sympy(): try: import sympy except ImportError: raise OSError(...) else: return sympy def to_symbolic(nodes): sympy = _load_sympy() ...

would minimize repetition which remaining linter-friendly?

dhermes · 2020-01-27T16:43:53Z

Will try to think about what I'd "prefer" to make Pylint happy.

While we're on this "avoid imports" kick, does the SciPy import also slow things down? It's only needed for the pure Python implementations, so if you've got a wheel (or you manually built the Fortran speedups from source) you never need SciPy.

anntzer · 2020-01-27T16:52:53Z

Sure, looks like (in a completely clean venv) the presence of scipy slows down importing bezier from ~150ms to 240ms, so while it's not as much as sympy, it's definitely quite significant. I can prepare a PR once we figure out the best pattern to deal with sympy.

dhermes · 2020-01-27T16:56:02Z

Somewhat related (i.e. caused builds in #195 to fail): I pushed a change to avoid pushing to coveralls for non-master branches (5b8e93b). Probably could've opened it up a bit and just checked CIRCLE_PR_NUMBER / CIRCLE_PULL_REQUEST / CIRCLE_PULL_REQUESTS.

Merge master in here (or rebase, which is my personal preferred strategy).

anntzer · 2020-01-27T16:58:07Z

I can rebase, but let's first figure out how you prefer me to import sympy?

dhermes · 2020-01-27T17:01:03Z

So one of the things that's throwing off Pylint is the fact that the decorator "swallows" one of the arguments. I'm not a fan of magic like that either, so rather than a decorator, we could at least consider starting each function with

sympy = require_sympy()

(and changing require_sympy() to just return sympy)

In _plot_helpers.py I didn't even both with a helpful error message on ImportError, I just tried the import at function call time. (I wrote _plot_helpers.py a long time ago.)

anntzer · 2020-01-27T17:02:03Z

basically #194 (comment)?

dhermes · 2020-01-27T17:02:46Z

basically #194 (comment)?

Haha absolutely yes!

anntzer · 2020-01-27T17:08:08Z

done

sympy is a rather slow import (~330ms for me) which contributes nearly half of bezier's import time (~750ms). Delay the import until actually needed. As a typical use case where this is visible: I have a command-line utility which does some computation using bezier. Right now invoking something as simple as `my-program --help` takes over a second to complete (certainly a noticeable delay), in particular due to such slow imports -- even if I do not use the symbolic computation part at all. Obviously there are other ways in which this can be implemented, e.g. by duplicating the `try... except ImportError` in each function, or with another helper function, but given that you already have a requires_sympy decorator, I thought I may as well reuse it for this purpose.

dhermes

LGTM, but I'm feeling like we can ditch require_sympy() altogether. Do you think it's a bad user experience to get an ImportError with no explanation why your dependency is missing?

anntzer · 2020-01-27T21:14:03Z

I actually think an ImportError is nicer than an OSError -- if I have a missing package, I'd rather have the "natural" exception being thrown (especially that with recent Pythons you get an even more accurate ModuleNotFoundError), rather than some wrapper around it.
In any case, I'm happy to implement whichever strategy you prefer.

dhermes · 2020-01-28T00:29:17Z

Good deal! So let's dump the require_sympy() wrapper altogether?

dhermes · 2020-01-28T05:12:32Z

@anntzer I just went ahead and merged as-is, thanks for pointing out this import-time issue. I'll follow up to get rid of the "extra" needed to make things work. I can do the same and obsolete #196 if you don't mind?

Follows along approach from previous commit and earlier work in #194, #195. This supercedes #196. This also fixes a "SympPy" -> "SymPy" typo from the previous commit.

anntzer · 2020-01-28T09:41:05Z

Please do :)
Edit: thanks for taking care of everything.

anntzer force-pushed the delaysympy branch 3 times, most recently from db50cae to 6327459 Compare January 27, 2020 11:23

anntzer mentioned this pull request Jan 27, 2020

Delay import of pkg_resources until needed (if ever). #195

Merged

dhermes reviewed Jan 27, 2020

View reviewed changes

dhermes self-assigned this Jan 27, 2020

dhermes added language: python packaging performance labels Jan 27, 2020

anntzer force-pushed the delaysympy branch from 6327459 to 24f30cf Compare January 27, 2020 17:08

anntzer mentioned this pull request Jan 27, 2020

Delay import of scipy until needed. #196

Closed

anntzer force-pushed the delaysympy branch from 24f30cf to 18d2228 Compare January 27, 2020 17:21

dhermes approved these changes Jan 27, 2020

View reviewed changes

dhermes merged commit fb4d416 into dhermes:master Jan 28, 2020

dhermes added a commit that referenced this pull request Jan 28, 2020

Moving scipy imports to happen at runtime.

38602d8

Follows along approach from previous commit and earlier work in #194, #195. This supercedes #196. This also fixes a "SympPy" -> "SymPy" typo from the previous commit.

anntzer deleted the delaysympy branch January 28, 2020 09:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Delay import of sympy until actually needed. #194

Delay import of sympy until actually needed. #194

anntzer commented Jan 27, 2020 •

edited

Loading

dhermes Jan 27, 2020 •

edited

Loading

dhermes Jan 27, 2020

anntzer Jan 27, 2020

dhermes commented Jan 27, 2020

anntzer commented Jan 27, 2020

dhermes commented Jan 27, 2020 •

edited

Loading

anntzer commented Jan 27, 2020

dhermes commented Jan 27, 2020 •

edited

Loading

anntzer commented Jan 27, 2020

dhermes commented Jan 27, 2020

anntzer commented Jan 27, 2020

dhermes left a comment

anntzer commented Jan 27, 2020 •

edited

Loading

dhermes commented Jan 28, 2020

dhermes commented Jan 28, 2020

anntzer commented Jan 28, 2020 •

edited

Loading

Delay import of sympy until actually needed. #194

Delay import of sympy until actually needed. #194

Conversation

anntzer commented Jan 27, 2020 • edited Loading

dhermes Jan 27, 2020 • edited Loading

Choose a reason for hiding this comment

dhermes Jan 27, 2020

Choose a reason for hiding this comment

anntzer Jan 27, 2020

Choose a reason for hiding this comment

dhermes commented Jan 27, 2020

anntzer commented Jan 27, 2020

dhermes commented Jan 27, 2020 • edited Loading

anntzer commented Jan 27, 2020

dhermes commented Jan 27, 2020 • edited Loading

anntzer commented Jan 27, 2020

dhermes commented Jan 27, 2020

anntzer commented Jan 27, 2020

dhermes left a comment

Choose a reason for hiding this comment

anntzer commented Jan 27, 2020 • edited Loading

dhermes commented Jan 28, 2020

dhermes commented Jan 28, 2020

anntzer commented Jan 28, 2020 • edited Loading

anntzer commented Jan 27, 2020 •

edited

Loading

dhermes Jan 27, 2020 •

edited

Loading

dhermes commented Jan 27, 2020 •

edited

Loading

dhermes commented Jan 27, 2020 •

edited

Loading

anntzer commented Jan 27, 2020 •

edited

Loading

anntzer commented Jan 28, 2020 •

edited

Loading