2.11.3 broken due to outdated dependency

[3nprob]

One of our existing applications still on 2.11.3 will no longer run on a fresh install due to use of removed function soft_unicode, which was removed from MarkupSafe v2.1.0.

Could be solved either by:

• Inserting dependency restriction for MarkupSafe<2.10
• Backporting the appropriate set of changes from 148a191 (changing soft_unicode to soft_str)

As flask v1.1.4 (released May 2021) still depends on the jinja 2.x branch, I suspect there are many projects hit by this.

cf pallets/flask#4455

  File "/usr/local/lib/python3.10/site-packages/mypackage/web/web.py", line 26, in <module>
    import flask
  File "/usr/local/lib/python3.10/site-packages/flask/__init__.py", line 14, in <module>
    from jinja2 import escape
  File "/usr/local/lib/python3.10/site-packages/jinja2/__init__.py", line 12, in <module>
    from .environment import Environment
  File "/usr/local/lib/python3.10/site-packages/jinja2/environment.py", line 25, in <module>
    from .defaults import BLOCK_END_STRING
  File "/usr/local/lib/python3.10/site-packages/jinja2/defaults.py", line 3, in <module>
    from .filters import FILTERS as DEFAULT_FILTERS  # noqa: F401
  File "/usr/local/lib/python3.10/site-packages/jinja2/filters.py", line 13, in <module>
    from markupsafe import soft_unicode

Environment:

• Python version: 3.10.2
• Jinja version: 2.11.3

[ThiefMaster]

You need to pin all your dependencies, not just the direct ones... See #1585

[3nprob]

@ThiefMaster Sure - but given that

jinja/setup.py

Line 6 in bb7f3a5

install_requires=["MarkupSafe>=2.0"],

pulls in an incompatible version, wouldn't it be prudent to at least release a jinja 1.1.5 which restricts the version further? It seems like an easy fix that will save hundreds if not thousands of hours of collective engineering time.

Arguably this is an issue of MarkupSafe breaking semver and releasing a breaking change in a minor version bump - but as-is, 2.11.3 will not work independently without changes.

[ThiefMaster]

Besides not really doing semver, we did add deprecation warnings a major release, announcing that the next minor release will remove it.

New projects should not be installing an older version of Jinja (since there's no reason to do that in a new project), and existing projects should have the proper pins to avoid this problem altogether...

Quoting the explanation used in all the other issues on this topic:

You are using an unsupported version of Jinja, please update to the latest version if possible. Additionally, please read https://hynek.me/articles/semver-will-not-save-you/, then use a tool like pip-tools to pin your dependencies and control when you get updates. Be sure to run your tests with deprecation warnings treated as errors so that you get notified of these types of changes early.

[3nprob]

Yeah, I get that and this is no longer an issue for us personally - just thinking it would be so easy to push an update to pypi and especially with the world events this week, individuals and teams who failed to pin properly and are currently fleeing or fighting for their lives would appreciate not having to deal with CI and build breakages.

If this would have happened a month or two ago I probably wouldn't have bothered to follow up on the issue here in the first place.

Not meaning to say "this is your responsibility, fix it", rather "so many would be helped by this"

[3nprob]

Besides not really doing semver

MarkupSafe>=2.0 assumes that a minor version update will not break anything, which is the relevant part of semver.

[ThiefMaster]

That pin indicates that earlier versions are incompatible; it does not make any statement about later versions.

Long but really good article to read: https://iscinumpy.dev/post/bound-version-constraints/