Cannot import name 'soft_unicode' from 'markupsafe'

[hrak]

Description of the issue:

The generator.py script is broken due to a new release of MarkupSafe that no longer defines soft_unicode.

python3 scripts/generator.py
Traceback (most recent call last):
  File "/opt/ecs/scripts/generator.py", line 7, in <module>
    from generators import asciidoc_fields
  File "/opt/ecs/scripts/generators/asciidoc_fields.py", line 4, in <module>
    import jinja2
  File "/usr/local/lib/python3.11/site-packages/jinja2/__init__.py", line 12, in <module>
    from .environment import Environment
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 25, in <module>
    from .defaults import BLOCK_END_STRING
  File "/usr/local/lib/python3.11/site-packages/jinja2/defaults.py", line 3, in <module>
    from .filters import FILTERS as DEFAULT_FILTERS  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/filters.py", line 13, in <module>
    from markupsafe import soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/usr/local/lib/python3.11/site-packages/markupsafe/__init__.py)

Any additional context or examples:

root@99e6a65bc879:/opt/ecs/scripts# cat requirements.txt
pip
PyYAML==5.4
autopep8==1.4.4
yamllint==1.19.0
mock==4.0.2
gitpython==3.1.2
Jinja2==2.11.3
root@99e6a65bc879:/opt/ecs/scripts# pip list
Package     Version
----------- -------
autopep8    1.4.4
gitdb       4.0.10
GitPython   3.1.2
Jinja2      2.11.3
MarkupSafe  2.1.2
mock        4.0.2
pathspec    0.10.3
pip         22.3.1
pycodestyle 2.10.0
PyYAML      5.4
setuptools  65.5.1
smmap       5.0.0
wheel       0.38.4
yamllint    1.19.0

Found a similar issue here that eventually opted to pin MarkupSafe==2.0.1 to fix this.

[hrak]

Bumping Jinja2 to 3.1.2 fixes this issue.

[ebeahan]

@hrak what version/tag/commit of ECS are you using?

scripts/requirements.txt on main and in 8.6 has different list of dependencies: https://github.com/elastic/ecs/blob/8.6/scripts/requirements.txt

The output above is from an older version before changes made in #1672.

[hrak]

Ah, you are correct, this was using the v1.12.2 tag (we are still on Elasticsearch 7). I will fix this locally and close this issue.