Full read-through of the Jinja documentation

[simonw]

I use Jinja enough that I should really do a full read-through of the docs to see what I've missed.

https://jinja.palletsprojects.com/en/3.1.x/

[simonw]

https://jinja.palletsprojects.com/en/3.1.x/intro/ says:

A sandboxed environment can safely render untrusted templates.

I didn't know Jinja had sandboxing!

[simonw]

Templates are compiled to optimized Python code just-in-time and cached, or can be compiled ahead-of-time.

Maybe Datasette should ship compiled-ahead-of-time templates?

[simonw]

https://jinja.palletsprojects.com/en/3.1.x/api/

Jinja uses a central object called the template Environment. Instances of this class are used to store the configuration and global objects, and are used to load templates from the file system or other locations. Even if you are creating templates from strings by using the constructor of Template class, an environment is created automatically for you, albeit a shared one.

[simonw]

from jinja2 import Environment, PackageLoader, select_autoescape
env = Environment(
    loader=PackageLoader("yourapp"),
    autoescape=select_autoescape()
)

This will create a template environment with a loader that looks up templates in the templates folder inside the yourapp Python package (or next to the yourapp.py Python module)

[simonw]

In future versions of Jinja we might enable autoescaping by default for security reasons. As such you are encouraged to explicitly configure autoescaping now instead of relying on the default.

That would be good!

[simonw]

Environment has a bunch of options I'd not seen before - trim_blocks and lstrip_blocks and suchlike.

extensions: List of Jinja extensions to use. This can either be import paths as strings or extension classes

[simonw]

autoescape: As of Jinja 2.4 this can also be a callable that is passed the template name and has to return True or False depending on autoescape should be enabled by default.

I guess that could be a function that returns True for .html files and False for .txt files.

Turns out the select_autoescape() function implements that pattern for you: https://github.com/pallets/jinja/blob/e740cc65d5c54fbebb0f3483add794ec2b47187f/src/jinja2/utils.py#L570-L623

[simonw]

JInja cache size defaults to 400 - that's 400 templates that will have their compiled versions cached. Increased from 50 in Jinja 2.8.

[simonw]

You can customize code_generator_class and context_class though "This should not be changed in most cases".

[simonw]

.overlay(options) is interesting - lets you create a new overlay environment that adds its own customizations: https://jinja.palletsprojects.com/en/3.1.x/api/#jinja2.Environment.overlay

[simonw]

You can return undefined objects with tips to help people debug what went wrong:

if not hasattr(obj, 'attr'):
    return environment.undefined(obj=obj, name='attr', hint='some hint message here')

[simonw]

You can reuse the Jinja expression language like this:

>>> env = Environment()
>>> expr = env.compile_expression('foo == 42')
>>> expr(foo=23)
False
>>> expr(foo=42)
True

Could be interesting to combine this with sandboxing.

[simonw]

.compile_templates() can be used to find all available templates and compile them and put the compiled code optionally in a zip file! https://jinja.palletsprojects.com/en/3.1.x/api/#jinja2.Environment.compile_templates

[simonw]

Both of these work the same:

template.render(knights='that say nih')
template.render({'knights': 'that say nih'})

[simonw]

template.generate() is interesting:

For very large templates it can be useful to not render the whole template at once but evaluate each statement after another and yield piece for piece. This method basically does exactly that and returns a generator that yields one item after another as strings.

Same arguments as render().

https://jinja.palletsprojects.com/en/3.1.x/api/#jinja2.Template.generate

And .stream() "Works exactly like generate() but returns a TemplateStream."

There's also a generate_async(context) equivalent for render_async(context).

[simonw]

>>> t = Template('{% macro foo() %}42{% endmacro %}23')
>>> str(t.module)
'23'
>>> t.module.foo() == u'42'
True

Neat trick for exposing template variables to Python!

[simonw]

TemplateStream:

A template stream works pretty much like an ordinary python generator but it can buffer multiple items to reduce the number of total iterations. Per default the output is unbuffered which means that for every unbuffered instruction in the template one string is yielded.

If buffering is enabled with a buffer size of 5, five items are combined into a new string. This is mainly useful if you are streaming big templates to a client via WSGI which flushes after each iteration.

[simonw]

. is valid in some names!

Filters and tests may contain dots to group filters and tests by topic. For example it’s perfectly valid to add a function into the filter dict and call it to.str

[simonw]

https://jinja.palletsprojects.com/en/3.1.x/api/#the-context

Template filters and global functions marked as pass_context() get the active context passed as first argument and are allowed to access the context read-only.

[simonw]

Filed this:

• Review if uptodatefunc returning True is correct datasette-edit-templates#4

[simonw]

jinja2.MemcachedBytecodeCache - turns out Jinja can cache bytecode in memcached! https://jinja.palletsprojects.com/en/3.1.x/api/#jinja2.MemcachedBytecodeCache

[simonw]

Policies are a bit weird, they're basically a bunch of additional settings for different things: https://jinja.palletsprojects.com/en/3.1.x/api/#policies

e.g. urlize.target and json.dumps_function.

[simonw]

Custom filters: https://jinja.palletsprojects.com/en/3.1.x/api/#custom-filters

def datetime_format(value, format="%H:%M %d-%m-%y"):
    return value.strftime(format)

environment.filters["datetime_format"] = datetime_format

Then:

{{ article.pub_date|datetimeformat }}
{{ article.pub_date|datetimeformat("%B %Y") }}

[simonw]

Tests are a cute feature I didn't know about: https://jinja.palletsprojects.com/en/3.1.x/api/#custom-tests

For example, the test {{ 42 is even }} is called behind the scenes as is_even(42).

[simonw]

Environment.globals are intended for data that is common to all templates loaded by that environment. Template.globals are intended for data that is common to all renders of that template, and default to Environment.globals unless they're given in Environment.get_template(), etc.

OK, so I should mainly use environment globals then.

But:

Environment globals should not be changed after loading any templates

[simonw]

https://jinja.palletsprojects.com/en/3.1.x/api/#low-level-api looks fun:

Environment.lex(*source*, *name=None*, *filename=None*)

Lex the given sourcecode and return a generator that yields tokens as tuples in the form (lineno, token_type, value).

[simonw]

https://jinja.palletsprojects.com/en/3.1.x/api/#the-meta-api has two useful things:

• jinja2.meta.find_undeclared_variables(ast) finds all variables that need to be in the context
• jinja2.meta.find_referenced_templates(ast) lists all imported or included templates

Could have fun with that second one building a debug tool of some sort, maybe even a dot graph visualization.

[simonw]

https://jinja.palletsprojects.com/en/3.1.x/sandbox/ Sandbox actually looks quite good - it's designed to let users craft custom emails for example.

It does have one BIG problem though:

It is possible to construct a relatively small template that renders to a very large amount of output, which could correspond to a high use of CPU or memory. You should run your application with limits on resources such as CPU and memory to mitigate this.

Might still work for Datasette Cloud though, since the only thing users will be hurting if they write a bad template is their own instance.

[simonw]

https://jinja.palletsprojects.com/en/3.1.x/nativetypes/ is interesting - it's designed for non-string-template use cases like config file parsing:

>>> env = NativeEnvironment()
>>> t = env.from_string('{{ x + y }}')
>>> result = t.render(x=4, y=2)
>>> print(result)
6
>>> print(type(result))
int

[simonw]

Import is interesting for macros: https://jinja.palletsprojects.com/en/3.1.x/templates/#import

{% import 'forms.html' as forms %}
<dl>
    <dt>Username</dt>
    <dd>{{ forms.input('username') }}</dd>
    <dt>Password</dt>
    <dd>{{ forms.input('password', type='password') }}</dd>
</dl>
<p>{{ forms.textarea('comment') }}</p>

OR

{% from 'forms.html' import input as input_field, textarea %}

Macros and variables starting with one or more underscores are private and cannot be imported.

[simonw]

Lists, tuples and dictionaries are valid Jinja literals.

[simonw]

Bit surprising (an old bug I think):

Unlike Python, chained pow is evaluated left to right. {{ 3**3**3 }} is evaluated as (3**3)**3 in Jinja, but would be evaluated as 3**(3**3) in Python. Use parentheses in Jinja to be explicit about what order you want.

[simonw]

~ "Converts all operands into strings and concatenates them."

{{ "Hello " ~ name ~ "!" }}

[simonw]

If expressions are neat: https://jinja.palletsprojects.com/en/3.1.x/templates/#if-expression

{% extends layout_template if layout_template is defined else 'default.html' %}
{{ "[{}]".format(page.title) if page.title }}

[simonw]

Lots of built-in filters: https://jinja.palletsprojects.com/en/3.1.x/templates/#list-of-builtin-filters

[simonw]

I like batch:

<table>
{%- for row in items|batch(3, '&nbsp;') %}
  <tr>
  {%- for column in row %}
    <td>{{ column }}</td>
  {%- endfor %}
  </tr>
{%- endfor %}
</table>

That second optional argument is used to fill in missing items.

[simonw]

There's a |filesizeformat filter!

I didn't need https://github.com/simonw/datasette/blob/5aa359b86907d11b3ee601510775a85a90224da8/datasette/utils/__init__.py#L849-L858 after all.

[simonw]

• Drop format_bytes for Jinja filesizeformat filter datasette#1841

[simonw]

Users on this page: {{ users|map(attribute='username')|join(', ') }}
Users on this page: {{ titles|map('lower')|join(', ') }}

[simonw]

|pprint looks useful.

These test ones are cute:

{{ numbers|reject("odd") }}
{{ numbers|select("divisibleby", 3) }}
{{ numbers|select("lessthan", 42) }}

[simonw]

I wonder how good |striptags is?

Looks like it uses markupsafe under the hood, implemented here: https://github.com/pallets/markupsafe/blob/33307792f69e9e6bd9589919620300985955800c/src/markupsafe/__init__.py#L155-L166

Which uses these two regexes:

_strip_comments_re = re.compile(r"<!--.*?-->", re.DOTALL)
_strip_tags_re = re.compile(r"<.*?>", re.DOTALL)

[simonw]

The |tojson filter says: https://jinja.palletsprojects.com/en/3.1.x/templates/#jinja-filters.tojson

The returned string is safe to render in HTML documents and <script> tags. The exception is in HTML attributes that are double quoted; either use single quotes or the |forceescape filter.

Implementation: https://github.com/pallets/jinja/blob/52843b5cbf635b37a82ac0b6c901921a8ee076ff/src/jinja2/utils.py#L657-L663

    return markupsafe.Markup(
        dumps(obj, **kwargs)
        .replace("<", "\\u003c")
        .replace(">", "\\u003e")
        .replace("&", "\\u0026")
        .replace("'", "\\u0027")
    )

[simonw]

|urlencode is handy:

Basic wrapper around urllib.parse.quote() when given a string, or urllib.parse.urlencode() for a dict or iterable.

[simonw]

You can test for if a filter is available or not:

{% if 'markdown' is filter %}
    {{ value | markdown }}
{% else %}
    {{ value }}
{% endif %}

[simonw]

It has lorem ipsum!

jinja-globals.lipsum(n=5, html=True, min=20, max=100)

Generates some lorem ipsum for the template. By default, five paragraphs of HTML are generated with each paragraph between 20 and 100 words. If html is False, regular text is returned. This is useful to generate simple contents for layout testing.

[simonw]

https://jinja.palletsprojects.com/en/3.1.x/templates/#i18n looks very handy:

{% trans %}Hello, {{ user }}!{% endtrans %}
{% trans user=user.username %}Hello, {{ user }}!{% endtrans %}

{% trans book_title=book.title, author=author.name %}
This is {{ book_title }} by {{ author }}
{% endtrans %}

{% trans count=list|length %}
There is {{ count }} {{ name }} object.
{% pluralize %}
There are {{ count }} {{ name }} objects.
{% endtrans %}

When translating blocks of text, whitespace and linebreaks result in hard to read and error-prone translation strings. To avoid this, a trans block can be marked as trimmed, which will replace all linebreaks and the whitespace surrounding them with a single space and remove leading and trailing whitespace.

{% trans trimmed book_title=book.title %}
    This is {{ book_title }}.
    You should read it!
{% endtrans %}

{{ _("Hello, World!") }}

{{ _("Hello, %(user)s!")|format(user=user.username) }}

[simonw]

Interesting extension: https://jinja.palletsprojects.com/en/3.1.x/templates/#expression-statement

{% do navigation.append('a string') %}

[simonw]

<pre>{% debug %}</pre>

Works if you turn on the extension: https://jinja.palletsprojects.com/en/3.1.x/templates/#debug-statement

[simonw]

This used to be an extension but is now built in:

{% with %}
    {% set foo = 42 %}
    {{ foo }}           foo is 42 here
{% endwith %}
foo is not visible here any longer

This too:

{% autoescape true %}
    Autoescaping is active within this block
{% endautoescape %}

{% autoescape false %}
    Autoescaping is inactive within this block
{% endautoescape %}

[simonw]

Extensions: https://jinja.palletsprojects.com/en/3.1.x/extensions/

Mainly i18n - the others are pretty tiny. You can write your own.

[simonw]

https://jinja.palletsprojects.com/en/3.1.x/extensions/#module-jinja2.ext

By writing extensions you can add custom tags to Jinja. This is a non-trivial task and usually not needed as the default tags and expressions cover all common use cases. The i18n extension is a good example of why extensions are useful. Another one would be fragment caching.

Extensions are pretty low-level AST and parser code.

[simonw]

https://jinja.palletsprojects.com/en/3.1.x/integration/#babel describes Babel integration, for extracting translatable strings.

[simonw]

https://jinja.palletsprojects.com/en/3.1.x/switching/ has some short notes on Django template comparisons.

[simonw]

https://jinja.palletsprojects.com/en/3.1.x/faq/ - only 3 FAQs there.

And I'm done!