Name	Name	Last commit message	Last commit date
parent directory ..
.flake8	.flake8
.pylintrc	.pylintrc
README.md	README.md
demo_bash.sh.j2	demo_bash.sh.j2
demo_bookmarks.html.j2	demo_bookmarks.html.j2
demo_bookmarks_data.yaml	demo_bookmarks_data.yaml
demo_doc.j2	demo_doc.j2
demo_doc_data.yaml	demo_doc_data.yaml
demo_names_data.csv	demo_names_data.csv
j2_01.py	j2_01.py
j2_02.py	j2_02.py
j2pp.py	j2pp.py
jinja2-overview.svg	jinja2-overview.svg
kids.j2	kids.j2
names_invalid.yaml	names_invalid.yaml
requirements.txt	requirements.txt

title	author	date	licence
Practical Jinja2: automation, reporting and documents	Gábor Nyers <[email protected]>	2022-11-23	CC BY-NC 4.0 https://creativecommons.org/licenses/by-nc/4.0/

Abstract

Jinja2 is one of the most popular templating engine in Python with a huge scope of applications. It is used by Ansible to automate your infrastructure or by web development frameworks, such as Flask. But it still can do more for us. As it turns out, the ability to automate the creation of parametrized text files can solve a lot of our smaller daily chores: create a shell script to do repeated tasks, create a report based on some data file or parametrized snippets for documents. In this session we'll create a simple Python program which can merge Jinja templates with some data provided in JSON or YAML format.

Introduction

(back to ToC)

Jinja2 is a template engine meant to programmatically create some text based on input data.
1. "programmatically create": create automatically by a program, as opposed to a human using an editor.
2. "some text": any type of text output, e.g.:
  - plain text, e.g.: email
  - markup text, e.g.: HTML, Markdown, etc...
  - documentation in either plain- or markup text
  - program source code, e.g.: bash script
  - configuration files, e.g.: router, switch, Bind zone files, Docker compose files, Kubernetes manifests, etc...
3. "based on input data": data that is available at run-time, e.g.:
  - registration data from a web form
  - dump from an Excel sheet or a database table in CSV
  - table-like data (CSV), containing sales records, hostnames, users, etc..
  - hierarchical data (JSON, YAML), containing dates, accounts, etc...
A template engine is somewhat similar to a compiler in that it takes source code as input and based on instructions in the code transforms it into a modified output.

There are also significant differences as well: a template engine will ignore everything in the input, except the specific and recognizable tokens meant for the engine.

Example Jinja2 template:
```
Hello {{name}}!
```
Output, if name="Jonny":
```
Hello Jonny!
```
Jinja2 has its own (simplified) programming language.
Jinja2 is frequently used for building web applications, mostly in one of the Python based development frameworks such as Flask or Django.

Another well-known project is Ansible, which is using Jinja2 to render (configuration) files and playbooks.

Simple Jinja2 examples

(back to ToC)

This sessions focuses on using Jinja2 for practical purposes, not on learning the Jinja2 templating language. The Template Designer Documentation is an excellent resource to learn more about that.

The following examples are meant to illustrate the basics. The required data is assumed to be loaded from the names.json file.

{
  "kids": {
    "Chris": "Family Guy",
    "Pebbles": "The Flintstones",
    "Bart": "The Simpsons"
  },
  "adults": {
    "Fred": "The Flintstones",
    "Betty": "The Flintstones",
    "Homer": "The Simpsons",
    "Lois": "Family Guy"
  },
  "other": {
    "Klaus": "American Dad",
    "Brian": "Family Guy",
    "Roger": "American Dad"
  }
}

Template

Refer to "Bart" in the show "The Simpsons":

Bart is one of the kids in the show {{ kids['Bart'] }}.

Will render the above Jinja2 template as:

Bart is one of the kids in the show The Simpsons.

Refer to non-existing element:

Lisa is Bart's sister in the show {{ kids['Lisa'] }}.

Will trigger an error:

ERROR: UndefinedError: template variable 'dict object' has no attribute 'Lisa'

Prevent an error in case referring to a non-existing element:
```
Lisa is Bart's sister in {{ kids['Lisa'] | default('a show') }}.
```
Will not trigger any error, because a default value has been provided:
```
Lisa is Bart's sister in a show.
```

Using comments in Jinja2 templates:

{#- Jinja2 comments start and end with the \{# and \#} markers  #}
Lisa is Bart's sister in {{ kids['Lisa'] | default('a show') }}.

Similar to the previous example; notice how the comment is not showing at all:

Lisa is Bart's sister in a show.

Iterate through the group kids:

The list of kids and the show they appear:
{% for character in kids %}
- {{ character }} appears in the show {{ kids[character] }}.
{% endfor %}

Is rendered as (note the empty lines!):


The list of kids and the show they appear:

- Chris appears in the show "Family Guy".

- Pebbles appears in the show "The Flintstones".

- Bart appears in the show "The Simpsons".

Same as the previous example, except manage the empty lines:

The list of kids and the show they appear:
{%- for character in kids %}
- {{ character }} appears in the show "{{ kids[character] }}".
{%- endfor %}

The added dash (-) character (e.g.: {%- in lines 2 and 4) will remove all preceding white space, e.g. the empty line where the Jinja2 code stands:

The list of kids and the show they appear:
- Chris appears in the show "Family Guy".
- Pebbles appears in the show "The Flintstones".
- Bart appears in the show "The Simpsons".

Render Jinja2 from Python

(back to ToC)

Basic usage of Jinja2 in Python

# this file: j2_01.py

import jinja2 as j2     # load Jinja2 module, refer to its content as "j2.*"

template = '''
The list of kids and the show they appear:
{%- for character in kids %}
- {{ character }} appears in the show "{{ kids[character] }}".
{%- endfor %}
'''.strip()                         # remove newlines from begin and end

data = {                            # the data
    "Chris": "Family Guy",
    "Pebbles": "The Flintstones",
    "Bart": "The Simpsons"
}

j2_tmpl = j2.Template(template)     # create a new template object
out = j2_tmpl.render(kids=data)     # render template with give data
print(out)                          # print the rendered text

NOTE:

line 3: will load the Jinja2 module, but instead of the Jinja2, we can use the more convenient j2 prefix.
lines 5 - 10: the definition of the template text
line 10: The .strip() string method will remove the newline characters that are embedded in the begin and end of the template string. These newlines are introduced by the '''\n (line 5) and %}\n (line 9).
lines 12 - 16: Define the data. Jinja2 is capable of handling much more data than in used in this example.
line 18: Create a new instance of the Template class.
line 19:
1. Make the content of the data Python variable (a dict object) available as kids Jinja2 variable
  
  NOTE: Jinja2 is very strict in separating the scopes of the Python program and that of the template.
2. render the tempate.
3. Assign the rendered str object to the (Python) variable out
line 20: Print out the rendered text.

The program output:

$ python3 j2_01.py
The list of kids and the show they appear:
- Chris appears in the show Family Guy.
- Pebbles appears in the show The Flintstones.
- Bart appears in the show The Simpsons.

Load both the template and the data from a file:

This example combines:

the basic Jinja2 example above, and
working with data files from Session 2

# this file: j2_02.py

'''Render a Jinja2 template based on data

Usage:
        python3 j2_02.py TEMPLATEFILE DATAFILE

Where:
TEMPLATEFILE: path to the text file containing a valid Jinja2 teamplte
DATAFILE    : path to a JSON file containing the data
'''

import json
import sys
import jinja2 as j2     # load Jinja2 module, refer to its content as "j2.*"

TEMPLATE = open(sys.argv[1]).read()    # load template from file given as
                                       # 1st CLI argument
TEMPLATE = TEMPLATE.strip()            # remove newlines from begin and end

DATA = json.load(open(sys.argv[2]))    # load data from file given as
                                       # 2nd CLI argument

j2_tmpl = j2.Template(TEMPLATE)        # new Jinja2 template instance
out = j2_tmpl.render(**DATA)           # pass unpacked data (assuming dict)
print(out)

NOTE:

lines 13 and 14: load the required modules:
- sys: in order to access the CLI arguments
- json: in order to load a JSON file
line 17: will execute the following steps:
1. open the file referred to by the 1st CLI argument
2. read its content as str
3. assign the content to the variable TEMPLTE
line 19: remove all leading- and trailing white space, i.e.: space, tab and newline characters.
line 21: will:
1. open the data file referred to by the 2nd CLI argument
2. load its content and interpret it as JSON data

line 25: the **DATA assumes that the loaded data is a dict (as opposed to a list) and unpacks its content as keyword arguments. In the template scope this will result in the following variables:

kids = { "Chris": "Family Guy",
         "Pebbles": "The Flintstones",
         "Bart": "The Simpsons" }

adults = { "Fred": "The Flintstones",
           "Betty": "The Flintstones",
           "Homer": "The Simpsons",
           "Lois": "Family Guy" }

other = { "Klaus": "American Dad",
          "Brian": "Family Guy",
          "Roger": "American Dad" }

Given the following content of the file kids.j2:

The list of kids and the show they appear:
{%- for character in kids %}
- {{ character }} appears in the show "{{ kids[character] }}".
{%- endfor %}

The program output:

$ python3 j2_02.py kids.j2  ../session02/names.json  
The list of kids and the show they appear:
- Chris appears in the show "Family Guy".
- Pebbles appears in the show "The Flintstones".
- Bart appears in the show "The Simpsons".

Generic Jinja2 pre-processor

(back to ToC)

Based on the concepts demonstrate in the previous part, the program j2pp.py has been created. Its purpose is to render a Jinja2 template with the provided data.

Example 1: Usage

$ ./j2pp.py --help
usage: j2pp.py [-h] [-D] [-d DATA_FILE] [-p [name=value [name=value ...]]]
               [-T TEMPLATE_DIRS [TEMPLATE_DIRS ...]] [-o OUTPUT]
               [template]

Jinja2 pre-processor

Render a Jinja2 template using data provided in a JSON, YAML or CSV file and
in the CLI arguments.

positional arguments:
  template              Template file (default: STDIN)

optional arguments:
  -h, --help            show this help message and exit
  -D, --debug           Dump the data that would be passed to the template
  -d DATA_FILE, --data-file DATA_FILE
                        file containing the data, will be passed to template
                        as variable "data"
  -p [name=value [name=value ...]], --params [name=value [name=value ...]]
                        additional parameters, will be passed to template as
                        variable "params", in the form of a dict
  -T TEMPLATE_DIRS [TEMPLATE_DIRS ...], --template-dirs TEMPLATE_DIRS [TEMPLATE_DIRS ...]
                        Template directories (default: the value of env.
                        variable "J2PP_PATH" or "."; now: ['.'])
  -o OUTPUT, --output OUTPUT
                        Write the output to this file (default: STDOUT)

Example 2: Render a simple template with data provided on the CLI

$ echo -en 'Hello {{ name }}!\n\n' | ./j2pp.py --param name=Jonny
Hello Jonny!

Example 3: error handling

The syntax of Jinja2 templates is sometimes complicated. A few examples to show how j2pp.py handles different errors:

Undefined template variable name:

$ echo -en 'Hello {{ name }}!\n\n' | ./j2pp.py --param f_name=Jonny
*** Template ERROR: 'name' is undefined

Jinja2 syntax error in template:

# Superfluous semicolon (;)
$ echo -en 'Hello {{ name; }}!\n\n' | ./j2pp.py --param name=Jonny
*** Template ERROR: expected token 'end of print statement', got ';'

# Missing closing accolade (})
$ echo -en 'Hello {{ name }!\n\n' | ./j2pp.py --param name=Jonny
*** Template ERROR: unexpected '}'

Invalid YAML data format:

$ ./j2pp.py --data  names_invalid.yaml  kids.j2 
expected '<document start>', but found '<block mapping start>'
  in "<unicode string>", line 6, column 1:
    kids:
    ^
File "names.yaml" could not be loaded

Example 4: template and data both from a file

$ ./j2pp.py --data ../session02/names.json  kids.j2 
The list of kids and the show they appear:
- Chris appears in the show "Family Guy".
- Pebbles appears in the show "The Flintstones".
- Bart appears in the show "The Simpsons".

j2pp.py automatically recognizes the supported data formats based on the file's extension. The same data from a YAML file:

adults:
  Betty: The Flintstones
  Fred: The Flintstones
  Homer: The Simpsons
  Lois: Family Guy
kids:
  Bart: The Simpsons
  Chris: Family Guy
  Pebbles: The Flintstones
other:
  Brian: Family Guy
  Klaus: American Dad
  Roger: American Dad

$ ./j2pp.py --data ../session02/names.yaml kids.j2
The list of kids and the show they appear:
- Bart appears in the show "The Simpsons".
- Chris appears in the show "Family Guy".
- Pebbles appears in the show "The Flintstones".

Example 5: dump the data

To verify what data is passed to Jinja use the --debug flag:

$ ./j2pp.py --data ../session02/names.yaml kids.j2 --debug
{'adults': {'Betty': 'The Flintstones',
            'Fred': 'The Flintstones',
            'Homer': 'The Simpsons',
            'Lois': 'Family Guy'},
 'data': {'adults': {'Betty': 'The Flintstones',
                     'Fred': 'The Flintstones',
                     'Homer': 'The Simpsons',
                     'Lois': 'Family Guy'},
          'kids': {'Bart': 'The Simpsons',
                   'Chris': 'Family Guy',
                   'Pebbles': 'The Flintstones'},
          'other': {'Brian': 'Family Guy',
                    'Klaus': 'American Dad',
                    'Roger': 'American Dad'}},
 'data_file': PosixPath('../session02/names.yaml'),
 'debug': True,
 'kids': {'Bart': 'The Simpsons',
          'Chris': 'Family Guy',
          'Pebbles': 'The Flintstones'},
 'now': datetime.datetime(2022, 12, 13, 12, 30, 18, 578173),
 'other': {'Brian': 'Family Guy',
           'Klaus': 'American Dad',
           'Roger': 'American Dad'},
 'output': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>,
 'params': None,
 'template': PosixPath('kids.j2'),
 'template_dirs': ['.']}

Automation examples

(back to ToC)

Ansible's templating module uses Jinja2 as well, so the following examples may seem very familiar.

Bash script

Use case

Create a system administration script to automate some tasks, in this case: create users.

Inputs

JSON file containing users and group information

Jijna2 template to create the users

# Script to create users from data file

# Create the groups
{%- for grp in data           %}
groupadd {{ grp }}
{%-  endfor                   %}

# Create the users
{%  for grp in data           %}
## Creating the users of the group '{{ grp }}':
{%- for user in data[grp]     %}
useradd -g {{ grp }} -c '{{ data[grp][user] }}' {{ user | lower }}
{%- endfor                    %}
{%  endfor                    %}

Generate bash script

$ ./j2pp.py  --data ../session02/names.json  demo_bash.sh.j2

# Script to create users from data file

# Create the groups
groupadd kids
groupadd adults
groupadd other

# Create the users

## Creating the users of the group 'kids':
useradd -g kids -c 'Family Guy' chris
useradd -g kids -c 'The Flintstones' pebbles
useradd -g kids -c 'The Simpsons' bart

## Creating the users of the group 'adults':
useradd -g adults -c 'The Flintstones' fred
useradd -g adults -c 'The Flintstones' betty
useradd -g adults -c 'The Simpsons' homer
useradd -g adults -c 'Family Guy' lois

## Creating the users of the group 'other':
useradd -g other -c 'American Dad' klaus
useradd -g other -c 'Family Guy' brian
useradd -g other -c 'American Dad' roger

Create importable bookmarks

Use case

An easy way to exchange bookmarks between different computers / user accounts.

Inputs

A data file containing the bookmark information, e.g.:

bookmarks:
  - title: ":wave: Music page 1"
    url: https://mydomain.nl/music/1
    tags: [ music, tag 1, tag 3 ]
    shortcut: page1
    icon_uri: https://www.defluit.nl/
  - title: Music page 2
    url: https://mydomain.nl/music/2
  - title: Music page 3
    url: https://mydomain.nl/music/3
  - title: Music page 4
    url: https://mydomain.nl/music/4
  - title: Music page 5
    url: https://mydomain.nl/music/5
  - title: Music page 6
    url: https://mydomain.nl/music/6
  - foldername: Test folder
    content:
    - title: Test page 5
      url: https://mydomain.nl/Test/5
    - title: Test page 6
      url: https://mydomain.nl/Test/6
    - foldername: Sub-sub folder
      content:
      - title: Bla!
        url: https://nu.nl
  - title: Music page 7
    url: https://mydomain.nl/music/7

Jijna2 template containing the logic and syntax to generate an HTML file that can be imported by both Firefox and Chrome as a bookmark collection.

<!--
Usage: 
  - Create importable bookmarks for Firefox,
    with additional bookmark attributes:
       
    ./j2pp.py --param firefox=1 title='Demo bookmarks' \
              --data demo_bookmarks_data.yaml \
              demo_bookmarks.html.j2

  - Create importable bookmarks for Chrome/Chromium:

    ./j2pp.py --param firefox=0 title='Demo bookmarks' \
              --data demo_bookmarks_data.yaml \
              demo_bookmarks.html.j2
-->
...

Generate docs

./j2pp.py --param firefox=1 title='Demo bookmarks' \
          --data demo_bookmarks_data.yaml \
          demo_bookmarks.html.j2

An example importable bookmark HTML is in file demo_bookmarks.html.

Other possible examples

Create a Bind zone file
Automate the management of VLANs
etc...

Create parametrized documentation

Use case

Create documentation (installation or administration instructions) that contains the specific settings but is reusable for different environments.

Inputs

Documentation as markup text (e.g.: Markdown, ReStructuredText, AsciiDoc or perhaps MediaWiki)

This is a Markdown template:

{#- this file: demo_doc.j2             -#}
{%- set env = environments[ envname ]  -%}

# Instruction start the webserver

1. Log in to `{{ env.fqdn_web }}` as user `{{ env.admin }}`.

1. Execute the following:

   ```bash
   systemctl start {{ env.svc_web }}
   ```

date last update: {{ now.strftime('%Y-%m-%d') }}

JSON, YAML or CSV file with the parameters

In this case the following YAML file:

# this file: demo_doc_data.yaml
environments:
  prd:
    fqdn_web: webprd32.prd.example.com
    admin: webmaster
    svc_web: httpd
  tst:
    fqdn_web: web2.tst.example.com
    admin: admin
    svc_web: httpd

Generate docs

For prd environment:

j2pp.py -d demo_doc_data.yaml -p envname=prd demo_doc.j2

The output:

# Instruction start the webserver

1. Log in to `webprd32.prd.example.com` as user `webmaster`.

1. Execute the following:

   ```bash
   systemctl start httpd
   ```

date last update: 2022-12-13

For tst environment:

# Instruction start the webserver

1. Log in to `web2.tst.example.com` as user `admin`.

1. Execute the following:

   ```bash
   systemctl start httpd
   ```


date last update: 2022-12-13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

session11

session11

README.md

Abstract

Agenda

Introduction

Simple Jinja2 examples

Template

Render Jinja2 from Python

Generic Jinja2 pre-processor

Example 1: Usage

Example 2: Render a simple template with data provided on the CLI

Example 3: error handling

Example 4: template and data both from a file

Example 5: dump the data

Automation examples

Bash script

Use case

Inputs

Generate bash script

Create importable bookmarks

Use case

Inputs

Generate docs

Other possible examples

Create parametrized documentation

Use case

Inputs

Generate docs

Files

session11

Directory actions

More options

Directory actions

More options

Latest commit

History

session11

Folders and files

parent directory

README.md

Abstract

Agenda

Introduction

Simple Jinja2 examples

Template

Render Jinja2 from Python

Generic Jinja2 pre-processor

Example 1: Usage

Example 2: Render a simple template with data provided on the CLI

Example 3: error handling

Example 4: template and data both from a file

Example 5: dump the data

Automation examples

Bash script

Use case

Inputs

Generate bash script

Create importable bookmarks

Use case

Inputs

Generate docs

Other possible examples

Create parametrized documentation

Use case

Inputs

Generate docs