Be one step ahead of Ansible errors with dansabel - a Jinja2/YAML linter doubling as a pre-commit git hook!

The name dansabel is a direct Danish translation of "danceable" - it detects out of tune elements in Ansible projects (such as typos or indentation errors) and allows you to adjust the code before the play has begun. The project currently consists of three main components:

• a Python script that will use the YAML parser (ruamel as used in Ansible) to parse your YAML files.
• a simplistic Jinja2 linter using the built-in Jinja2 lexer/parser to attempt to detect typos and errors.
• a git pre-commit hook to launch the other scripts on patches staged for commit to your git repository. (scroll to pre-commit installation guide)

Getting the code

Clone the repository from Semaphor's GitHub page:

 git clone https://github.com/semaphor-dk/dansabel 

Overview and Usage

Contained within the repository is the executable jinjalint.py script, used for linting YAML files.

$ <path-to-repo>/jinjalint.py YAML-FILE

Note that the the linter will also accept a list of .yml files and parse them one by one. More detailed information regarding usage can be found by appending "--help"

$ jinjalint.py --help
usage: jinjalint.py [-h] [-C CONTEXT_LINES] [-q] [-v] [-e] FILE [FILE ...] 

List external variables used from Jinja: jinjalint.py -qe ./*.j2 ./*.yml 

positional arguments: 
 FILE 

optional arguments: 
 -h, --help            show this help message and exit 
 -C CONTEXT_LINES, --context-lines CONTEXT_LINES 
                       Number of context lines controls LAST_THRESHOLD 
 -q, --quiet           No normal output to stdout 
 -v, --verbose         Print verbose output. -v prints all Jinja snippets, 
                       regardless of errors. -vv prints full AST for each 
                       Jinja node. 
 -e, --external        List external variables used. Use -q -e to only print 
                       this summary. Note that -e only works for files that 
                       can be parsed without errors. 

By default the linter will highlight plausible errors when found, but return nothing if no errors are found. This can be demonstrated by executing the script on test cases which are also present in the repository. These are split into good and bad cases with the following structure:

├── bad
│   ├── error-path-reports-inside-vars-not-block.yml
│   ├── fun-stuff.yml
│   ├── multiline-broken.yml
│   ├── namefiledebug.yml
│   ├── templates
│   │   ├── bad-if-scoping.j2
│   │   ├── bad-loop-scoping.j2
│   │   ├── foo.j2
│   │   ├── for-missing-if.j2
│   │   ├── if-elif-eof.j2
│   │   └── if-what.j2
│   └── unknown-filter.yml
└── good
    ├── empty.yml
    ├── for-loop.yml
    ├── inline-dict.yml
    ├── known-filter.yml
    ├── namefiledebug-fixed.yml
    ├── raw.yml
    ├── simple-expansions.yml
    └── templates
        ├── if-elif-endif.j2
        └── if-endif.j2 

The included Makefile can be run to validate that the testcases are lintable by the jinjalint.py script, i.e. not broken - but still syntactically wrong in the bad cases :-)

$ make test 
OK testcases/bad/unknown-filter.yml
OK testcases/bad/templates/bad-loop-scoping.j2
OK testcases/bad/templates/if-elif-eof.j2
OK testcases/bad/templates/if-what.j2
OK testcases/bad/templates/for-missing-if.j2
OK testcases/bad/templates/bad-if-scoping.j2
OK testcases/bad/templates/foo.j2
OK testcases/bad/fun-stuff.yml
OK testcases/bad/namefiledebug.yml
OK testcases/bad/error-path-reports-inside-vars-not-block.yml
OK testcases/bad/multiline-broken.yml
OK testcases/good/templates/if-endif.j2
OK testcases/good/templates/if-elif-endif.j2
OK testcases/good/known-filter.yml
OK testcases/good/simple-expansions.yml
OK testcases/good/inline-dict.yml
OK testcases/good/empty.yml
OK testcases/good/for-loop.yml
OK testcases/good/namefiledebug-fixed.yml
OK testcases/good/raw.yml 

Examples - Correct Syntax

The included testcases are a great place to learn how to interpret the output from the linter. First, let's view the very simple "good" testcase, raw.yml :

root:
  val: '{% raw %} something raw {% endraw %}'

The syntax looks good and as expected, running

$ <path-to-repo>/jinjalint.py raw.yml

will return nothing :



This is normally a good sign, but in the interest of learning how the linter operates let's try again and add the "-v" (--verbose) flag, which returns :



Let's unpack - The leftmost number at the top is the line number, followed by the respective value. The "raw.yml:root.val" describes the file that is being linted and the current key being evaluated. So far so good, now let's demand even more verbosity using the "-vv" flag, which returns :



The output from before is still present, but now the linter unpacks the "root.val" value even further and also includes column offset. The value is separated into multiple tokens: raw_begin, data and raw_end. How a value is unpacked depends on the value itself and what operators are used, but the added verbosity can shine light on potential errors in how values are defined. A slightly more advanced example is the "for-loop.yml" case:

---
myfor: '{% for i in [1,2,3] %}{{ i }}{% endfor %}'

amazing: |
  {% set for = [4,5] %}
  {% for in in for %}
  {{ in }}
  {% endfor %}

In which two values are defined, namely "myfor" and "amazing". Running the linter with "-vv" returns:



While immediately intimidating, the output really is as understandable and organized as the simple example from before. Each value is shown in colors that match the unpacked tokens found below and each value is punctuated by the respective file in which is is located and key that it is stored in. It is worth trying the "-e" (--external) flag as well, which will inform if any external variables are used. As an example, examine the "good" template "if-elif-endif.j2":

{% if True %}
  {{ good }}
{% elif True %}
  {{ also_good }}
{% endif %}

Evidently two external variables are called upon here. Linting with the --external flag returns:

{"if-elif-endif.j2": ["also_good", "good"]}

We are thus informed that the specific template calls upon external variables "good" and "also_good". If the user is only interested in listing external variables without any other output, the "-e" flag can be paired with the "-q" (--quiet) flag.

Examples - Wrong Syntax

We've now some insight into how the linter operates and how to interpret the output, but only when the parsed file is syntactically correct. Let's now explore some bad test cases, consider firstly the bad template "foo.j2":

{% if 1 %}
{{ x }}

Notice how the if statement is incomplete and is missing and "endif". Linting the template with jinjalint.py (without the --verbose flag) reveals:



Even without the call for verbosity the Jinja linter will return an error message if an error is found. In this case the linter warns the user that a code block might not be closed. It even points to the specific line (and column offset) in question and provides more than one solution! A simple solution to this would be to modify:

{% if 1 %}
{{ x }}
{% endif %}

This would make both the linter (and you) happy! Let's see another case, the "bad-loop-scoping.j2":

HEAD
{% for i in [1,2,3] %}
{% if %}
{% endfor %}
FOOT
{{ footer }}

Depending on your Jinja proficiency, it may take a little longer to spot what could be wrong in this template. Running the linter again, the following is returned:

Again, an "if"-statement appears to be the problem. The linter informs both that the if-block may be unclosed, but also that an "endfor" cannot end an "if"-statement. Let's see one final example, the "unknown-filter.yml" case:

---
x: |
  {{ [1,2] | first is strin }}

Though the mistake may be obvious, consider a scenario where this definition is tucked in a wall of code. Running the linter reveals:

The linter will even attempt to match what it thinks you meant to write! Now that's helpful :-) This is achieved by using the get_close_matches() function from the difflib module, which compares what was entered to a library of known words. Hopefully these examples have demonstrated the usefulness of the jinjalint.py script. It is highly encouraged to explore all testcases with varying levels of verbosity and maybe even constructing some test cases yourself to test the ability of the linter as well as your own YAML and Jinja2 scripting skills.

Using dansabel as a pre-commit hook

While manually executing the linter is a great tool for syntax checking Ansible project, dansabel really should be configured as a pre-commit hook. To setup the hook from scratch first install the pre-commit package:

$ pip3 install pre-commit

You can check if pre-commit has been installed by running:

$ pre-commit --version
pre-commit 2.17.0

Then, within the git repo you wish to install the hook, you run the following:

git init
cat > .pre-commit-config.yaml << EOF
repos:
- repo: https://github.com/semaphor-dk/dansabel
  rev: main
  hooks:
  - id: dansabel
EOF
pre-commit autoupdate
pre-commit install --install-hooks

In this way you may prevent faulty playbooks from ever being pushed to your git repository. As an example, consider the case above. You are about to commit this syntactically incorrect YAML file but as you do, the pre-commit hook will lint your file and respond:



This allows you to intercept the error before pushing, potentially saving you wasted time! It should be noted, that the hook can be overruled by running:

 git commit --no-verify

Information on how to set up pre-commit can be found on their website https://pre-commit.com/.

Appendix: Extensible output overview