Bookmark this page

Chapter 7.  Transforming Data with Filters and Plug-ins

Abstract

Goal

Populate, manipulate, and manage data in variables using filters and plug-ins.

Objectives
  • Format, parse, and define the values of variables using filters.

  • Populate variables with data from external sources using lookup plug-ins.

  • Implement loops using structures other than simple lists by using lookup plug-ins and filters.

  • Use filters to inspect, validate, and manipulate variables containing networking information.

Sections
  • Processing Variables Using Filters  (and Guided Exercise)

  • Templating External Data Using Lookups  (and Guided Exercise)

  • Implementing Advanced Loops  (and Guided Exercise)

  • Using Filters to Work with Network Addresses  (and Guided Exercise)

Lab
  • Transforming Data with Filters and Plug-ins

Processing Variables Using Filters

Objectives

  • Format, parse, and define the values of variables using filters.

Ansible Filters

Ansible applies variable values to playbooks and templates by using Jinja2 expressions. For example, the following Jinja2 expression replaces the name of the variable enclosed in double braces with its value:

{{ variable }}

Jinja2 expressions also support filters. Filters are used to modify or process the value from the variable that is placed in the playbook or template. Some filters are provided by the Jinja2 language; others are included with Red Hat Ansible Automation Platform as plug-ins. You can also create custom filters, but that is beyond the scope of this course. Filters can be extremely useful for preparing data for use in your playbook or template.

To use a filter in a Jinja2 expression, add a pipe (|) character after the variable name or value and then write the filter or filter expression. You can specify multiple filters in a pipeline, separating each filter or filter expression with a pipe character.

{{ variable | filter }}

Providing Default Variable Values

You can use the default filter to ignore an undefined variable or to provide a value to an undefined variable. Both use cases prevent a playbook from generating an error. The following portion of an error message indicates that a task uses the shell variable, but that the variable is undefined:

FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'shell'

You an use the default filter to prevent this error.

The following generic task uses the ansible.builtin.user module to manage users defined in the user_list variable. Each user in the list must define the name key. Additionally, each user in the list can optionally define the groups, system, shell, state, and remove keys.

Use the default filter with each optional key. Consult the module documentation to identify the keys required by the module and the default values for keys.

- name: Manage user
  ansible.builtin.user:
    name: "{{ item['name'] }}"
    groups: "{{ item['groups'] | default(omit) }}" 1
    system: "{{ item['system'] | default(false) }}" 2
    shell: "{{ item['shell'] | default('/bin/bash') }}" 3
    state: "{{ item['state'] | default('present') }}" 4
    remove: "{{ item['remove'] | default(false) }}" 5
  loop: "{{ user_list }}"

1

If the groups key is not defined for the item variable, then do not generate an error and do not provide a value for the key.

2

If the system key is not defined for the item variable, then use the false Boolean value for the key. Pass a value of true to create a system user.

3

If the shell key is not defined for the item variable, then use the /bin/bash value for the key. You might pass a value of /sbin/nologin if you create a system user.

4

If the state key is not defined for the item variable, then use the present value for the key. Pass a value of false to remove the user.

5

If the remove key is not defined for the item variable, then use the false Boolean value for the key. Passing a value of true when removing a user is the equivalent of the userdel -r command.

Whenever you use a module, you must decide which module keys to use in the task. You can manually specify values for the keys or you can configure the keys to set their values based on variables passed to the task. Because undefined variables produce errors, you can use the default filter to either ignore an undefined variable or supply a value for the undefined variable.

Typically, the default filter only provides a value if a variable is not defined. In some situations, you might want to provide a value if the variable passed to the default filter is an empty string or evaluates to the false Boolean value.

You can do this by adding true to the default filter. Consider the following playbook:

---
- name: Default filter examples
  hosts: localhost
  tasks:
    - name: Default filter examples
      vars:
        pattern: "some text"
      ansible.builtin.debug:
        msg: "{{ item }}"
      loop:
        - "{{ pattern | regex_search('test') | default('MESSAGE') }}" 1
        - "{{ pattern | regex_search('test') | default('MESSAGE', true) }}" 2
        - "{{ pattern | bool | default('MESSAGE') }}" 3
        - "{{ pattern | bool | default('MESSAGE', true) }}" 4

1

Because the regular expression is not found in the variable, the regex_search filter returns an empty string. The default filter is not used.

2

Although the regex_search filter returns an empty string, the default filter is used because it includes true.

3

Because the string evaluates to the false Boolean, the default filter is not used.

4

Although the string evaluates to the false Boolean, the default filter is used because it includes true.

Where appropriate, you can use the default filter before passing a variable to additional filters.

Variable Types

To understand filters, you must first know more about how variable values are handled by Ansible.

Ansible stores runtime data in variables. The YAML structure or the content of the value defines the exact type of data. The following table lists some value types:

TypeDescription
StringsA sequence of characters.
NumbersA numeric value.
BooleansTrue or false values.
DatesISO-8601 calendar date.
NullThe variable becomes undefined.
Lists or ArraysA sorted collection of values.
DictionariesA collection of key-value pairs.

Sometimes, you might first need to convert the value to an integer with the int filter, or to a float with the float filter. For example, the following Jinja2 expression increments the current hour value, which is collected as a fact and stored as a string, not an integer:

{{ ( ansible_facts['date_time']['hour'] | int ) + 1 }}

Various filters are available that can perform mathematical operations on numbers, such as log, pow, root, abs, and round. The root filter, for example, takes the square root of the variable or value.

{{ 1764 | root }}

Manipulating Lists

Many filters are available to analyze and manipulate lists.

If the list consists of numbers, you can use the max, min, or sum filters to find the largest number, the smallest number, or the sum of all list items.

{{ [2, 4, 6, 8, 10, 12] | sum }}

Extracting List Elements

You can obtain information about the contents of lists, such as the first or last elements, or the length of a list:

- name: All three of these assertions are true
  ansible.builtin.assert:
    that:
      - "{{ [ 2, 4, 6, 8, 10, 12 ] | length }} is eq( 6 )"
      - "{{ [ 2, 4, 6, 8, 10, 12 ] | first }} is eq( 2 )"
      - "{{ [ 2, 4, 6, 8, 10, 12 ] | last }} is eq( 12 )"

The random filter returns a random element from the list:

{{ ['Douglas', 'Marvin', 'Arthur'] | random }}

Modifying the Order of List Elements

You can use any of the following methods to reorder a list. The sort filter returns a list that is sorted by the natural order of its elements. The reverse filter returns a list where the order is the opposite of the original order. The shuffle filter returns a list with the same elements, but in a random order.

- name: reversing and sorting lists
  ansible.builtin.assert:
    that:
      - "{{ [ 2, 4, 6, 8, 10 ] | reverse }} is eq( [ 10, 8, 6, 4, 2] )"
      - "{{ [ 4, 8, 10, 6, 2 ] | sort }} is eq( [ 2, 4, 6, 8, 10 ] )"

Merging Lists

Sometimes it is useful to merge several lists into a single list to simplify iteration. The flatten filter recursively takes any inner list in the input list value, and adds the inner values to the outer list.

- name: Flatten turns nested lists on the left to list on the right
  ansible.builtin.assert:
    that:
      - "{{ [ 2, [4, [6, 8]], 10 ] | flatten }} is eq( [ 2, 4, 6, 8, 10] )"

Use the flatten filter to merge list values that come from iterating a parent list.

Operating on Lists as Sets

Use the unique filter to ensure that a list has no duplicate elements. This filter is useful if you are operating on a list of facts that you have collected, such as usernames or hostnames that might have duplicate entries.

- name: The 'unique' filter leaves unique elements
  ansible.builtin.assert:
    that:
      - "{{ [ 1, 1, 2, 2, 2, 3, 4, 4 ] | unique }} is eq( [ 1, 2, 3, 4 ] )"

If two lists have no duplicate elements, then you can use set theory operations on them.

  • The union filter returns a set with elements from both input sets.

  • The intersect filter returns a set with elements common to both sets.

  • The difference filter returns a set with elements from the first set that are not present in the second set.

- name: The 'difference' filter provides elements not in specified set
  ansible.builtin.assert:
    that:
      - "{{ [2, 4, 6, 8, 10] | difference([2, 4, 6, 16]) }} is eq( [8, 10] )"

Manipulating Dictionaries

Unlike lists, dictionaries are not ordered in any way, but rather are just a collection of key-value pairs. You can use filters to construct dictionaries and you can convert those dictionaries into lists, and vice versa.

Joining Dictionaries

Use the combine filter to join two dictionaries. Entries from the second dictionary have higher priority than entries from the first dictionary, as seen in the following task:

- name: The 'combine' filter combines two dictionaries into one
  vars:
    expected:
      A: 1
      B: 4
      C: 5
  ansible.builtin.assert:
    that:
      - "{{ {'A':1,'B':2} | combine({'B':4,'C':5}) }} is eq( expected )"

Converting Dictionaries

Use the dict2items filter to convert a dictionary to a list. Use the items2dict filter to convert a list to a dictionary.

- name: converting between dictionaries and lists
  vars:
    characters_dict:
      Douglas: Human
      Marvin: Robot
      Arthur: Human
    characters_items:
      - key: Douglas
        value: Human
      - key: Marvin
        value: Robot
      - key: Arthur
        value: Human
  ansible.builtin.assert:
    that:
      - "{{ characters_dict | dict2items }} is eq( characters_items )"
      - "{{ characters_items | items2dict }} is eq( characters_dict )"

Hashing, Encoding, and Manipulating Strings

Various filters are available to manipulate the text of a value. You can compute checksums, create password hashes, and convert text to and from Base64 encoding, as used by a number of applications.

Hashing Strings and Passwords

The hash filter returns the hash value of the input string, using the provided hashing algorithm:

- name: the string's SHA-1 hash
  vars:
    expected: '8bae3f7d0a461488ced07b3e10ab80d018eb1d8c'
  ansible.builtin.assert:
    that:
      - "'{{ 'Arthur' | hash('sha1') }}' is eq( expected )"

Use the password_hash filter to generate password hashes:

{{ 'secret_password' | password_hash('sha512') }}

Encoding Strings

Use the b64encode filter to translate binary data to Base64, or translate Base64 encoded data back to binary data with the b64decode filter:

- name: Base64 encoding and decoding of values
  ansible.builtin.assert:
    that:
      - "'{{ 'âÉïôú' | b64encode }}' is eq( 'w6LDicOvw7TDug==' )"
      - "'{{ 'w6LDicOvw7TDug==' | b64decode }}' is eq( 'âÉïôú' )"

Before sending strings to the underlying shell, and to avoid parsing or code injection issues, it is a good practice to sanitize the string by using the quote filter:

- name: Put quotes around 'my_string'
  shell: echo {{ my_string | quote }}

Formatting Text

Use the lower, upper, or capitalize filters to enforce the case of an input string:

- name: Change case of characters
  ansible.builtin.assert:
    that:
      - "'{{ 'Marvin' | lower }}' is eq( 'marvin' )"
      - "'{{ 'Marvin' | upper }}' is eq( 'MARVIN' )"
      - "'{{ 'marvin' | capitalize }}' is eq( 'Marvin' )"

Replacing Text

Use the replace filter to replace all occurrences of a substring inside the input string:

- name: Replace 'ar' with asterisks
  ansible.builtin.assert:
    that:
      - "'{{ 'marvin, arthur' | replace('ar','**') }}' is eq( 'm**vin, **thur' )"

Use the regex_search and regex_replace filters to perform more complex searches and replacements by using regular expressions.

- name: Test results of regex search and search-and-replace
  ansible.builtin.assert:
    that:
      - "'{{ 'marvin, arthur' | regex_search('ar\S*r') }}' is eq( 'arthur' )"
      - "'{{ 'arthur up' | regex_replace('ar(\S*)r','\\1mb') }}' is eq( 'thumb up' )"

Manipulating Data Structures

Many data structures used by Ansible are in JSON format. JSON and YAML are closely related, and Ansible data structures can be processed as JSON. Likewise, many APIs that Ansible Playbooks might interact with consume or provide information in JSON format. Because this format is widely used, JSON filters are particularly useful.

Data Structure Queries

You can combine selectattr with the map filter to extract information from Ansible data structures.

Use the selectattr filter to select a sequence of objects based on attributes of the objects in the list. Use the map filter to turn a list of dictionaries into a simple list based on a given attribute.

Note

Although the community.general collection provides the json_query filter, you can usually achieve the same functionality using the selectattr and map filters. Red Hat does not support the community.general collection.

Consider the following playbook, which queries the /api/v2/execution_environments/ API endpoint and displays the ID for the Control Plane Execution Environment automation controller resource:

---
- name: Query automation controller execution environments
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Query EEs
      vars:
        username_password: "admin:redhat"
      ansible.builtin.uri:
        url: https://controller.lab.example.com/api/v2/execution_environments/
        method: GET
        headers:
          Authorization: Basic {{ username_password | string | b64encode }}
        validate_certs: false
      register: query_results

    - name: Show execution environment ID
      ansible.builtin.debug:
        msg: "{{ query_results['json']['results'] | selectattr('name', '==', 'Control Plane Execution Environment') | map(attribute='id') | first }}"

The query_results['json']['results'] variable is a list containing entries for each execution environment resource. Because each entry defines a name key, you can select the entry for the Control Plane Execution Environment resource by using the selectattr filter. After selecting the correct entry, you can use the map filter to display the value of any key in that entry, such as the value of the id key. Because the selectattr filter combined with the map filter frequently creates a list consisting of one item, you can use the first filter to select that item.

The playbook might produce the following output to indicate that the ID for the Control Plane Execution Environment resource is 4:

PLAY [Query automation controller EEs] *****************************************

TASK [Query projects] **********************************************************
ok: [localhost]

TASK [Show execution environment ID] *******************************************
ok: [localhost] => {
    "msg": "4"
}

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0  ...

Similarly, you might list all the execution environment names by sending the query_results['json']['results'] variable to the map filter:

    - name: Show execution environment names
      ansible.builtin.debug:
        msg: "{{ query_results['json']['results'] | map(attribute='name') }}"

Adding this task to the previous playbook generates this additional output:

TASK [Show execution environment names] ****************************************
ok: [localhost] => {
    "msg": [
        "Control Plane Execution Environment",
        "Automation Hub Default execution environment",
        "Automation Hub Ansible Engine 2.9 execution environment",
        "Automation Hub Minimal execution environment"
    ]
}

You can use the selectattr and map filters with both JSON and YAML data structures.

Parsing and Encoding Data Structures

Transforming data structures to and from text is useful for debugging and communication. Data structures serialize to JSON or YAML format with the to_json and to_yaml filters. Use to_nice_json and to_nice_yaml filters to obtain a formatted, human-readable output.

- name: Convert between JSON and YAML format
  vars:
    hosts:
      - name: bastion
        ip:
          - 172.25.250.254
          - 172.25.252.1
    hosts_json: '[{"name": "bastion", "ip": ["172.25.250.254", "172.25.252.1"]}]'
  ansible.builtin.assert:
    that:
      - "'{{ hosts | to_json }}' is eq( hosts_json )"

Other filters are introduced in the following sections and chapters that are suited to specific scenarios. Review the official Ansible and Jinja2 documentation to discover more useful filters to meet your needs.

Revision: do374-2.2-82dc0d7