Attempt to configure devices, detect errors, and automatically recover from errors by restoring the original device state and verifying that it has been restored.
Outcomes
Modify a playbook that configures the Network Time Protocol (NTP) on Juniper Junos and Cisco IOS managed nodes.
Use blocks to handle the errors found in the NTP managed nodes' settings.
Specify failure conditions in the block tasks.
Use the rescue section of the block to restore settings on managed nodes that contain an erroneous configuration, and use the always section to verify that the initial configuration was restored.
As the student user on the workstation machine, use the lab command to prepare your system for this exercise, and to ensure that all required resources are available.
This command also creates a project directory with the files needed for the exercise.
[student@workstation ~]$ lab start task-failure
Instructions
The lab command provides the playbooks necessary to complete this exercise.
You must ensure that the correct NTP servers are configured on the managed nodes specified in the inventory file.
The following table lists the NTP servers that are present in the configuration on some managed nodes:
Table 5.2. Existing NTP Servers
| NTP server | Description |
|---|---|
172.25.23.240
| First NTP server |
172.25.23.230
| Second NTP server |
The following table lists the NTP servers that should be configured on the managed nodes. These should be the only NTP servers configured, and any other NTP servers should be removed from the configuration.
Table 5.3. New NTP Servers
| NTP server | Description |
|---|---|
172.25.254.254
| First NTP server |
172.25.250.220
| Second NTP server |
Open the /home/student/task-failure directory in VS Code and perform the following tasks:
Review the inventory file.
Review the ntp_configuration.yml playbook file and identify the tasks that configure NTP on your managed nodes.
Locate the variables that define the NTP servers.
Update the values of the NTP server variables to use the IP addresses for the new NTP servers.
Open VS Code and then click → .
Navigate to → and then click .
If prompted, select , and then click .
Click the inventory file to open it.
Note that there are two groups of managed nodes in your inventory.
The junos group corresponds to the Juniper Junos managed nodes.
The ios group corresponds to the Cisco IOS managed nodes.
[junos] junos[1:2].lab.example.com [ios] iosxe[1:2].lab.example.com
Click the ntp_configuration.yml playbook file to open it.
Identify the tasks that configure NTP on your managed nodes.
Notice that the playbook runs against all the managed nodes in the inventory using conditions to establish when to run a task on a managed node.
...output omitted... - name: Configure NTP on Juniper Junos managed nodes junipernetworks.junos.junos_ntp_global: config: servers: - server: "{{ ntp_server_1 }}" - server: "{{ ntp_server_2 }}" state: mergedwhen: ansible_network_os == "junipernetworks.junos.junos"- name: Configure NTP on IOS managed nodes cisco.ios.ios_ntp_global: config: servers: - server: "{{ ntp_server_1 }}" - server: "{{ ntp_server_2 }}" state: mergedwhen: ansible_network_os == "cisco.ios.ios"...output omitted...
The group_vars/all/ntp.yml file has the variables used in the playbook to define the NTP servers.
Edit the group_vars/all/ntp.yml file and change the values of the NTP servers to the IP addresses from the "New NTP Servers" table:
--- ntp_server_1: 172.25.254.254 ntp_server_2: 172.25.250.220
Click the ntp_configuration.yml playbook file and analyze the tasks that create backups of the managed node configurations.
In case something goes wrong, you have recovery backups.
View the task that backs up the Juniper Junos managed nodes:
...output omitted... - name: Enable the NETCONF service vars: ansible_connection: ansible.netcommon.network_cli junipernetworks.junos.junos_netconf: listens_on: 830 state: present when: ansible_network_os == "junipernetworks.junos.junos"- name: Back up the Juniper Junos configurationsjunipernetworks.junos.junos_config:backup: truebackup_options:filename: "{{ inventory_hostname }}.txt"when: ansible_network_os == "junipernetworks.junos.junos"...output omitted...
View the task that backs up the IOS managed nodes:
...output omitted... - name: Back up the Juniper Junos configurations junipernetworks.junos.junos_config: backup: true backup_options: filename: "{{ inventory_hostname }}.txt" when: ansible_network_os == "junipernetworks.junos.junos"- name: Back up the IOS configurationscisco.ios.ios_config:backup: truebackup_options:filename: "{{ inventory_hostname }}.txt"when: ansible_network_os == "cisco.ios.ios"...output omitted...
By default, a directory named backup is created and the configuration backups are stored in that directory.
Run the playbooks/backup_configs.yml playbook to create an initial backup of the managed node configurations.
Switch to the tab in VS Code, or change to the /home/student/task-failure directory in a GNOME terminal, and then run the playbooks/backup_configs.yml playbook:
[student@workstation task-failure]$ansible-navigator run \playbooks/backup_configs.yml...output omitted...
Each time you run the ntp_configuration.yml playbook, it makes a backup of the configuration found on the managed nodes.
Because the playbook changes the NTP configuration, when you run the playbook again the backup is overwritten with the new configuration backup.
This behavior is desired for the final version of your playbook. During its development, however, you must keep a copy of the first backup of your managed nodes.
Copy the first backup of your managed nodes from the backup directory into the current project directory:
[student@workstation task-failure]$ cp -v playbooks/backup/*.txt .
'playbooks/backup/iosxe1.lab.example.com.txt' -> './iosxe1.lab.example.com.txt'
'playbooks/backup/iosxe2.lab.example.com.txt' -> './iosxe2.lab.example.com.txt'
'playbooks/backup/junos1.lab.example.com.txt' -> './junos1.lab.example.com.txt'
'playbooks/backup/junos2.lab.example.com.txt' -> './junos2.lab.example.com.txt'Run the ntp_configuration.yml playbook to back up the configurations and configure NTP on the managed nodes.
After you run the ntp_configuration.yml playbook, run the playbooks/junos_ntp_verify.yml and playbooks/ios_ntp_verify.yml playbooks to verify the NTP configuration on the managed nodes.
Run the ntp_configuration.yml playbook by using the ansible-navigator run command.
The playbook runs successfully, first creating a backup of the managed nodes and then configuring the managed nodes to use the new NTP servers:
[student@workstation task-failure]$ansible-navigator run \ntp_configuration.ymlPLAY [Configure NTP on managed nodes]******************************************** TASK [Enable the NETCONF service] ************************************************ skipping: [iosxe1.lab.example.com] skipping: [iosxe2.lab.example.com] ok: [junos1.lab.example.com] ok: [junos2.lab.example.com]TASK [Back up the Juniper Junos configurations]********************************** skipping: [iosxe1.lab.example.com] skipping: [iosxe2.lab.example.com]changed: [junos2.lab.example.com]changed: [junos1.lab.example.com]TASK [Back up the IOS configurations]******************************************** skipping: [junos1.lab.example.com] skipping: [junos2.lab.example.com]changed: [iosxe2.lab.example.com]changed: [iosxe1.lab.example.com]TASK [Configure NTP on Juniper Junos managed nodes]****************************** skipping: [iosxe1.lab.example.com] skipping: [iosxe2.lab.example.com]changed: [junos1.lab.example.com]changed: [junos2.lab.example.com]TASK [Configure NTP on IOS managed nodes]**************************************** skipping: [junos1.lab.example.com] skipping: [junos2.lab.example.com]changed: [iosxe2.lab.example.com]changed: [iosxe1.lab.example.com]PLAY RECAP *********************************************************************** iosxe1.lab.example.com : ok=2 changed=2 unreachable=0 failed=0 ... iosxe2.lab.example.com : ok=2 changed=2 unreachable=0 failed=0 ... junos1.lab.example.com : ok=3 changed=2 unreachable=0 failed=0 ... junos2.lab.example.com : ok=3 changed=2 unreachable=0 failed=0 ...
Run the playbooks/junos_ntp_verify.yml playbook to verify the current NTP configuration on the Juniper Junos managed nodes.
Note that the resulting verification is not what was expected.
The junos1.lab.example.com managed node does not have the desired configuration:
[student@workstation task-failure]$ansible-navigator run \playbooks/junos_ntp_verify.yml...output omitted... TASK [Display NTP configuration] ************************************************* ok: [junos1.lab.example.com] => { "junos_ntp['stdout_lines'][0]": ["server 172.25.23.240;","server 172.25.23.230;","server 172.25.254.254;","server 172.25.250.220;"] } ok: [junos2.lab.example.com] => { "junos_ntp['stdout_lines'][0]": [ "server 172.25.254.254;", "server 172.25.250.220;" ] } ...output omitted...
If the ntp_configuration.yml playbook runs on managed nodes without any NTP server configuration, then the managed nodes are configured as expected.
If the ntp_configuration.yml playbook runs on managed nodes with an existing NTP server configuration, then the new NTP servers are added to the previous NTP server configuration.
Run the playbooks/ios_ntp_verify.yml playbook to verify the current NTP configuration on the IOS managed nodes.
As with the junos1.lab.example.com managed node, the NTP configuration of the iosxe2.lab.example.com managed node does not have the desired configuration:
[student@workstation task-failure]$ansible-navigator run \playbooks/ios_ntp_verify.yml...output omitted... TASK [Display NTP configuration] ************************************************* ok: [iosxe1.lab.example.com] => { "ios_ntp['stdout_lines'][0]": [ "ntp server 172.25.254.254", "ntp server 172.25.250.220" ] } ok: [iosxe2.lab.example.com] => { "ios_ntp['stdout_lines'][0]": ["ntp server 172.25.23.240","ntp server 172.25.23.230","ntp server 172.25.254.254","ntp server 172.25.250.220"] } ...output omitted...
Restore the initial configuration of your managed nodes using the backups that you previously created.
The junos_recover.yml and ios_recover.yml playbooks in the playbooks directory enable you to restore the backups to your managed nodes.
Run the playbooks/junos_recover.yml playbook to restore the Juniper Junos backup configuration:
[student@workstation task-failure]$ansible-navigator run \playbooks/junos_recover.yml...output omitted...
Run the playbooks/ios_recover.yml playbook to restore the Cisco IOS backup configuration:
[student@workstation task-failure]$ansible-navigator run \playbooks/ios_recover.yml...output omitted...
Create another playbook by copying the ntp_configuration.yml file to the ntp_configuration_blocks.yml file.
Modify the new ntp_configuration_blocks.yml playbook to use Ansible blocks by performing the following tasks:
Create a block section that nests the task that configures NTP on the Juniper Junos managed nodes.
Create another block section that nests the task that configures the NTP service on the IOS managed nodes.
Add a task to each of the blocks to verify that the previous NTP servers are no longer configured on the managed nodes.
Use the failed_when failure condition for this purpose.
As a good practice, name the blocks.
For each block, move the condition for the ansible_network_os variable to apply to all the tasks in the block.
For your convenience, you can find the already completed ntp_configuration_blocks.yml playbook file under the playbooks directory.
Copy the ntp_configuration.yml file to the ntp_configuration_blocks.yml file:
[student@workstation task-failure]$cp ntp_configuration.yml \ntp_configuration_blocks.yml
Modify the ntp_configuration_blocks.yml playbook to add the required block sections.
The block section for the Juniper Junos managed nodes must contain the following content:
...output omitted... when: ansible_network_os == "cisco.ios.ios"- name: Reconfiguring Juniper managed nodesblock:- name: Configure NTP on Juniper Junos managed nodes junipernetworks.junos.junos_ntp_global: config: servers: - server: "{{ ntp_server_1 }}" - server: "{{ ntp_server_2 }}" state: merged -name: Verify previous NTP configuration no longer existsjunipernetworks.junos.junos_command:commands:- show configuration system ntpregister: ntp_serversfailed_when: >('server 172.25.23.240;' in ntp_servers['stdout_lines'][0])or('server 172.25.23.230;' in ntp_servers['stdout_lines'][0])when: ansible_network_os == "junipernetworks.junos.junos" - name: Configure NTP on IOS managed nodes ...output omitted...
Pay attention to the fact that the line with the when condition must have the same indentation as the block keyword to apply to all the tasks in the block.
The block section for the IOS managed nodes must have the following content:
...output omitted... when: ansible_network_os == "junipernetworks.junos.junos"- name: Reconfiguring IOS managed nodesblock:- name: Configure NTP on IOS managed nodes cisco.ios.ios_ntp_global: config: servers: - server: "{{ ntp_server_1 }}" - server: "{{ ntp_server_2 }}" state: merged- name: Verify previous NTP configuration no longer existscisco.ios.ios_command:commands:- show run | include ntpregister: ntp_serversfailed_when: >('ntp server 172.25.23.240' in ntp_servers['stdout_lines'][0])or('ntp server 172.25.23.230' in ntp_servers['stdout_lines'][0])when: ansible_network_os == "cisco.ios.ios"
Create a rescue section for the Reconfiguring Juniper managed nodes and Reconfiguring IOS managed nodes blocks in your playbook.
If Ansible detects errors when the playbook attempts to configure NTP, the corresponding backup must be restored on the managed nodes.
The rescue section must be placed after the block section and before the when condition for the ansible_network_os variable.
Use the path to the first backup of your managed nodes for the task that restores its configuration. In a later step you modify this path to use the backup created by the playbook.
The rescue section in your playbook must contain the following content:
...output omitted... ('server 172.25.23.230;' in ntp_servers['stdout_lines'][0])rescue:- name: Remove all existing configurationjunipernetworks.junos.junos_ntp_global:config:state: deleted- name: Restore the Juniper Junos configurationsjunipernetworks.junos.junos_config:update: replacesrc: "{{ inventory_hostname }}.txt"when: ansible_network_os == "junipernetworks.junos.junos" ...output omitted...
The rescue section for the block related to IOS managed nodes must contain the following content:
...output omitted... ('ntp server 172.25.23.230' in ntp_servers['stdout_lines'][0])rescue:- name: Remove all existing configurationcisco.ios.ios_ntp_global:state: deleted- name: Restore the IOS configurationscisco.ios.ios_config:src: "{{ inventory_hostname }}.txt"when: ansible_network_os == "cisco.ios.ios"
Due to the way certain network platforms handle restoring configuration, some *_config modules merge the backup configuration with the active configuration on the target device instead of overwriting it.
For examples of how to restore network configurations, see https://github.com/network-automation/toolkit/tree/master/roles/restore.
Create an always section for each of the blocks in the ntp_configuration.yml playbook.
The tasks in the always section of the blocks must verify and display how the NTP configurations look after running the block and recovery tasks against the managed nodes.
The always section must be placed after the rescue section in each of the blocks and before the when condition for the ansible_network_os variable.
Create an always section for the block related to the Juniper Junos managed nodes.
The always section must contain the following content:
...output omitted...
always:
- name: Verifying NTP configuration
junipernetworks.junos.junos_command:
commands:
- show configuration system ntp
register: junos_ntp
- name: Display NTP configuration
ansible.builtin.debug:
var: junos_ntp['stdout_lines'][0]
when: ansible_network_os == "junipernetworks.junos.junos"
...output omitted...Create an always section for the block related to the Cisco IOS managed nodes.
The always section must contain the following content:
...output omitted...
always:
- name: Verifying NTP configuration
cisco.ios.ios_command:
commands:
- show run | include ntp
register: ios_ntp
- name: Display NTP configuration
ansible.builtin.debug:
var: ios_ntp['stdout_lines'][0]
when: ansible_network_os == "cisco.ios.ios"Run the ntp_configuration_blocks.yml playbook.
If Ansible generates an error when running any of the tasks in the block section, then Ansible runs the tasks in the rescue section to restore the managed nodes to their original state.
The tasks in each always section verify the current NTP configuration on the managed nodes.
When the ntp_configuration_blocks.yml playbook runs as expected, modify it so that the restore tasks in the rescue section of the Ansible blocks use the path to the backups created by the playbook instead of the first backup of your managed nodes.
Run the ntp_configuration_blocks.yml playbook by using the ansible-navigator run command:
[student@workstation task-failure]$ansible-navigator run \ntp_configuration_blocks.yml
The ntp_configuration_blocks.yml playbook creates a back up of the current NTP configuration for the Juniper Junos and IOS managed nodes:
...output omitted... TASK [Back up the Juniper Junos configurations] ********************************** skipping: [iosxe1.lab.example.com] skipping: [iosxe2.lab.example.com] ok: [junos2.lab.example.com] ok: [junos1.lab.example.com] TASK [Back up the IOS configurations] ******************************************** skipping: [junos1.lab.example.com] skipping: [junos2.lab.example.com] changed: [iosxe2.lab.example.com] changed: [iosxe1.lab.example.com] ...output omitted...
The tasks in the first block in your playbook start running against the Juniper Junos managed nodes. The playbook configures the NTP service and then verifies that the previous NTP configuration no longer exists.
The failure condition in the Verify previous NTP configuration no longer exists task makes the task fail for the junos1.lab.example.com managed node:
...output omitted... TASK [Configure NTP on Juniper managed nodes] ************************************ skipping: [iosxe1.lab.example.com] skipping: [iosxe2.lab.example.com] changed: [junos1.lab.example.com] changed: [junos2.lab.example.com]TASK [Verify previous NTP configuration no longer exists]************************ skipping: [iosxe1.lab.example.com] skipping: [iosxe2.lab.example.com]fatal: [junos1.lab.example.com]: FAILED! => {"changed": false,"failed_when_result": true, "stdout": ["server 172.25.23.240;\nserver 172.25.23.230;\nserver 172.25.254.254;\nserver 172.25.250.220;"], "stdout_lines": [["server 172.25.23.240;", "server 172.25.23.230;", "server 172.25.254.254;", "server 172.25.250.220;"]]} ok: [junos2.lab.example.com] ...output omitted...
The ntp_configuration_blocks.yml playbook uses the backup of the junos1.lab.example.com managed node to restore its original configuration.
This rescue task only runs against the junos1.lab.example.com managed node.
If no tasks fail in the section under the block keyword, then this rescue task does not run.
...output omitted... TASK [Restore the Juniper Junos configurations] ********************************** changed: [junos1.lab.example.com] ...output omitted...
The ntp_configuration_blocks.yml playbook displays the NTP configuration for the Juniper Junos managed nodes:
...output omitted... TASK [Display NTP configuration] ************************************************* skipping: [iosxe1.lab.example.com] skipping: [iosxe2.lab.example.com] ok: [junos1.lab.example.com] => { "junos_ntp['stdout_lines'][0]": [ "server 172.25.23.240;", "server 172.25.23.230;" ] } ok: [junos2.lab.example.com] => { "junos_ntp['stdout_lines'][0]": [ "server 172.25.254.254;", "server 172.25.250.220;" ] } ...output omitted...
The tasks in the second block in your playbook start running against the IOS managed nodes. The playbook configures the NTP service and then verifies that the previous NTP configuration no longer exists.
The failure condition in the Verify previous NTP configuration no longer exists task makes the task fail for the iosxe2.lab.example.com managed node:
...output omitted... TASK [Configure NTP on IOS managed nodes] **************************************** skipping: [junos1.lab.example.com] skipping: [junos2.lab.example.com] changed: [iosxe1.lab.example.com] changed: [iosxe2.lab.example.com]TASK [Verify previous NTP configuration no longer exists]************************ skipping: [junos1.lab.example.com] skipping: [junos2.lab.example.com] ok: [iosxe1.lab.example.com]fatal: [iosxe2.lab.example.com]: FAILED! => {"changed": false,"failed_when_result": true, "stdout": ["ntp server 172.25.23.240\nntp server 172.25.250.220\nntp server 172.25.23.230\nntp server 172.25.254.254"], "stdout_lines": [["ntp server 172.25.23.240", "ntp server 172.25.250.220", "ntp server 172.25.23.230", "ntp server 172.25.254.254"]]} ...output omitted...
The ntp_configuration_blocks.yml playbook uses the backup of the iosxe2.lab.example.com managed node to restore its original configuration.
This rescue task only runs against the iosxe2.lab.example.com managed node.
...output omitted... TASK [Restore the IOS configurations] ******************************************** [WARNING]: To ensure idempotency and correct diff the input configuration lines should be similar to how they appear if present in the running configuration on device including the indentation changed: [iosxe2.lab.example.com] ...output omitted...
The ntp_configuration_blocks.yml playbook displays the NTP configuration for the IOS managed nodes:
...output omitted... TASK [Display NTP configuration] ************************************************* skipping: [junos1.lab.example.com] skipping: [junos2.lab.example.com] ok: [iosxe2.lab.example.com] => { "ios_ntp['stdout_lines'][0]": [ "ntp server 172.25.23.240", "ntp server 172.25.23.230" ] } ok: [iosxe1.lab.example.com] => { "ios_ntp['stdout_lines'][0]": [ "ntp server 172.25.254.254", "ntp server 172.25.250.220" ] } ...output omitted...
Edit the ntp_configuration_blocks.yml playbook.
The path for the backup in the Restore the Juniper Junos configurations and Restore the IOS configuration tasks in the rescue section of the blocks must use the backup file created by the ntp_configuration_blocks.yml playbook.
The rescue section for the block related to Juniper Junos managed nodes must contain the following content:
rescue:
- name: Remove all existing configuration
junipernetworks.junos.junos_ntp_global:
config:
state: deleted
- name: Restore the Juniper Junos configurations
junipernetworks.junos.junos_config:
src: "backup/{{ inventory_hostname }}.txt"The rescue section for the block related to Cisco IOS managed nodes must contain the following content:
rescue:
- name: Remove all existing configuration
cisco.ios.ios_ntp_global:
state: deleted
- name: Restore the IOS configurations
cisco.ios.ios_config:
src: "backup/{{ inventory_hostname }}.txt"Run the ntp_configuration_blocks.yml playbook again and verify that the playbook works as expected:
[student@workstation task-failure]$ansible-navigator run \ntp_configuration_blocks.yml...output omitted... TASK [Display NTP configuration] ************************************************* skipping: [iosxe1.lab.example.com] skipping: [iosxe2.lab.example.com] ok: [junos1.lab.example.com] => { "junos_ntp['stdout_lines'][0]": [ "server 172.25.23.240;", "server 172.25.23.230;" ] } ok: [junos2.lab.example.com] => { "junos_ntp['stdout_lines'][0]": [ "server 172.25.254.254;", "server 172.25.250.220;" ] } ...output omitted... TASK [Display NTP configuration] ************************************************* skipping: [junos1.lab.example.com] skipping: [junos2.lab.example.com] ok: [iosxe2.lab.example.com] => { "ios_ntp['stdout_lines'][0]": [ "ntp server 172.25.23.240", "ntp server 172.25.23.230" ] } ok: [iosxe1.lab.example.com] => { "ios_ntp['stdout_lines'][0]": [ "ntp server 172.25.254.254", "ntp server 172.25.250.220" ] } ...output omitted...
Note that after executing the playbook the managed nodes are correctly configured with only the new NTP servers.
Close the /home/student/task-failure directory in VS Code.
If you are using the GNOME terminal, return to the /home/student directory.
Click → in VS Code to close the /home/student/task-failure directory.
If you are using the GNOME terminal, run the cd command to return to the student home directory:
[student@workstation task-failure]$ cd