Bookmark this page

Guided Exercise: Troubleshooting Playbooks

In this exercise, you will troubleshoot a playbook that has been given to you that does not work properly.

Outcomes

You should be able to troubleshoot and resolve issues in playbooks.

Log in to workstation as student using student as the password.

On workstation, run the lab troubleshoot-playbook start script. It verifies whether Ansible is installed on workstation. It also creates the /home/student/troubleshoot-playbook/ directory, and downloads to this directory the inventory, samba.yml, and samba.conf.j2 files from http://materials.example.com/labs/troubleshoot-playbook/.

[student@workstation ~]$ lab troubleshoot-playbook start

Procedure 8.1. Instructions

  1. On workstation, change to the /home/student/troubleshoot-playbook/ directory.

    [student@workstation ~]$ cd ~/troubleshoot-playbook/
    [student@workstation troubleshoot-playbook]$
  2. Create a file named ansible.cfg in the current directory. It should set the log_path parameter to write Ansible logs to the /home/student/troubleshoot-playbook/ansible.log file. It should set the inventory parameter to use the /home/student/troubleshoot-playbook/inventory file deployed by the lab script.

    When you are finished, ansible.cfg should have the following contents:

    [defaults]
    log_path = /home/student/troubleshoot-playbook/ansible.log
    inventory = /home/student/troubleshoot-playbook/inventory
  3. Run the playbook. It will fail with an error.

    This playbook would set up a Samba server if everything were correct. However, the run will fail due to missing double quotes on the random_var variable definition. Read the error message to see how ansible-playbook reports the problem. Notice the variable random_var is assigned a value that contains a colon and is not quoted.

    [student@workstation troubleshoot-playbook]$ ansible-playbook samba.yml
    ERROR! We were unable to read either as JSON nor YAML, these are the errors we got from each:
    JSON: Expecting value: line 1 column 1 (char 0)
    
    Syntax Error while loading YAML.
      mapping values are not allowed in this context
    
    The error appears to be in '/home/student/troubleshoot-playbook/samba.yml': line 8, column 30, but may be elsewhere in the file depending on the exact syntax problem.
    
    The offending line appears to be:
    
        install_state: installed
        random_var: This is colon: test
                                 ^ here
  4. Confirm that the error has been properly logged to the /home/student/troubleshoot-playbook/ansible.log file.

    [student@workstation troubleshoot-playbook]$ tail ansible.log
    
    The error appears to be in '/home/student/troubleshoot-playbook/samba.yml': line 8, column 30, but may be elsewhere in the file depending on the exact syntax problem.
    
    The offending line appears to be:
    
        install_state: installed
        random_var: This is colon: test
                                 ^ here
  5. Edit the samba.yml playbook and correct the error by adding quotes to the entire value being assigned to random_var. The corrected version of the playbook contains the following content:

    ...output omitted...
      vars:
        install_state: installed
        random_var: "This is colon: test"
    ...output omitted...
  6. Check the playbook using the --syntax-check option. Another error is issued due to extra white space in the indentation on the last task, deliver samba config.

    [student@workstation troubleshoot-playbook]$ ansible-playbook --syntax-check \
    > samba.yml
    ERROR! We were unable to read either as JSON nor YAML, these are the errors we got from each:
    JSON: Expecting value: line 1 column 1 (char 0)
    
    Syntax Error while loading YAML.
      did not find expected key
    
    The error appears to be in '/home/student/troubleshoot-playbook/samba.yml': line 44, column 4, but may be elsewhere in the file depending on the exact syntax problem.
    
    The offending line appears to be:
    
    
       - name: deliver samba config
       ^ here
  7. Edit the playbook and remove the extra space for all lines in that task. The corrected playbook should appear as follows:

    ...output omitted...
      - name: configure firewall for samba
        firewalld:
          state: enabled
          permanent: true
          immediate: true
          service: samba
    
      - name: deliver samba config
        template:
          src: templates/samba.conf.j2
          dest: /etc/samba/smb.conf
          owner: root
          group: root
          mode: 0644
  8. Run the playbook using the --syntax-check option. An error is issued due to the install_state variable being used as a parameter in the install samba task. It is not quoted.

    [student@workstation troubleshoot-playbook]$ ansible-playbook --syntax-check \
    > samba.yml
    ERROR! We were unable to read either as JSON nor YAML, these are the errors we got from each:
    JSON: Expecting value: line 1 column 1 (char 0)
    
    Syntax Error while loading YAML.
      found unacceptable key (unhashable type: 'AnsibleMapping')
    
    The error appears to be in '/home/student/troubleshoot-playbook/samba.yml': line 14, column 15, but may be elsewhere in the file depending on the exact syntax problem.
    
    The offending line appears to be:
    
          name: samba
          state: {{ install_state }}
                  ^ here
    We could be wrong, but this one looks like it might be an issue with missing quotes. Always quote template expression brackets when they start a value. For instance:
    
        with_items:
          - {{ foo }}
    
    Should be written as:
    
        with_items:
          - "{{ foo }}"
  9. Edit the playbook and correct the install samba task. The reference to the install_state variable should be in quotes. The resulting file content should look like the following:

    ...output omitted...
      tasks:
      - name: install samba
        yum:
          name: samba
          state: "{{ install_state }}"
    ...output omitted...
  10. Run the playbook using the --syntax-check option. It should not show any additional syntax errors.

    [student@workstation troubleshoot-playbook]$ ansible-playbook --syntax-check \
    > samba.yml
    
    playbook: samba.yml
  11. Run the playbook. An error, related to SSH, will be issued.

    [student@workstation troubleshoot-playbook]$ ansible-playbook samba.yml
    PLAY [Install a samba server] **************************************************
    
    TASK [Gathering Facts] *********************************************************
    fatal: [servera.lab.exammple.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host servera.lab.exammple.com port 22: Connection timed out", "unreachable": true}
    
    PLAY RECAP *********************************************************************
    servera.lab.exammple.com   : ok=0    changed=0    unreachable=1    failed=0  ...
  12. Ensure the managed host servera.lab.example.com is running, using the ping command.

    [student@workstation troubleshoot-playbook]$ ping -c3 servera.lab.example.com
    PING servera.lab.example.com (172.25.250.10) 56(84) bytes of data.
    64 bytes from servera.lab.example.com (172.25.250.10): icmp_seq=1 ttl=64 time=0.247 ms
    64 bytes from servera.lab.example.com (172.25.250.10): icmp_seq=2 ttl=64 time=0.329 ms
    64 bytes from servera.lab.example.com (172.25.250.10): icmp_seq=3 ttl=64 time=0.320 ms
    
    --- servera.lab.example.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 1999ms
    rtt min/avg/max/mdev = 0.247/0.298/0.329/0.041 ms
  13. Ensure that you can connect to the managed host servera.lab.example.com as the devops user using SSH, and that the correct SSH keys are in place. Log off again when you have finished.

    [student@workstation troubleshoot-playbook]$ ssh devops@servera.lab.example.com
    Activate the web console with: systemctl enable --now cockpit.socket
    ...output omitted...
    [devops@servera ~]$ exit
    logout
    Connection to servera.lab.example.com closed.
  14. Rerun the playbook with -vvvv to get more information about the run. An error is issued because the servera.lab.example.com managed host is not reachable.

    [student@workstation troubleshoot-playbook]$ ansible-playbook -vvvv samba.yml
    ...output omitted...
    
    PLAYBOOK: samba.yml ******************************************************
    1 plays in samba.yml
    
    PLAY [Install a samba server] ********************************************
    
    TASK [Gathering Facts] ***************************************************
    task path: /home/student/troubleshoot-playbook/samba.yml:2
    <servera.lab.exammple.com> ESTABLISH SSH CONNECTION FOR USER: devops
    ...output omitted...
    fatal: [servera.lab.exammple.com]: UNREACHABLE! => {
        "changed": false,
        "msg": "Failed to connect to the host via ssh: OpenSSH_8.0p1, OpenSSL ... Control socket \"/home/student/.ansible/cp/d4775f48c9\" does not exist\r\ndebug2: resolving \"servera.lab.exammple.com\" port 22\r\ndebug2: ssh_connect_direct\r\ndebug1: Connecting to servera.lab.exammple.com [3.223.115.185] port 22.\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug1: connect to address 3.223.115.185 port 22: Connection timed out\r\nssh: connect to host servera.lab.exammple.com port 22: Connection timed out",
        "unreachable": true
    }
    
    ...output omitted...
    PLAY RECAP ***************************************************************
    servera.lab.exammple.com   : ok=0    changed=0    unreachable=1    failed=0
  15. When using the highest level of verbosity with Ansible, examining the Ansible log file is a better option than checking console output. You might view the log file using the less command, or you might search for patterns in the log file using the grep command. Search for the word fatal in the /home/student/troubleshoot-playbook/ansible.log file.

    [student@workstation troubleshoot-playbook]$ grep -i fatal ansible.log
    2021-07-15 13:56:21,766 p=45752 u=student n=ansible | fatal: [servera.lab.exammple.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host servera.lab.exammple.com port 22: Connection timed out", "unreachable": true}
    2021-07-15 14:22:43,262 p=46055 u=student n=ansible | fatal: [servera.lab.exammple.com]: UNREACHABLE! => {
  16. Investigate the inventory file for errors. Notice the [samba_servers] group has misspelled servera.lab.example.com. Correct this error as shown below:

    [samba_servers]
    servera.lab.example.com
    ...output omitted...
  17. Run the playbook again. The debug install_state variable task returns the message The state for the samba service is installed. This task makes use of the debug module, and displays the value of the install_state variable. An error is also shown in the deliver samba config task, because no samba.j2 file is available in the working directory, /home/student/troubleshoot-playbook/.

    [student@workstation troubleshoot-playbook]$ ansible-playbook samba.yml
    
    PLAY [Install a samba server] **************************************************
    ...output omitted...
    TASK [debug install_state variable] ********************************************
    ok: [servera.lab.example.com] => {
        "msg": "The state for the samba service is installed"
    }
    ...output omitted...
    TASK [deliver samba config] ****************************************************
    fatal: [servera.lab.example.com]: FAILED! => {"changed": false, "msg": "Could not find or access 'samba.j2'\nSearched in:\n\t/home/student/troubleshoot-playbook/templates/samba.j2\n\t/home/student/troubleshoot-playbook/samba.j2\n\t/home/student/troubleshoot-playbook/templates/samba.j2\n\t/home/student/troubleshoot-playbook/samba.j2 on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}
    ...output omitted...
    PLAY RECAP *********************************************************************
    servera.lab.example.com    : ok=7    changed=3    unreachable=0    failed=1  ...
  18. Edit the playbook, and correct the src parameter in the deliver samba config task to be samba.conf.j2. When you are finished it should look like the following:

    ...output omitted...
      - name: deliver samba config
        template:
          src: samba.conf.j2
          dest: /etc/samba/smb.conf
          owner: root
    ...output omitted...
  19. Run the playbook again. Execute the playbook using the --step option. It should run without errors.

    [student@workstation troubleshoot-playbook]$ ansible-playbook samba.yml --step
    
    PLAY [Install a samba server] ************************************************
    Perform task: TASK: Gathering Facts (N)o/(y)es/(c)ontinue: y
    ...output omitted...
    Perform task: TASK: install samba (N)o/(y)es/(c)ontinue: y
    ...output omitted...
    Perform task: TASK: install firewalld (N)o/(y)es/(c)ontinue: y
    ...output omitted...
    Perform task: TASK: debug install_state variable (N)o/(y)es/(c)ontinue: y
    ...output omitted...
    Perform task: TASK: start samba (N)o/(y)es/(c)ontinue: y
    ...output omitted...
    Perform task: TASK: start firewalld (N)o/(y)es/(c)ontinue: y
    ...output omitted...
    Perform task: TASK: configure firewall for samba (N)o/(y)es/(c)ontinue: y
    ...output omitted...
    Perform task: TASK: deliver samba config (N)o/(y)es/(c)ontinue: y
    ...output omitted...
    PLAY RECAP *********************************************************************
    servera.lab.example.com    : ok=8    changed=1    unreachable=0    failed=0

Finish

On workstation, run the lab troubleshoot-playbook finish script to clean up this exercise.

[student@workstation ~]$ lab troubleshoot-playbook finish

This concludes the guided exercise.

Revision: rh294-8.4-9cb53f0