Bookmark this page

Lab: Troubleshooting Ansible

Performance Checklist

In this lab, you will troubleshoot problems that occur when you try to run a playbook that has been provided to you.

Outcomes

You should be able to:

  • Troubleshoot playbooks.

  • Troubleshoot managed hosts.

Log in to workstation as student using student as the password. Run the lab troubleshoot-review start command.

[student@workstation ~]$ lab troubleshoot-review start

This script verifies that Ansible is installed on workstation, and creates the ~student/troubleshoot-review/html/ directory. It downloads the ansible.cfg, inventory-lab, secure-web.yml, and vhosts.conf files from http://materials.example.com/labs/troubleshoot-review/ to the /home/student/troubleshoot-review/ directory. It also downloads the index.html file to the /home/student/troubleshoot-review/html/ directory.

Procedure 8.3. Instructions

  1. From the ~/troubleshoot-review directory, check the syntax of the secure-web.yml playbook. This playbook contains one play that sets up Apache HTTPD with TLS/SSL for hosts in the group webservers. Fix the issue that is reported.

    1. On workstation, change to the /home/student/troubleshoot-review project directory.

      [student@workstation ~]$ cd ~/troubleshoot-review/
    2. Check the syntax of the secure-web.yml playbook. This playbook sets up Apache HTTPD with TLS/SSL for hosts in the webservers group when everything is correct.

      [student@workstation troubleshoot-review]$ ansible-playbook --syntax-check \
      > secure-web.yml
      ERROR! We were unable to read either as JSON nor YAML, these are the errors we got from each:
      JSON: Expecting value: line 1 column 1 (char 0)
      
      Syntax Error while loading YAML.
        mapping values are not allowed in this context
      
      The error appears to be in '/home/student/troubleshoot-review/secure-web.yml': line 7, column 30, but may be elsewhere in the file depending on the exact syntax problem.
      
      The offending line appears to be:
      
        vars:
          random_var: This is colon: test
                                   ^ here
    3. Correct the syntax issue in the definition of the random_var variable by adding double quotes to the This is colon: test string. The resulting change should appear as follows:

      ...output omitted...
        vars:
          random_var: "This is colon: test"
      ...output omitted...
  2. Check the syntax of the secure-web.yml playbook again. Fix the issue that is reported.

    1. Check the syntax of secure-web.yml using ansible-playbook --syntax-check again.

      [student@workstation troubleshoot-review]$ ansible-playbook --syntax-check \
      > secure-web.yml
      ERROR! We were unable to read either as JSON nor YAML, these are the errors we got from each:
      JSON: Expecting value: line 1 column 1 (char 0)
      
      Syntax Error while loading YAML.
        did not find expected '-' indicator
      
      The error appears to be in '/home/student/troubleshoot-review/secure-web.yml': line 38, column 10, but may be elsewhere in the file depending on the exact syntax problem.
      
      The offending line appears to be:
      
      
               - name: start and enable web services
               ^ here
    2. Correct any syntax issues in the indentation. Remove the extra space at the beginning of the start and enable web services task elements. The resulting change should appear as follows:

      ...output omitted...
                args:
                  creates: /etc/pki/tls/certs/serverb.lab.example.com.crt
      
              - name: start and enable web services
                service:
                  name: httpd
                  state: started
                  enabled: yes
      
              - name: deliver content
                copy:
                  dest: /var/www/vhosts/serverb-secure
                  src: html/
      ...output omitted...
  3. Check the syntax of the secure-web.yml playbook a third time. Fix the issue that is reported.

    1. Check the syntax of the secure-web.yml playbook.

      [student@workstation troubleshoot-review]$ ansible-playbook --syntax-check \
      > secure-web.yml
      ERROR! We were unable to read either as JSON nor YAML, these are the errors we got from each:
      JSON: Expecting value: line 1 column 1 (char 0)
      
      Syntax Error while loading YAML.
        found unacceptable key (unhashable type: 'AnsibleMapping')
      
      The error appears to be in '/home/student/troubleshoot-review/secure-web.yml': line 13, column 20, but may
      be elsewhere in the file depending on the exact syntax problem.
      
      The offending line appears to be:
      
                yum:
                  name: {{ item }}
                         ^ here
      We could be wrong, but this one looks like it might be an issue with
      missing quotes. Always quote template expression brackets when they
      start a value. For instance:
      
          with_items:
            - {{ foo }}
      
      Should be written as:
      
          with_items:
            - "{{ foo }}"
    2. Correct the item variable in the install web server packages task. Add double quotes to {{ item }}. The resulting change should appear as follows:

      ...output omitted...
              - name: install web server packages
                yum:
                  name: "{{ item }}"
                  state: latest
                notify:
                  - restart services
                loop:
                  - httpd
                  - mod_ssl
      ...output omitted...
  4. Check the syntax of the secure-web.yml playbook a fourth time. It should not show any syntax errors.

    1. Review the syntax of the secure-web.yml playbook. It should not show any syntax errors.

      [student@workstation troubleshoot-review]$ ansible-playbook --syntax-check \
      > secure-web.yml
      
      playbook: secure-web.yml
  5. Run the secure-web.yml playbook. Ansible is not able to connect to serverb.lab.example.com. Fix this problem.

    1. Run the secure-web.yml playbook. This will fail with an error.

      [student@workstation troubleshoot-review]$ ansible-playbook secure-web.yml
      
      PLAY [create secure web service] ***********************************************
      
      TASK [Gathering Facts] *********************************************************
      fatal: [serverb.lab.example.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: students@serverc.lab.example.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).", "unreachable": true}
      
      PLAY RECAP *********************************************************************
      serverb.lab.example.com    : ok=0    changed=0    unreachable=1    failed=0  ...
    2. Run the secure-web.yml playbook again, adding the -vvv parameter to increase the verbosity of the output.

      Notice that Ansible appears to be connecting to serverc.lab.example.com instead of serverb.lab.example.com.

      [student@workstation troubleshoot-review]$ ansible-playbook secure-web.yml -vvv
      ...output omitted...
      TASK [Gathering Facts] *********************************************************
      task path: /home/student/troubleshoot-review/secure-web.yml:3
      <serverc.lab.example.com> ESTABLISH SSH CONNECTION FOR USER: students
      <serverc.lab.example.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="students"' -o ConnectTimeout=10 -o ControlPath=/home/student/.ansible/cp/bc0c05136a serverc.lab.example.com '/bin/sh -c '"'"'echo ~students && sleep 0'"'"''
      ...output omitted...
    3. Correct the line in the inventory file. Delete the ansible_host host variable so the file appears as shown below:

      [webservers]
      serverb.lab.example.com
  6. Run the secure-web.yml playbook again. Ansible should authenticate as the devops remote user on the managed host. Fix this issue.

    1. Run the secure-web.yml playbook.

      [student@workstation troubleshoot-review]$ ansible-playbook secure-web.yml -vvv
      ...output omitted...
      TASK [Gathering Facts] *********************************************************
      task path: /home/student/troubleshoot-review/secure-web.yml:3
      <serverb.lab.example.com> ESTABLISH SSH CONNECTION FOR USER: students
      ...output omitted...
      fatal: [serverb.lab.example.com]: UNREACHABLE! => {
      ...output omitted...
    2. Edit the secure-web.yml playbook to make sure devops is the remote_user for the play. The first lines of the playbook should appear as follows:

      ---
      # start of secure web server playbook
      - name: create secure web service
        hosts: webservers
        remote_user: devops
      ...output omitted...
  7. Run the secure-web.yml playbook a third time. Fix the issue that is reported.

    1. Run the secure-web.yml playbook.

      [student@workstation troubleshoot-review]$ ansible-playbook secure-web.yml -vvv
      ...output omitted...
      failed: [serverb.lab.example.com] (item=mod_ssl) => {
          "ansible_loop_var": "item",
          "changed": false,
          "invocation": {
              "module_args": {
                  "allow_downgrade": false,
                  "autoremove": false,
      ...output omitted...
                  "validate_certs": true
              }
          },
          "item": "mod_ssl",
          "msg": "This command has to be run under the root user.",
          "results": []
      }
      ...output omitted...
    2. Edit the play to make sure that it has become: true or become: yes set. The resulting change should appear as follows:

      ---
      # start of secure web server playbook
      - name: create secure web service
        hosts: webservers
        remote_user: devops
        become: true
      ...output omitted...
  8. Run the secure-web.yml playbook one more time. It should complete successfully. Use an ad hoc command to verify that the httpd service is running.

    1. Run the secure-web.yml playbook.

      [student@workstation troubleshoot-review]$ ansible-playbook secure-web.yml
      
      PLAY [create secure web service] ***********************************************
      ...output omitted...
      TASK [install web server packages] *********************************************
      changed: [serverb.lab.example.com] => (item=httpd)
      changed: [serverb.lab.example.com] => (item=mod_ssl)
      ...output omitted...
      TASK [httpd_conf_syntax variable] **********************************************
      ok: [serverb.lab.example.com] => {
          "msg": "The httpd_conf_syntax variable value is {'cmd': ['/sbin/httpd', '-t'], 'stdout': '', 'stderr': 'Syntax OK', 'rc': 0, 'start': '2021-07-16 14:08:35.304347', 'end': '2021-07-16 14:08:35.342415', 'delta': '0:00:00.038068', 'changed': True, 'stdout_lines': [], 'stderr_lines': ['Syntax OK'], 'failed': False, 'failed_when_result': False}"
      }
      ...output omitted...
      RUNNING HANDLER [restart services] *********************************************
      changed: [serverb.lab.example.com]
      
      PLAY RECAP *********************************************************************
      serverb.lab.example.com    : ok=10   changed=7    unreachable=0    failed=0  ...
    2. Use an ad hoc command to determine the state of the httpd service on serverb.lab.example.com. The httpd service should now be running on serverb.lab.example.com.

      [student@workstation troubleshoot-review]$ ansible all -u devops -b \
      > -m command -a 'systemctl status httpd'
      serverb.lab.example.com | CHANGED | rc=0 >>
      ● httpd.service - The Apache HTTP Server
         Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
         Active: active (running) since Fri 2021-07-16 14:08:37 EDT; 3min 1s ago
      ...output omitted...

Evaluation

On workstation, run the lab troubleshoot-review grade script to confirm success on this exercise.

[student@workstation ~]$ lab troubleshoot-review grade

Finish

On workstation, run the lab troubleshoot-review finish script to clean up this lab.

[student@workstation ~]$ lab troubleshoot-review finish

This concludes the lab.

Revision: rh294-8.4-9cb53f0