Bookmark this page

Guided Exercise: Troubleshooting Containerized Applications

Configure container health checks, and inspect the container logs. You also configure a container host to record the container events.

Outcomes

You should be able to configure the health check option on a container and inspect the container logs for the source of an unexpected behavior. You should also be able to configure the container host to record container events.

As the student user on the workstation machine, use the lab command to prepare your system for this exercise.

[student@workstation ~]$ lab start application-container

This command confirms that the required hosts for this exercise are accessible and creates the containerized service.

Instructions

A new containerized website is not displaying the service test page. You are requested to resolve the issue. Additionally, you install mechanisms that monitor the behavior of the container.

  1. Log in to servera as the student user.

    [student@workstation ~]$ ssh student@servera
    [student@servera ~]$
  2. Verify the container and its status.

    [student@servera ~]$ podman ps -a
    CONTAINER ID  IMAGE                                            COMMAND               CREATED         STATUS             PORTS                    NAMES
    97ae356eb822  registry.access.redhat.com/ubi8/httpd-24:latest  /usr/bin/run-http...  52 minutes ago  Up 52 minutes ago  0.0.0.0:40195->8080/tcp  website
  3. Verify that container events are logged.

    1. Discover the container event logger mechanism.

      [student@servera ~]$ grep logger /usr/share/containers/containers.conf
      events_logger = "file"
    2. Change the logger mechanism to journald.

      [student@servera ~]$ sudo vim /usr/share/containers/containers.conf
      [sudo] password for student: student
      ...output omitted...
      events_logger = "journald"
    3. Restart the container and verify that container events are logged to the system journal.

      [student@servera ~]$ podman stop website
      ...output omitted...
      [student@servera ~]$ podman start website
      website
      [student@servera ~]$ podman events --since 5m --until 0m
      2021-10-21 18:33:52.624255505 -0400 EDT container cleanup 97ae...a673 (image=registry.access.redhat.com/ubi8/httpd-24:latest, name=website,
      ...output omitted...
      2021-10-21 18:33:56.054608729 -0400 EDT container init 97ae...a673 (image=registry.access.redhat.com/ubi8/httpd-24:latest, name=website,
      ...output omitted...
    4. Filter the events by container name. Format the output as JSON to read the events more easily.

      [student@servera ~]$ podman events --since 5m --filter container=website --format json --until 0m
      {"ID":"97ae...a673","Image":
      "registry.access.redhat.com/ubi8/httpd-24:latest","Name":"website","Status":
      "died", "Time":"2021-10-21T18:33:52.576507917-04:00","Type":"container",
      "Attributes":null}
      {"ID":"97ae...a673","Image":
      "registry.access.redhat.com/ubi8/httpd-24:latest","Name":"website","Status":
      "cleanup","Time":"2021-10-21T18:33:52.624255505-04:00","Type":"container",
      ...output omitted...
  4. View information about the containerized service.

    1. Verify that the containerized service process is running.

      [student@servera ~]$ ps auxf | grep httpd
      101000 4827 0.0 1.1 391232  21800 pts/0  Ss+ 18:33 0:00 \_ httpd -D FOREGROUND
      101000 4868 0.0 0.2 369144  5500  pts/0  S+  18:33 0:00   \_ httpd -D FOREGROUND
      101000 4869 0.0 1.0 2038556 20336 pts/0  Sl+ 18:33 0:00   \_ httpd -D FOREGROUND
      101000 4871 0.0 1.2 1907420 22432 pts/0  Sl+ 18:33 0:00   \_ httpd -D FOREGROUND
      101000 4873 0.0 0.9 1907420 18336 pts/0  Sl+ 18:33 0:00   \_ httpd -D FOREGROUND
    2. Verify the rootless container configuration.

      [student@servera ~]$ cat /etc/subuid
      student:100000:65536
      devops:165536:65536
      [student@servera ~]$ cat /etc/subgid
      student:100000:65536
      devops:165536:65536
      [student@servera ~]$ podman unshare cat /proc/self/uid_map
               0       1000          1
               1     100000      65536
      [student@servera ~]$ podman unshare cat /proc/self/gid_map
               0       1000          1
               1     100000      65536
  5. Manage the containerized service.

    1. View the running container's attributes.

      [student@servera ~]$ podman inspect website
      ...output omitted...
      "CreateCommand": [
                      "podman",
                      "run",
                      "--name",
                      "website",
                      "-dt",
                      "-p",
                      "8080",
                      "registry.access.redhat.com/ubi8/httpd-24"
                  ],
      ...output omitted...
    2. Attempt to verify that the container is showing the test page.

      The connection fails.

      [student@servera ~]$ curl localhost:8080
      curl: (7) Failed to connect to localhost port 8080: Connection refused
    3. Verify the ports that the container image exposes.

      [student@servera ~]$ podman inspect -f "{{.Config.ExposedPorts}}" registry.access.redhat.com/ubi8/httpd-24
      map[8080/tcp:{} 8443/tcp:{}]
    4. List the ports that the containerized service exposes.

      [student@servera ~]$ podman ps -a --format "{{.Names}} {{.Ports}}"
      website 0.0.0.0:40195->8080/tcp
    5. Stop the container. Run the container again with port forwarding of container port 8080 to host port 8080.

      [student@servera ~]$ podman stop website
      ...output omitted...
      [student@servera ~]$ podman rm website
      ...output omitted...
      [student@servera ~]$ podman run --name website -dt -p 8080:8080 registry.access.redhat.com/ubi8/httpd-24
      ...output omitted...
    6. Attempt again to verify that the container is showing the test page.

      The connection succeeds. Ignore any "403 Forbidden" errors, the purpose is to confirm a response appears.

      [student@servera ~]$ curl -Iv localhost:8080
      * Rebuilt URL to: localhost:8080/
      *   Trying ::1...
      * TCP_NODELAY set
      * Connected to localhost (::1) port 8080 (#0)
      > HEAD / HTTP/1.1
      > Host: localhost:8080
      > User-Agent: curl/7.61.1
      > Accept: /
      ...output omitted...
  6. Add a healthcheck option to the container to monitor the containerized service.

    1. Stop the container. Run the container with a curl command as the health check to validate the container's health.

      [student@servera ~]$ podman stop website
      ...output omitted...
      [student@servera ~]$ podman rm website
      ...output omitted...
      [student@servera ~]$ podman run --name website --health-cmd='curl http://localhost:8080 || exit 1' --health-interval=0 -dt -p 8080:8080 registry.access.redhat.com/ubi8/httpd-24
      ...output omitted...
    2. Use the podman healthcheck run command to verify the health of the container.

      [student@servera ~]$ podman healthcheck run website
      healthy
  7. View the container logs with the podman logs command.

    [student@servera ~]$ podman logs website
    ...output omitted...
    AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.0.2.100. Set the 'ServerName' directive globally to suppress this message
    [Fri Oct 22 00:08:27.724749 2021] [mpm_event:notice] [pid 1:tid 140441417752000] AH00489: Apache/2.4.37 (Red Hat Enterprise Linux) OpenSSL/1.1.1g configured -- resuming normal operations
    [Fri Oct 22 00:08:27.724772 2021] [core:notice] [pid 1:tid 140441417752000] AH00094: Command line: 'httpd -D FOREGROUND'
    [Fri Oct 22 00:12:00.474978 2021] [autoindex:error] [pid 44:tid 140440489633536] [client 127.0.0.1:41236] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html) found, and server-generated directory index forbidden by Options directive
    127.0.0.1 - - [22/Oct/2021:00:12:00 +0000] "GET / HTTP/1.1" 403 4481 "-" "curl/7.61.1"
  8. Return to workstation as the student user.

    [student@servera ~]$ exit
    [student@workstation ~]$

Finish

On the workstation machine, use the lab command to complete this exercise. This is important to ensure that resources from previous exercises do not impact upcoming exercises.

[student@workstation ~]$ lab finish application-container

Revision: rh342-8.4-6dd89bd