Managing Virtual Machines with Red Hat OpenShift Virtualization

Bookmark this page
P123456789
PreviousNext

Lab: Configuring Kubernetes High Availability for Virtual Machines

Configure Kubernetes resources to provide high availability to virtual machines and their applications.

Outcomes

  • Configure load balancing for a web application on two VMs.

  • Configure health probes and watchdog devices on VMs.

  • Configure VMs to automatically recover from a failed node.

As the student user on the workstation machine, use the lab command to prepare your environment for this exercise, and to ensure that all required resources are available.

[student@workstation ~]$ lab start ha-review

Instructions

  1. From a command-line window on the workstation machine, use the oc command to log in to your Red Hat OpenShift cluster as the admin user with redhatocp as the password. The OpenShift cluster API endpoint is https://api.ocp4.example.com:6443.

    Confirm that the www1, www2, and mariadb-server VMs run under the ha-review project.

    1. From a command-line window, log in to your OpenShift cluster as the admin user.

      [student@workstation ~]$ oc login -u admin -p redhatocp \
  https://api.ocp4.example.com:6443
Login Successful
...output omitted...

    2. Change to the ha-review project.

      [student@workstation ~]$ oc project ha-review
Now using project "ha-review" on server "https://api.ocp4.example.com:6443".

    3. Confirm that the www1, www2, and mariadb-server VMs run under the ha-review project.

      [student@workstation ~]$ oc get vm
NAME             AGE   STATUS    READY
mariadb-server   15m   Running   True
www1             22m   Running   True
www2             21m   Running   True

  2. Configure Kubernetes networking resources to provide high availability to the web application that is hosted on the www1 and www2 VMs.

    Create a web service that uses the app: web selector in the ha-review project. Confirm that the web service endpoints resolve to the IP addresses of the www1 and www2 VMIs.

    Create a www route that exposes the web service for external access. Set the route host to web-ha-review.apps.ocp4.example.com and configure a web custom cookie for the route.

    The lab command prepares the service.yaml and route.yaml files in the ~/DO316/labs/ha-review directory for you. Edit and then apply the files to create the web service and the www route.

    1. Edit the ~/DO316/labs/ha-review/service.yaml file to configure the web service in the ha-review project with the app: web selector.

      apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: ha-review
spec:
  type: ClusterIP
  selector:
    app: web
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

    2. Create the route resource.

      [student@workstation ~]$ oc apply -f ~/DO316/labs/ha-review/service.yaml
service/web created

    3. List the VMI resources in the ha-review project. Note the IP addresses for the www1 and www2 VMIs. The IP addresses might differ in your environment.

      [student@workstation ~]$ oc get vmi
NAME             AGE   PHASE     IP          NODENAME   READY
mariadb-server   14m   Running   10.9.2.67   worker01   True
www1             25m   Running   10.9.2.54   worker01   True
www2             20m   Running   10.9.2.61   worker01   True

    4. Confirm that the web service endpoints resolve to the IP addresses of the www1 and www2 VMIs.

      [student@workstation ~]$ oc get endpoints
NAME   ENDPOINTS                   AGE
web    10.9.2.54:80,10.9.2.61:80   109s

    5. Expose the web service for external access by creating a route. Edit the ~/DO316/labs/ha-review/route.yaml file to configure the www route in the ha-review project. Specify the web-ha-review.apps.ocp4.example.com host, and then configure a web custom cookie for the route.

      apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: www
  namespace: ha-review
  annotations:
    router.openshift.io/cookie_name: "web"
spec:
  host: web-ha-review.apps.ocp4.example.com
  port:
    targetPort: 80
  to:
    kind: Service
    name: web

    6. Use the oc apply command to create the resource.

      [student@workstation ~]$ oc apply -f ~/DO316/labs/ha-review/route.yaml
route.route.openshift.io/wwww created

    7. Use the oc command to verify the www route settings.

      [student@workstation ~]$ oc get route
NAME   HOST/PORT                             PATH   SERVICES   PORT   ...
www    web-ha-review.apps.ocp4.example.com          web        80     ...

  3. Confirm that clients can access the www route at the web-ha-review.apps.ocp4.example.com host, by using the curl command. Confirm that OpenShift load balances the web application between the www1 and www2 VMIs.

    Extract the web custom cookie from the www route, and then use the cookie to test session stickiness. Use the curl command with the -c /tmp/cookie-jar option to save the web cookie to the workstation machine. Confirm that the web cookie provides session stickiness by using the curl command with the -b /tmp/cookie-jar option.

    1. Use the curl command to verify that the www route is accessible.

      [student@workstation ~]$ curl web-ha-review.apps.ocp4.example.com
Welcome to www1

    2. Use the curl command to send five subsequent requests to the www route. Confirm that the web application is load balanced between the www1 and www2 VMs.

      [student@workstation ~]$ curl web-ha-review.apps.ocp4.example.com/?[1-5]
[1/5]: web-ha-review.apps.ocp4.example.com/?1 --> <stdout>
--_curl_--web-ha-review.apps.ocp4.example.com/?1
Welcome to www2

[2/5]: web-ha-review.apps.ocp4.example.com/?2 --> <stdout>
--_curl_--web-ha-review.apps.ocp4.example.com/?2
Welcome to www1

[3/5]: web-ha-review.apps.ocp4.example.com/?3 --> <stdout>
--_curl_--web-ha-review.apps.ocp4.example.com/?3
Welcome to www2

[4/5]: web-ha-review.apps.ocp4.example.com/?4 --> <stdout>
--_curl_--web-ha-review.apps.ocp4.example.com/?4
Welcome to www1

[5/5]: web-ha-review.apps.ocp4.example.com/?5 --> <stdout>
--_curl_--web-ha-review.apps.ocp4.example.com/?5
Welcome to www2

    3. Use the curl command to save the web cookie to the workstation machine.

      [student@workstation ~]$ curl web-ha-review.apps.ocp4.example.com \
  -c /tmp/cookie-jar
Welcome to www1

    4. Confirm that the web cookie provides session stickiness. Use the curl with the web cookie to access the www route.

      [student@workstation ~]$ curl web-ha-review.apps.ocp4.example.com \
  -b /tmp/cookie-jar
Welcome to www1

  4. The web servers that run inside the www1 and www2 VMs expose the /health endpoint to monitor the application status. Add a readiness probe to the www1 VM that uses HTTP GET requests to test the /health endpoint.

    The lab command prepares the ~/DO316/labs/ha-review/readiness.yaml file for you. Reference this file to create the readiness probe for the www1 VM.

    After you configure the readiness probe, restart the VM to regenerate the www1 VMI.

    If the readiness probe fails, then OpenShift removes the www1 VMI as a web service endpoint. To test the readiness probe, log in to the VM console as the root user with redhat as the password. Then, stop the httpd service and confirm that OpenShift removes the www1 VMI from the web service endpoints.

    Finally, start the httpd service and confirm that the web service endpoints include the www1 VMI again.

    1. Open a web browser and navigate to https://console-openshift-console.apps.ocp4.example.com. Select htpasswd_provider and log in as the admin user with redhatocp as the password.

    2. Navigate to Virtualization → VirtualMachines and then select the ha-review project. Select the www1 VM and then click the YAML tab. Use the YAML editor to create a readiness probe for the VM.

      You can reference the readinessProbe section in the ~/DO316/labs/ha-review/readiness.yaml file.

      Click Save to save the changes.

      apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
...output omitted...
spec:
  ...output omitted...
  template:
    metadata:
    ...output omitted...
    spec:
      architecture: amd64
      domain:
        ...output omitted...
      readinessProbe:
        httpGet:
          path: /health
          port: 80
        initialDelaySeconds: 10
        periodSeconds: 5
        timeoutSeconds: 2
        failureThreshold: 2
        successThreshold: 1
      evictionStrategy: LiveMigrate
      hostname: www1
...output omitted...

    3. Click Actions → Restart to restart the www1 VMI resource so Red Hat OpenShift Virtualization recognizes the probe. Wait a few minutes until the VMI is again in the Running state.

    4. Click the Console tab and then log in to the www1 VM console as the root user with redhat as the password. Stop the httpd service on the VM.

      ...output omitted...
www1 login: root
Password: redhat
[root@www1 ~]# systemctl stop httpd

    5. From the command-line window, use the oc command to confirm that the web service endpoints do not include the IP address of the www1 VMI.

      [student@workstation ~]$ oc get endpoints
NAME   ENDPOINTS      AGE
web    10.9.2.61:80   13m

    6. From the www1 console, start the httpd service.

      [root@www1 ~]# systemctl start httpd

    7. Return to the command-line window on the workstation machine and confirm that the web service endpoints include the IP address of the www1 VMI.

      [student@workstation ~]$ oc get endpoints
NAME   ENDPOINTS                   AGE
web    10.8.2.63:80,10.9.2.61:80   14m

  5. Add an i6300esb watchdog device named testwatchdog on the www2 VM. Set the action field to the poweroff value, to power off the VM if the VM's operating system is unresponsive. The lab command prepares the ~/DO316/labs/ha-review/watchdog.yaml file for your reference. Restart the VM to apply the changes.

    Then, log in to the www2 VM as the root user with redhat as the password. Confirm that the watchdog device works by using the echo > /dev/watchdog command.

    1. From the OpenShift web console, navigate to Virtualization → VirtualMachines. Select the www2 VM, and then navigate to the YAML tab.

    2. Use the YAML editor to declare the watchdog device, and then click Save. You can copy and paste the watchdog section from the ~/DO316/labs/ha-review/watchdog.yaml file that the lab command prepares.

      apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
...output omitted...
spec:
  ...output omitted...
  template:
    metadata:
    ...output omitted...
    spec:
      architecture: amd64
      domain:
        cpu:
          ...output omitted...
        devices:
          disks:
          - bootOrder: 1
            disk:
              bus: virtio
            name: www2
          - disk:
              bus: virtio
            name: cloudinitdisk
          interfaces:
          - macAddress: 02:7e:be:00:00:02
            masquerade: {}
            name: default
          networkInterfaceMultiqueue: true
          rng: {}
          watchdog:
            i6300esb:
              action: poweroff
            name: testwatchdog
        machine:
          type: pc-q35-rhel8.4.0
...output omitted...

    3. Click Actions → Restart to restart the www2 VMI resource so Red Hat OpenShift Virtualization recognizes the probe. Wait a few minutes until the VMI is again in the Running state.

    4. Confirm that the watchdog device works on the www2 VM. Click the Console tab and log in as the root user with redhat as the password. Use the echo > /dev/watchdog command to simulate an unresponsive operating system. Confirm that OpenShift restarts the VM after 30 seconds.

      ...output omitted...
www2 login: root
Password: redhat
[root@www2 ~]# echo > /dev/watchdog
[  45.680382] watchdog: watchdog0: watchdog did not stop!

  6. The MariaDB database that runs inside the mariadb-server VM listens on TCP port 3306. Add a liveness probe that tests the service by sending requests to the TCP socket.

    The lab command prepares the ~/DO316/labs/ha-review/liveness.yaml file for you. Reference this file to create the liveness probe on the mariadb-server VM.

    After you configure the liveness probe, restart the VM to regenerate the mariadb-server VMI.

    If the liveness probe fails, then OpenShift automatically restarts the mariadb-server VM. To test the liveness probe, log in to the VM console as the root user with redhat as the password. Then, stop the mysql service and confirm that OpenShift restarts the VM.

    1. From the OpenShift web console, navigate to Virtualization → VirtualMachines. Select the mariadb-server VM, and then navigate to the YAML tab.

    2. Use the YAML editor to declare the probe, and then click Save. You can copy and paste the livenessProbe section from the ~/DO316/labs/ha-review/liveness.yaml file.

      apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
...output omitted...
spec:
  ...output omitted...
  template:
    metadata:
    ...output omitted...
    spec:
      architecture: amd64
      domain:
        ...output omitted...
      livenessProbe:
        tcpSocket:
          port: 3306
        initialDelaySeconds: 10
        periodSeconds: 5
      evictionStrategy: LiveMigrate
      hostname: mariadb-server
...output omitted...

    3. Click Actions → Restart to restart the mariadb-server VMI resource so Red Hat OpenShift Virtualization recognizes the probe. Wait a few minutes until the VMI is again in the Running state.

    4. Navigate to the Console tab. Log in as the root user with redhat as the password, and then stop the mysql service.

      mariadb-server login: root
Password: redhat
[root@mariadb-server ~]# systemctl stop mysql
[root@mariadb-server ~]#

      Because the liveness probe fails, the VM restarts after a few seconds.

  7. Monitor the availability of the web application during a node failure. Open a command-line window and then execute the ~/DO316/labs/ha-review/loop.sh file.

    Prevent new workloads from scheduling on the node that hosts the www2 VMI, and then drain the node of its workloads.

    1. Open a command-line window on the workstation machine. Execute the ~/DO316/labs/ha-review/loop.sh file.

      [student@workstation ~]$ sh ~/DO316/labs/ha-review/loop.sh
Welcome to www2
Welcome to www2
Welcome to www1
Welcome to www1
...output omitted...

    2. Change to the command-line window where you confirmed the new VMI instance. Identify the node that hosts the www2 VMI.

      [student@workstation ~]$ oc get vmi
NAME             AGE     PHASE     IP          NODENAME   READY
mariadb-server   92s     Running   10.9.2.74   worker01   True
www1             17m     Running   10.8.2.63   worker02   True
www2             6m24s   Running   10.9.2.71   worker01   True

      Important

      The identified node in your lab might differ from the example output. Be sure to modify your commands accordingly.

    3. Prevent new workloads from scheduling on the identified node.

      [student@workstation ~]$ oc adm cordon worker01
node/worker01 cordoned

    4. Verify that the node is in the SchedulingDisabled status.

      [student@workstation ~]$ oc get nodes
NAME      STATUS                    ROLES                        ...
master01  Ready                     control-plane,master,worker  ...
master02  Ready                     control-plane,master,worker  ...
master03  Ready                     control-plane,master,worker  ...
worker01  Ready,SchedulingDisabled  worker                       ...
worker02  Ready                     worker                       ...

    5. Drain the identified node of its workloads.

      [student@workstation ~]$ oc adm drain worker01 --ignore-daemonsets=true \
  --delete-emptydir-data --force
node/worker01 already cordoned
...output omitted...
node/worker01 drained

    6. Return the command-line window that executes the loop.sh file. Notice that only the www1 VM responds to requests.

      ...output omitted...
Welcome to www1
Welcome to www1
Welcome to www1
Welcome to www1
...output omitted...

    7. Return to the command-line window where you ran the oc adm drain command. Retrieve the status of the www2 VMI.

      [student@workstation ~]$ oc get vmi
NAME             AGE     PHASE       IP           NODENAME   READY
mariadb-server   3m38s   Running     10.10.0.11   master03   True
www1             19m     Running     10.8.2.63    worker02   True
www2             8m30s   Succeeded   10.9.2.71    worker01   False

      The www2 VM does not automatically fail over to another cluster node.

  8. Configure the www2 VM to automatically fail over to another cluster node during node failure, by setting a LiveMigrate eviction strategy for the VM. Then, recover the VM from the failed node and confirm that the VM responds to requests.

    1. From the OpenShift web console, navigate to Virtualization → VirtualMachines. Select the www2 VM and click the Configuration menu.

    2. Navigate to the Scheduling section, and click None in the Eviction strategy subsection. Click the LiveMigrate flag and click Save.

    3. Click Actions → Start to power on and reschedule the VM on another node in the cluster. Wait a few moments before you proceed to the next step.

    4. From a command-line window, confirm that the www2 VMI is scheduled on another node.

      [student@workstation ~]$ oc get vmi
NAME             AGE     PHASE     IP           NODENAME   READY
mariadb-server   7m45s   Running   10.10.0.11   master03   True
www1             23m     Running   10.8.2.63    worker02   True
www2             45s     Running   10.9.0.46    master02   True

    5. Return to the command-line window that executes the loop.sh file. Confirm that the www2 VM responds to requests. Press Ctrl+C to stop the command, and then close the command-line window.

      ...output omitted...
Welcome to www1
Welcome to www1
Welcome to www1
Welcome to www2
Welcome to www1
Welcome to www2
Welcome to www2
Welcome to www1
Welcome to www1
Welcome to www2
...output omitted...

Evaluation

As the student user on the workstation machine, use the lab command to grade your work. Correct any reported failures and rerun the command until successful.

[student@workstation ~]$ lab grade ha-review

Finish

As the student user on the workstation machine, use the lab command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises.

[student@workstation ~]$ lab finish ha-review

PreviousNext
Revision: do316-4.14-d8a6b80