Managing Virtual Machines with Red Hat OpenShift Virtualization

RHOCP provides a monitoring stack for cluster resources that is based on the open source Prometheus project.

Users can view details by using logging and monitoring dashboards for each VM in the OpenShift web console.

To view a VM's metrics, such as resource utilization and cluster-level events, use the OpenShift web console to navigate to Virtualization → VirtualMachines. Select the VM's project from the Projects menu and then click the VM's name to open the Overview tab for the VM.

The Details card in the Overview tab provides identifying information about the VM, including its name, status, date of creation, hostname, and operating system.

The Utilization card includes CPU, Memory, Storage, and Network Transfer usage charts. From this card, cluster administrators can select a duration for the utilization data.

The Overview tab provides additional cards to summarize hardware devices, alerts, snapshots, network interfaces, and disks.

In the Developer perspective, cluster administrators can access the Observe menu to review dashboards with information about each project's resource utilization, such as usage metrics for CPU, memory, network, and storage I/O.

The oc client is a command-line utility for managing RHOCP resources. Administrators and developers can use the oc command to develop, build, deploy, and run applications or VM workloads in Red Hat OpenShift. You can review and diagnose events and conditions for VM and storage resources. For example, cluster administrators can review a VM's logs from the virt-launcher pod to debug issues by using the oc logs command:

[user@host ~]$ oc logs virt-launcher-name

Cluster administrators can also use the oc get and the oc describe commands to retrieve the status of all VMs in the cluster or a specific VM in a namespace.

As a cluster administrator, you can use the oc get command with the -A flag to list the status of all VMs in the cluster:

[user@host ~]$ oc get vm -A

Use the following oc get command to retrieve the status of all VMs in a specific namespace:

[user@host ~]$ oc get vm -n namespace

You can also use the oc describe command to retrieve detailed information for a VM, such as recent events and specifications:

[user@host ~]$ oc describe vm vm-name -n namespace

As a cluster administrator, you can manage and inspect a VM by using virsh commands within the VM's virt-launcher pod and its libvirtd container.

Execute virsh commands to manage a VM from the console of the VM's virt-launcher pod by using either the OpenShift web console or the oc client. From the OpenShift web console, navigate to Workloads → Pods and select the VM's virt-launcher pod. Click the Console tab to open the pod's console. Use the following command to access the virt-launcher pod's console with the oc client:

[user@host ~]$ oc exec -it virt-launcher-pod-name -n namespace -- bash

RHOCP 4.14 provides functions to the virtctl command that incorporate several virsh commands, to reduce the need to access the virt-launcher pod's console. For example, as a cluster administrator, you can view a list of file systems that are available to the VMI with the virtctl fslist vmi-name command.

[user@host ~]$ virtctl fslist production-mariadb -n production-vms
{
  "metadata": {},
  "items": [
    {
      "diskName": "vda2",
      "mountPoint": "/boot/efi",
      "fileSystemType": "vfat",
      "usedBytes": 6006784,
      "totalBytes": 104634368
    },
    {
      "diskName": "vda3",
      "mountPoint": "/",
      "fileSystemType": "xfs",
      "usedBytes": 2419097600,
      "totalBytes": 10619924480
    }
  ]
}

Additionally, you can view guest agent information about the VMI's operating system with the virtctl guestosinfo vmi-name command.

As a cluster administrator, you can monitor and query metrics on all projects in the cluster. RHOCP uses Role Based Access Control (RBAC) to grant permissions to cluster resources, including project monitoring.

RBAC is a technique for managing access to computer system resources. In Red Hat OpenShift, RBAC determines whether a user can perform certain actions within the cluster or project. You can choose between two role types, cluster or local, depending on the user's level of responsibility.

The authorization process is managed by rules, roles, and bindings.

RHOCP defines two groups of roles and bindings according to the user's scope and responsibility: cluster roles and local roles.

By default, developer users can monitor only projects where they have the view role permissions. Developer users can view monitoring information in the OpenShift web console by navigating to the Observe → Metrics menu. From there, a developer can run performance metric queries, one at a time.

Cluster administrators can grant permissions for additional projects to developer users.

As a cluster administrator, use the following command to grant view role permissions to a user:

[user@host ~]$ oc create rolebinding view --clusterrole=view --user=user -n project

To grant the view role permissions to a group, run the following command:

[user@host ~]$ oc create rolebinding view --clusterrole=view --group=group -n project

Enable monitoring for user-defined projects by setting the enableUserWorkload: true field in the cluster-monitoring-config configuration map in the openshift-monitoring namespace.

Cluster administrators can make this change:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true

Only cluster administrators have access to all namespaces concurrently in the Observe → Metrics interface. From the Developer perspective, you can navigate to Observe and then click the Dashboard tab to review various metrics, including CPU and memory usage charts, current network usage, bandwidth, and storage I/O for individual projects.

Figure 2.18: Developer view of monitoring dashboard

As a cluster administrator, you can navigate to Observe → Dashboards and then select the KubeVirt / Infrastructure Resources / Top Consumers dashboard from the Dashboard list. You can use this dashboard to review metrics for all VMs in the cluster and to identify the VMs that are consuming the most resources.

Figure 2.19: Administrator view of monitoring dashboard

You can also access the Observe → Metrics interface to execute Prometheus Query Language (PromQL) queries against an entire Red Hat OpenShift Virtualization environment. Use this tool to examine information about the cluster and user-defined workloads as a dashboard or a graph.

Various metrics can help administrators and developers when writing PromQL queries for virtualization monitoring: