DO316 - ch02s07

Bookmark this page

Lab: Create, Configure and Monitor a Virtual Machine

Create VMs in different projects and retrieve metrics to monitor the resource consumption of the VMs.

Outcomes

Grant permissions to users and groups to create, manage, and monitor VMs.
Create VMs through the OpenShift web console and the command line.
Inspect the configuration of a VM.
Access the console of a VM through the OpenShift web console and the virtctl client.
Monitor the resource consumption of a VM.

As the student user on the workstation machine, use the lab command to prepare your system for this exercise.

[student@workstation ~]$ lab start accessing-review

Instructions

The database-admins group must create and manage the development and staging versions of a MariaDB VM to gather performance metrics. The image for the MariaDB VMs exists within the vm-images project.

As the cluster administrator, grant the admin role to the database-admins group within the development-db and staging-db projects. You must also grant the view role to the database-admins group within the vm-images project to permit the database-admins group access to the mariadb Persistent Volume Claim (PVC).

As a member of the database-admins group, create the dev-mariadb and staging-mariadb VMs in their respective projects by using the mariadb PVC in the vm-images project. Permit the developer user to view and access the dev-mariadb VM and its performance metrics. Finally, use the collected metrics to determine whether the VMs are experiencing resource contention.

Log in to RHOCP as the admin user with redhatocp as the password. List the role bindings in the development-db and staging-db projects.

Use the command line to log in to your Red Hat OpenShift cluster as the admin user.

[student@workstation ~]$ oc login -u admin -p redhatocp \
\https://api.ocp4.example.com:6443
Login successful.
...output omitted...

Retrieve the role bindings for the development-db project.

[student@workstation ~]$ oc get rolebinding -n development-db
NAME                    ROLE                               AGE
system:deployers        ClusterRole/system:deployer        2m
system:image-builders   ClusterRole/system:image-builder   2m
system:image-pullers    ClusterRole/system:image-puller    2m

Retrieve the role bindings for the staging-db project.

[student@workstation ~]$ oc get rolebinding -n staging-db
NAME                    ROLE                               AGE
system:deployers        ClusterRole/system:deployer        2m
system:image-builders   ClusterRole/system:image-builder   2m
system:image-pullers    ClusterRole/system:image-puller    2m

Add the admin role binding for all resources in the development-db and staging-db projects, and the view role binding for all resources in the vm-images project to the database-admins group.

Add the admin role binding on all resources in the development-db project. Confirm that the role binding is successfully applied to the project.

[student@workstation ~]$ oc create rolebinding admin --clusterrole=admin \
  --group=database-admins -n development-db
rolebinding.rbac.authorization.k8s.io/admin created
[student@workstation ~]$ oc get rolebinding -n development-db
NAME                    ROLE                               AGE
admin                   ClusterRole/admin                  26s
system:deployers        ClusterRole/system:deployer        4m
system:image-builders   ClusterRole/system:image-builder   4m
system:image-pullers    ClusterRole/system:image-puller    4m

Add the admin role binding on all resources in the staging-db project. Confirm that the role binding is successfully applied to the project.

[student@workstation ~]$ oc create rolebinding admin --clusterrole=admin \
  --group=database-admins -n staging-db
rolebinding.rbac.authorization.k8s.io/admin created
[student@workstation ~]$ oc get rolebinding -n staging-db
NAME                    ROLE                               AGE
admin                   ClusterRole/admin                  20s
system:deployers        ClusterRole/system:deployer        5m
system:image-builders   ClusterRole/system:image-builder   5m
system:image-pullers    ClusterRole/system:image-puller    5m

Add the view role binding on all resources in the vm-images project. Confirm that the role binding is successfully applied to the project.

[student@workstation ~]$ oc create rolebinding view --clusterrole=view \
  --group=database-admins -n vm-images
rolebinding.rbac.authorization.k8s.io/view created
[student@workstation ~]$ oc get rolebinding -n vm-images
NAME                    ROLE                               AGE
allow-clone-t-user      ClusterRole/datavolume-cloner      6m
system:deployers        ClusterRole/system:deployer        6m
system:image-builders   ClusterRole/system:image-builder   6m
system:image-pullers    ClusterRole/system:image-puller    6m
view                    ClusterRole/view                   25s

Log in to the OpenShift web console as the dbadmin1 user with redhatocp as the password. Create a VM with the Red Hat Enterprise Linux 8 VM template in the development-db project. Select the mariadb PVC in the vm-images project as the boot source. Name the VM dev-mariadb and assign 1 CPU and 2 GiB memory to the VM. Customize the VM to create the developer user with developer as the password.
Open a web browser and navigate to https://console-openshift-console.apps.ocp4.example.com. Click htpasswd_provider and log in as the dbadmin1 user with redhatocp as the password.
Change to the Administrator perspective and navigate to Virtualization → VirtualMachines. Select the development-db project from the Projects list.
Select the Create VirtualMachine → From template menu option.
Click the Red Hat Enterprise Linux 8 VM template.
Click the Customize VirtualMachine button.
Specify dev-mariadb as the VM name.
Select PVC (clone PVC) from the Disk source list. Select the vm-images project from the PVC project list. Select the mariadb PVC from the PVC name list.
Selecting the boot source
Enter developer in the Optional parameters → CLOUD_USER_PASSWORD field. Click the Customize VirtualMachine parameters button to specify additional settings.
Confirm that the network interface type is masquerade by selecting the Network Interfaces tab.
Confirm that the cloudinitdisk and rootdisk disks are attached by selecting the Disks tab.
Select the Scripts tab
Click the Cloud-init Edit button and specify the developer user in the Cloud-init form. Click Apply.
Defining the user account and password
Review the settings and click Create Virtual Machine. Wait a few moments and confirm that the VM's status is Running. Log out of the OpenShift web console.

Use the command line to log in to the OpenShift cluster as the dbadmin1 user with redhatocp as the password. Add the view role binding for the developer user in the development-db project.

[student@workstation ~]$ oc login -u dbadmin1 -p redhatocp
Login successful.

You have access to the following projects and can switch between them with the 'oc project projectname' command:

  * development-db
    openshift-virtualization-os-images
    staging-db
    vm-images

Using project "development-db".

Add the view role binding to the developer user.

[student@workstation ~]$ oc create rolebinding view --clusterrole=view \
  --user=developer
rolebinding.rbac.authorization.k8s.io/view created

Log in to the OpenShift web console as the developer user with developer as the password. Confirm that the dev-mariadb VM is accessible to the developer user. Open the console of the dev-mariadb VM and log in as the developer user with developer as the password.
In the dev-mariadb VM console, use the watch vmstat -S M command to retrieve the performance metrics, in MiB, that the VM's operating system reports. Recall that the dev-mariadb VM is configured with 1 CPU and 2 GiB (or 2048 MiB). The output of the watch vmstat -S M command includes the swpd column to report the used memory in swap spaces. The output also includes the free column to report available free memory and the wa column to report CPU wait.
Open a web browser and navigate to https://console-openshift-console.apps.ocp4.example.com. Click htpasswd_provider and log in as the developer user with developer as the password.
Switch to the Administrator perspective and navigate to Virtualization → VirtualMachines. Select the development-db project from the Projects list.
Click dev-mariadb and click the Console tab.
Log in as the developer user with developer as the password in the VM's VNC console window.
Within the VNC console window, use the watch vmstat -S M command to retrieve the VM's performance metrics that the operating system reports.
[developer@dev-mariadb ~]$ watch vmstat -S M procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 13 2 0 229 0 508 0 0 5447 170256 1190 550 20 34 40 4 0
The wa column reports a value above 0. The free column reports a low value of free memory. The results indicate potential issues with CPU wait and available free memory.
Press Ctrl+C to exit the watch vmstat command. Log out of the dev-mariadb console and close the VNC console window.
[developer@dev-mariadb ~]$ logout
As the developer user, use the OpenShift web console to retrieve the performance metrics of the dev-mariadb VM in the development-db project. Review the memory and CPU usage metrics for the last 15 minutes by using the performance monitoring dashboards.
Switch to the Developer perspective in the OpenShift web console.
Navigate to Observe and confirm that the developement-db project is selected from the Projects list.
Confirm that the Kubernetes / Compute Resources / Namespaces(Pods) dashboard is selected from the Dashboard drop-down menu. Click the Last 15 minutes option from the Time Range menu.
Kubernetes / Compute Resources / Namespaces (Pods) dashboard
Scroll down to review the CPU Usage and CPU Quota charts.
CPU Usage chart
Scroll down to review the Memory Usage and Memory Quota charts.
Memory Usage chart
Log out of the OpenShift web console.

As the dbadmin1 user, use the oc client to deploy the staging-mariadb VM in the staging-db project. Use the resource file at ~/DO316/labs/accessing-review/staging-mariadb.yml to create the VM.

Before deploying the VM, change the file as needed. Specify the mariadb PVC in the vm-images project as the source PVC. Modify the cloudInitNoCloud object to create a local account for the dbadmin2 user with redhatocp as the password.

After creating the VM, wait a few moments and confirm that the VM is running.

Open the ~/DO316/labs/accessing-review/staging-mariadb.yml file in a text editor.
```
[student@workstation ~]$ vi ~/DO316/labs/accessing-review/staging-mariadb.yml
```

Locate the .spec.dataVolumeTemplates.spec.source object. Modify the file to specify mariadb as the source PVC name and vm-images as the source PVC namespace.

...ouput omitted...
  name: staging-mariadb
  namespace: staging-db
spec:
  dataVolumeTemplates:
  - apiVersion: cdi.kubevirt.io/v1beta1
    kind: DataVolume
    metadata:
      name: staging-mariadb
    spec:
      pvc:
        accessModes:
        - ReadWriteMany
        resources:
          requests:
            storage: 10Gi
        storageClassName: ocs-external-storagecluster-ceph-rbd
        volumeMode: Block
      source:
        pvc:
          name: mariadb
          namespace: vm-images
...output omitted...

Locate the spec.template.spec.volumes.cloudInitNoCloud object. Modify the #cloud-config object to create the dbadmin2 user.

...ouput omitted...
      volumes:
      - dataVolume:
          name: staging-mariadb
        name: staging-mariadb
      - cloudInitNoCloud:
          userData: |-
            #cloud-config
            user: dbadmin2
            password: redhatocp
            chpasswd: { expire: False }
...output omitted...

Confirm that all <CHANGE_ME> objects are updated before saving and closing the file.

Use the oc create -f command to create the staging-mariadb VM. Wait a few moments and confirm that the VM's status is Running.

[student@workstation ~]$ oc create -f \
  ~/DO316/labs/accessing-review/staging-mariadb.yml
virtualmachine.kubevirt.io/staging-mariadb created
[student@workstation ~]$ oc get vm -n staging-db
NAME              AGE   STATUS    READY
staging-mariadb   47s   Running   True

Confirm that the dbadmin2 user can access and log in to the staging-mariadb VM. Use the oc client to log in to the OpenShift cluster as the dbadmin2 user with redhatocp as the password. Connect to the staging-mariadb VNC console with the virtctl client and log in as the dbadmin2 user.
In the staging-mariadb console, use the watch vmstat -S M command to retrieve the performance metrics, in MiB, that the VM's operating system. reports Recall that the output of the watch vmstat -S M command includes the swpd column to report the used memory in swap spaces. The output also includes the free column to report available free memory and the wa column to report CPU wait.
Open another command-line window. Log in as the dbadmin2 user with redhatocp as the password.
[student@workstation ~]$ oc login -u dbadmin2 -p redhatocp Login successful. You have access to the following projects and can switch between them with the 'oc project _projectname' command: * development-db openshift-virtualization-os-images staging-db Using project "development-db".
Switch to the staging-db project.
[student@workstation ~]$ oc project staging-db Now using project "staging-db" on server "https://api.opc4.example.com:6443".
Connect to the VNC console of the staging-mariadb VM with the virtctl client.
[student@workstation ~]$ virtctl vnc staging-mariadb
Note
The VM takes several minutes to be completely available. Wait for it before moving to the next step.
Log in as the dbadmin2 user with redhatocp as the password in the staging-mariadb VNC console window.
Accessing the VNC console
Within the VNC console window, use the watch vmstat -S M command to retrieve the VM's performance metrics as reported by the VM's operating system.
[dbadmin2@staging-mariadb ~]$ watch vmstat -S M procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 14 2 0 250 0 422 0 4 1481 52022 750 490 43 13 43 2 0
The wa column reports a value above 0. The free column reports a low value of free memory. The results show potential issues with CPU wait and available free memory.
The developer and dbadmin2 users have both reported performance issues with the dev-mariadb and staging-mariadb VMs. As the admin user, confirm that the VMs are experiencing performance issues. Use the KubeVirt / Infrastructure Resources / Top Consumers dashboard and PromQL queries to retrieve the current performance metrics of both VMs for the last 5 minutes. Identify whether the VMs are waiting for Input/Output (I/O) with the vcpu_wait_seconds metric. Verify whether the VMs have sufficient available memory with the memory_available_bytes metric.
The ~/DO316/labs/accessing-review/queries.txt file contains the necessary queries. Use this file to copy and paste the PromQL queries as needed.
In a new browser window, navigate to https://console-openshift-console.apps.ocp4.example.com. Click htpasswd_provider and log in as the admin user with redhatocp as the password.
If necessary, switch to the Administrator perspective in the OpenShift web console.
Navigate to Observe → Dashboards.
Select KubeVirt / Infrastructure Resources / Top Consumers dashboard from the Dashboard menu. Select Last 5 minutes from the Time Range menu.
KubeVirt top consumers
The charts indicate that the dev-mariadb and staging-mariadb VMs are experiencing issues with memory and CPU.
Navigate to Observe → Metrics. Use PromQL syntax to confirm that the VMs are waiting for Input/Output (I/O) with the vcpu_wait_seconds metric. In the Expression field, type the following query and press Enter to retrieve the performance metric:
topk(3, sum by (name,namespace) (rate(kubevirt_vmi_vcpu_wait_seconds{namespace=~".*-db"}[5m])))
A returned value above 0 indicates that the vCPU wants to run, but the host scheduler cannot run it. Values above 0 indicate a potential issue with I/O and a need to increase available CPU resources to the VMs.
vCPU wait metrics
Identify whether the VMs have available free memory with the memory_available_bytes metric. Click Add query and execute the following query:
topk(3, sum by (name, namespace) (rate(kubevirt_vmi_memory_available_bytes{namespace=~".*-db"}[5m])))
A returned value of 0 indicates that the VMs need more memory.
Available memory metrics
Both the dev-mariadb and the staging-mariadb VMs require an increase in their CPU and memory resources. As the admin user, use the OpenShift web console to increase the available CPU by 1. Increase the available memory by 2 GiB. Restart the VM to apply the changes.
In the OpenShift web console, navigate to Virtualization → VirtualMachines. Select the development-db project from the Project list.
Click the dev-mariadb VM to open the Virtual Machine Overview window.
Click the Details tab and scroll down to the CPU | Memory section.
Click the 1 CPU|2 GiB Memory link to open the edit window.
CPU | Memory settings for dev-mariadb
Enter 2 in the CPU field. Enter 4 GiB in the Memory field and click Save.
Edit CPU and Memory
Click the Action → Restart option to restart the VM. Wait for the VM to reach the Running status.
As the admin user, modify the CPU and memory settings of the staging-mariadb VM. Increase the available CPU by 1. Increase the available memory by 2 GiB. Restart the VM to apply the changes.
In the OpenShift web console, select the staging-db project from the Project list.
Click the staging-mariadb VM to open the Virtual Machine Overview window.
Click the Details tab and scroll down to the CPU | Memory section.
Click the 1 CPU|2 GiB Memory link to open the edit window.
Enter 2 in the CPU field. Enter 4 GiB in the Memory field. Click Save.
Click the Action → Restart option to restart the VM. Wait for the VM to reach the Running status.
As the dbadmin1 user, confirm that the performance metrics improved for the dev-mariadb and staging-mariadb VMs with the watch vmstat -S M command. Use the virtctl client to connect to the VNC console of the VMs. In the console window of each VM, log on as the local user accounts to run the watch vmstat -S M command.
Open a new command-line window and log in to the cluster as the dbadmin1 user.
[student@workstation ~]$ oc login -u dbadmin1 -p redhatocp Login successful.
Use the virtctl client to open the VNC console of the dev-mariadb VM in the developement-db project.
[student@workstation ~]$ virtctl vnc dev-mariadb -n development-db
In the new VNC console window, log in as the developer user with the developer password.
Within the VNC console window, use the watch vmstat -S M command to retrieve the VM's performance metrics that the operating system reports.
[developer@dev-mariadb ~]$ watch vmstat -S M procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 0 1315 3 2120 0 0 109 25068 540 22 17 35 48 0 0
The results reflect the impact of increasing the available memory and CPU resources. Close the console window.
In the command-line window, use the virtctl client to open the VNC console of the staging-mariadb VM in the staging-db project.
[student@workstation ~]$ virtctl vnc staging-mariadb -n staging-db
In the new VNC console window, log in as the dbadmin2 user with redhatocp as the password.
Within the VNC console window, use the watch vmstat -S M command to retrieve the VM's performance metrics that the operating system reports.
[dbadmin2@staging-mariadb ~]$ watch vmstat -S M procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 1756 4 2482 0 0 423 23940 586 146 42 9 48 0 0
The results reflect the impact of increasing the available memory and CPU resources. Close the console window.

Performance testing is finished for the dev-mariadb and staging-mariadb VMs. As the dbadmin1 user, stop the dev-mariadb and staging-mariadb VMs with the virtctl client.

In the command-line window, use the virtctl client to stop the dev-mariadb VM in the development-db project. Wait a few moments and confirm that the VM is stopped.

[student@workstation ~]$ virtctl stop dev-mariadb -n development-db
VM dev-mariadb was scheduled to stop
[student@workstation ~]$ oc get vm -n development-db
NAME          AGE   STATUS    READY
dev-mariadb   25m   Stopped   False

Use the virtctl client to stop the staging-mariadb VM in the staging-db project. Wait a few moments and confirm that the VM is stopped.

[student@workstation ~]$ virtctl stop staging-mariadb -n staging-db
VM staging-mariadb was scheduled to stop
[student@workstation ~]$ oc get vm -n staging-db
NAME             AGE   STATUS    READY
stagingmariadb   27m   Stopped   False

The developer user no longer needs the dev-mariadb VM. Use the oc delete command to delete the VM. Log out of the OpenShift cluster with the oc logout command. You can close any remaining command-line windows.
Delete the VM.
[student@workstation ~]$ oc delete vm/dev-mariadb -n development-db virtualmachine.kubevirt.io "dev-mariadb" deleted
Log out of the OpenShift cluster.
[student@workstation ~]$ oc logout Logged "dbadmin1" out on "\https://api.ocp4.example.com:6443"

Evaluation

As the student user on the workstation machine, use the lab command to grade your work. Correct any reported failures and rerun the command until successful.

[student@workstation ~]$ lab grade accessing-review

Finish

As the student user on the workstation machine, use the lab command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises.

[student@workstation ~]$ lab finish accessing-review

Discuss Managing Virtual Machines with Red Hat OpenShift Virtualization

Go to community

Welcome to the Managing Virtual Machines with Red Hat OpenShift (DO316) course in RHLC!

Syed

11 sie 2023

We are excited to launch a space dedicated to the Red Hat Training course, Managing Virtual Machines with Red Hat OpenShift Virtualization! To gain the most value from this group - click the "Join Group" button in the upper right hand corner of the group home page.We encourage group members to collaborate in this group to discuss topics, ask questions, share best practices and tips, provide course feedback, and share their accomplishments as it relates to DO316.Read more about Managing Virtual Machines with Red Hat OpenShift Virtualization here.

2372

Revision: do316-4.14-d8a6b80

Managing Virtual Machines with Red Hat OpenShift Virtualization

Lab: Create, Configure and Monitor a Virtual Machine

Note