Red Hat Enterprise Linux Diagnostics and Troubleshooting

Bookmark this page
P1234567891011
PreviousNext

Resolving Virtualization Issues

Objectives

  • Identify and resolve virtualization issues.

Virtualization Issues

Kernel-based Virtual Machine (KVM) is a virtualization technology that is available across all Red Hat products. KVM provides the virtualization kernel drivers that turn a Linux host into a hypervisor that can run multiple, isolated virtual machines (VMs). User-space components of virtualization include the QEMU emulator that simulates a virtualized hardware platform and manages resources between the host and VMs. The libvirt collection of tools provides an interface for management and communication to help to interact with QEMU for configuring and running VMs.

Issues can arise when running multiple virtual machines with KVM and libvirt. Some issues relate to hardware or firmware, while others are configuration issues. This section identifies some issues, their symptoms, and solutions.

Hardware Virtualization Support

KVM requires hardware virtualization support in both the CPU and the host system firmware. To identify support for KVM, search the flags section of the /proc/cpuinfo file. For Intel machines, the vmx flag is present. For AMD-based machines, the svm flag is present.

View the CPU flags for each processor on the host machine:

[root@host ~]# egrep 'processor|vmx|svm' /proc/cpuinfo
processor	: 0
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand
...output omitted...
processor	: 1
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand
...output omitted...

If the virtualization flag is available, then you can test support for KVM by loading the kvm-intel or kvm-amd kernel module manually. If the module loads without errors, or is already loaded, then hardware virtualization support should be available. If the module load fails with an error message, then either the CPU does not support hardware virtualization or the feature is disabled in the host system firmware:

[root@host ~]# modprobe -v kvm-intel
modprobe: ERROR: could not insert 'kvm_intel': Operation not supported

Use the virsh capabilities command to check for hardware virtualization support. The command lists all supported virtual machine types on the host system.

[root@host ~]# virsh capabilities
<capabilities>

  <host>
...output omitted...
    <cpu>
      <arch>x86_64</arch>
      <model>Cascadelake-Server-noTSX</model>
      <vendor>Intel</vendor>
      <microcode version='83898374'/>
      <topology sockets='2' dies='1' cores='1' threads='1'/>
      <feature name='ss'/>
      <feature name='vmx'/> 1
...output omitted...
  </host>
...output omitted...
  <guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'> 2
      <wordsize>64</wordsize>
      <emulator>/usr/libexec/qemu-kvm</emulator>
...output omitted...
      <machine maxCpus='384'>pc-q35-rhel8.2.0</machine>
      <machine canonical='pc-q35-rhel8.2.0' maxCpus='384'>q35</machine>
...output omitted...
      <domain type='qemu'/> 3
      <domain type='kvm'/> 4
    </arch>
    <features>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='on' toggle='no'/>
    </features>
  </guest>

</capabilities>

1

The vmx feature indicates that support for kvm-intel is available.

2

The architecture name indicates the type of hardware that is virtualized; in this case, a 64-bit x86 machine. In some cases, the output contains multiple architecture names; for example, both 32-bit and 64-bit VMs can be emulated.

3

This domain type indicates that this type of machine can be emulated with a qemu software CPU. Use of emulation can be measurably slower than full hardware virtualization.

4

This domain type indicates support for hardware virtualization by using KVM. If the value is missing, then this type of machine cannot be virtualized in hardware and requires emulation.

Important

If hardware virtualization is not available, and cannot be turned on in the host system firmware, then virtualized machines run on an emulated processor.

Using Overcommitted Resources

Administrators can use overcommit features in libvirt to assign more virtual resources to VMs than are physically available on the host system.

When overcommit limitations are respected, overcommitting resources is efficient and should not negatively affect system performance.

  • VM CPU usage is rarely 100%, and typically less than 50% in many use cases. Assigning more virtual CPUs than physical cores is a recommended practice for increasing CPU utilization, within sensible limits.

  • Memory can be overcommitted, and virtual machine memory can be swapped out to disk, or usage can be compressed by Kernel Samepage Merging (KSM), where duplicate memory pages are reduced to a single page and split again with Copy-on-Write (COW) when one VM writes to that page.

  • Sparse disk images can have a larger total size than the available space on the underlying storage, provided that the size in use cannot exceed the available physical space on the host storage.

These overcommit methods should not impact the overall performance of the running VMs, unless a resource becomes stressed. For example, if many virtual machines used 100% of their virtual CPU allocation at the same time, the VMs would compete for physical CPU time and experience queue delays due to CPU saturation.

When resources become scarce, overall performance and latency can significantly decrease. The host must balance the available resources between the VMs, while also using those resources for the balancing.

Viewing Resource Use

KVM virtual machines are implemented as regular processes on the host. To view resource usage, use normal performance metrics tools such as top. More specialized tools are also available in the form of virt-manager, and various virsh subcommands, such as virsh nodecpustats, virsh nodememstats, and virsh dommemstats <DOMAIN>.

Other monitoring tools, such as collectd and web console, have plug-ins to monitor virtual machine resource usage.

Troubleshooting Scarce Resources

Several approaches are used to resolve scarce resources:

  • Scale up by adding more resources, with more memory, CPU, disk space, or other resources.

  • Add more hypervisor hosts to scale out the virtual machines.

  • Limit the resource usage of specific virtual machines, either by reducing their allocation or by imposing limits by using cgroups.

  • Disable or power off unnecessary or noncritical applications or virtual machines until resource usage returns to a manageable state.

Libvirt XML Configuration

The libvirt service stores virtual machine definitions and related configuration as XML files. You can validate these XML files with the Relax NG schemas in the /usr/share/libvirt/schemas/ directory.

If files were updated with libvirt tools such as virsh and virt-manager only, and not by direct file editing, then the files should successfully validate against the provided schema files. Manually changing files under the /etc/libvirt/ directory is not recommended, as this approach can introduce configuration issues.

Validate the syntax of an XML file, and then validate the file against the libvirt schema:

  • To ensure that the file is valid XML, use the xmllint tool.

    [root@host ~]# xmllint --noout FILENAME

    It is recommended to solve errors in the xmllint reported order, because earlier failures commonly cause later related failures that are not the real problem source.

  • After the file validates as well-structured XML, use the virt-xml-validate tool to validate conformity with a schema. The tool parses the XML file to identify its type of configuration, and then validates it against the correct schema.

    [root@host ~]# virt-xml-validate FILENAME

Libvirt Networking

Libvirt uses software bridges to provide virtual networks to virtual machines. These bridges act as virtual switches, connecting all virtual network interfaces that are assigned to them.

Networks are defined in files in the /etc/libvirt/qemu/networks/ directory, and can be configured to autostart by adding a symbolic link in the /etc/libvirt/qemu/networks/autostart/ directory. Use tools such as virsh or virt-manager to edit these files and to avoid common mistakes during manual editing.

Apart from virtual networks created by the libvirt command, administrators can use NetworkManager to configure regular bridges with one or more physical network interfaces assigned to them.

Virtual Networking Issues

If networking on virtual machines is not working as expected, various issues might be the cause:

A virtual machine is unreachable from the outside

If a connection from outside the hypervisor machine to a virtual machine cannot be established, the following issues might be the cause:

  • The virtual network is operating in NAT mode.

  • A firewall on the hypervisor, or on the VM, is blocking connections.

  • The client machine is missing a defined route to reach the VM.

    The outside world is unreachable from the virtual machine

    If the virtual machine itself cannot reach the outside world, the following issues might be the cause:

  • The virtual network is operating in isolated mode.

  • A firewall rule on the hypervisor might be blocking outgoing connections.

    Connection issues in both directions

    To allow network traffic to virtual networks of the type NAT and routed, the libvirt command creates firewall rules with iptables. If an administrator clears all firewall rules, or adds blocking rules at the top of chains, an interruption of regular network traffic to and from virtual machines might occur. These issues might be solved by restarting the libvirtd service, or by restarting the individual libvirt networks that encountered the error.

References

For more information, refer to the Configuring Virtual Machine Network Connections chapter in the Red Hat Enterprise Linux 8 Virtualization Guide at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_and_managing_virtualization/index#configuring-virtual-machine-network-connections_configuring-and-managing-virtualization

For further information, refer to the Viewing Information About Virtual Machines chapter in the Red Hat Enterprise Linux 8 Virtualization Guide at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/configuring_and_managing_virtualization/index#viewing-information-about-virtual-machines_configuring-and-managing-virtualization

PreviousNext
Revision: rh342-8.4-6dd89bd