Red Hat Virtualization

Objectives

After completing this section, you should be able to perform maintenance tasks to manage RHHI-V pods and storage.

Maintaining a RHHI-V Pod

Some operational tasks are unique to RHHI-V because of the inclusion of Red Hat Gluster Storage on the hyperconverged hosts. Storage administrators who are already familiar with Gluster will recognize the storage management procedures. This section discusses a few unique RHHI-V tasks, including pod startup and shutdown, and scaling storage using bricks and volumes.

Performing an Ordered Hyperconverged Pod Shutdown

A hyperconverged environment must be shut down in a particular order. The simplest way to do this is to create a shutdown playbook that can be run from the hosted engine virtual machine.

The ovirt.shutdown_env role enables the global maintenance mode, and initiates shutdown for all virtual machines and hosts in the cluster. Host shutdown is asynchronous. The playbook terminates before hyperconverged hosts are actually shut down.

[root@hosta ~]# yum install ovirt-ansible-shutdown-env -y

To create a shutdown playbook for your environment, create a file similar to the following example:

[root@hosta ~]# cat shutdown_rhhi-v.yml
---
- name: oVirt shutdown environment
  hosts: localhost
  connection: local
  gather_facts: false

  vars:
    engine_url: https://ovirt-engine.example.com/ovirt-engine/api
    engine_user: admin@internal
    engine_password: redhat
    engine_cafile: /etc/pki/ovirt-engine/ca.pem

  roles:
    - ovirt.shutdown_env

Run the shutdown playbook against the hosted engine virtual machine.

[root@hosta ~]# ansible-playbook -i localhost shutdown_rhhi-v.yml

Performing an Ordered Hyperconverged Pod Startup

Starting up a hyperconverged cluster is more complex than starting up a traditional compute or storage cluster. Follow these instructions to start your hyperconverged cluster safely.

Power on all hosts in the cluster. Ensure that the required services are available. Verify that the glusterd service started correctly on all hosts. If glusterd is not started, start it.

[root@hosta ~]# systemctl status glusterd
...output ommitted...
[root@hosta ~]# systemctl start glusterd
...output ommitted...

Verify that host networks are available and have IP addresses assigned to required interfaces.

[root@hosta ~]# ip -br addr show
...output ommitted...

Verify that all hosts are in the storage cluster with state Peer in Cluster (Connected).

[root@hosta ~]# gluster peer status
Number of Peers: 2

Hostname: 172.25.250.11
Uuid: 6e1cfcc6-6c21-48f3-882f-3c10f8c73ff3
State: Peer in Cluster (Connected)

Hostname: 172.25.250.11
Uuid: a79122c1-326f-4816-a7cb-082552bf1621
State: Peer in Cluster (Connected)

Verify that all bricks are online..

[root@hosta ~]# gluster volume status engine
Status of volume: engine
Gluster process                                    TCP       RDMA Online  Pid
-------------------------------------------------------------------------------
Brick 172.25.250.10:/gluster_bricks/engine/engine  49153     0     Y      23160
Brick 172.25.250.11:/gluster_bricks/engine/engine  49160     0     Y      12392
Brick 172.25.250.12:/gluster_bricks/engine/engine  49157     0     Y      15200
Self-heal Daemon on localhost                      N/A       N/A   Y      23008
Self-heal Daemon on 172.25.250.11                  N/A       N/A   Y      10905
Self-heal Daemon on 172.25.250.12                  N/A       N/A   Y      13568

Start the hosted engine virtual machine. Run the following command on the host that you want to be the hosted engine node. Verify that the hosted engine virtual machine has started correctly.

[root@hosta ~]# hosted-engine --vm-start
[root@hosta ~]# hosted-engine --vm-status

Take the hosted engine virtual machine out of global maintenance mode. Using the Administration Portal, locate the hosted engine node on the Hosts page. Click Disable Global HA Maintenance to return to normal operation. You may now start other virtual machines using the Web Console.

Managing and Scaling Storage

Managing and scaling available storage is a necessary skill in hyperconverged infrastructures. The embedded Red Hat Gluster Storage is implemented in volumes that span all hosts in the RHHI-V pod. Volumes are built using physical disks configured as bricks, which are evenly distributed across the hyperconverged hosts. Managing storage requires creating and maintaining bricks, and building or restructuring RAID volumes, to be used as RHHI-V data storage domains.

Managing Bricks

Bricks can be created and maintained using the host Web Console, the Administration Portal, or from the command line. Using the Administration Portal is typically the simplest method. The default result is to create a thinly provisioned logical volume on a storage disk device, to be presented to Gluster for use as a brick. On the Hosts page, in the Storage Devices window, click the Create a Brick button, and then give the brick a name and mount point. You can specify if the device presented is a RAID device, and assign a cache device for this disk, which is only useful for legacy hard drives and should not be used for solid state drives.

Figure 14.11: Creating a brick

Bricks must be free of partitions and labels before being configured. You can reconfigure a brick by resetting it, allowing the brick to be used as if the brick is being added to the cluster for the first time. Resetting a brick uses the same UUID, host name, and path as before.

Bricks can be replaced when they fail or for other maintenance reasons. Simply select the volume that contains the brick, and then choose the Replace Brick button to specify the host and directory of the replacement brick. Bricks can also be removed from a volume, but the volume must be stopped, and the brick unused, before the brick can be removed.

Managing Volumes

A volume is a logical set of bricks, where each brick is an export directory on a hyperconverged host in the pod storage pool. To create a volume in the Administration Portal, navigate to the Storage page. In the Volumes window, click the New Volume button, and then select the volume cluster. Give the volume a name and type, such as as Replicated or Distributed, and then select at least 3 bricks to use in the volume. When using arbitrated volumes, select the Arbiter check box to configure one or more arbiter bricks for the volume.

Figure 14.12: Creating a brick

After volumes have been created, they must be started. In the Volumes window, select the volume and click the Start button. Stopping a volume is done from the same Volumes window using the Stop button.

Scaling Volumes

Volumes can be expanded by adding bricks. Bricks must be distributed evenly across the hyperconverged hosts in the RHHI-V pod. Prepare bricks on each host, and then add them in multiples of the replica count of the volume. After increasing the bricks in a volume, rebalance the volume to distribute data more equally across all bricks, which will enhance performance of the volume.

Volumes can also be made smaller by removing bricks. Stop the volume before attempting to remove bricks. The volume must be stopped to keep the storage routines from attempting to balance data to a brick while you are trying to remove data from that brick. Bricks must also be removed in multiples of the replica count of the volume. After removing bricks, the volume should be rebalanced for best performance.

Rebalancing Volumes

After expanding or shrinking a volume, rebalance the data among the hosts. In non-replicated volumes, all bricks must be online to perform the rebalance operation. In a replicated volume, at least one of the bricks in the replica must be online. Start the rebalance process from the Volumes window. The time needed to rebalance a volume depends on the volume size and the current activity on the volume.

Volumes can be deleted from the Volumes page. Stop the volume first, then click the Remove button, and then confirm that the volume should be removed. Volume data remains on the disk devices until the volumes are wiped or repurposed.