Red Hat High Availability Clustering for SAP Solutions

After completing this section, you should be able to describe the role of fencing and its involvement in the takeover process to maintain integrity and high availability.

When a node in the cluster fails, it can lead to a loss of data integrity in the cluster, because the node could still have access to the cluster resources. For this reason, it is necessary for an external method to restrict the access of the unresponsive node to the cluster resources. This method is called fencing. The Red Hat Enterprise Linux High Availability Add-On uses fencing to ensure data integrity in the cluster.

Powering off the node often accomplishes fencing, because a dead node cannot do anything. In other cases, fencing uses a combination of operations to cut off the node from the network (to stop new work from arriving), or from storage (to stop the node from writing to shared storage). Fencing is a necessary step in service and resource recovery in a cluster. The Red Hat Enterprise Linux High Availability Add-On does not start resource and service recovery for an unresponsive node until the cluster has fenced that node. For fencing to be correctly configured in your cluster, every node in the cluster must be able to fence every other node.

Fencing Mechanism Overview

Fencing comes in two methods: power fencing and storage fencing. Both fencing methods require a fence device, such as a power switch or the virtual fencing daemon, and fencing agent software to enable communication between the cluster and the fencing device. When the cluster needs to fence a node, it delegates the operation to the fence agent. In turn, the fence agent contacts the fence device to perform the action. In environments where SAP HANA databases are managed within the cluster, it is recommended to use the power fencing method only, because SAP HANA databases are in-memory, and so the storage fencing method would not be useful.

Describing Power Fencing:

Power fencing entails cutting off power to a server.

Two types of power fencing devices exist:

• External fencing hardware that cuts off the power, such as a network-controlled power strip.

• Internal fencing hardware, such as iLO, DRAC, IPMI, or virtual machine fencing, which powers off the hardware of the node.

Configuration of the power fencing can turn off the target machine and keep it off, or turn it off and then on again. Turning a machine back on has the added benefit that it should come back up cleanly, and rejoin the cluster if the cluster services are enabled.

The following graphic shows an example of power fencing with a network-controlled power controller and two power supplies in a server.

Figure 6.1: Power fencing with two power supplies

Warning

When using power fencing, if the machine has more than one power supply, then all power supplies should turn off before being turned on again. Otherwise, the fenced machine never turns off, because it does not lose power. Doing so could cause the machine to be apparently fenced without being fenced, and risking data loss.

Describing Storage Fencing

Storage fencing entails disconnecting a machine from storage at the storage level. It can be done by closing ports on a Fibre Channel switch, or by using SCSI reservations. If a machine is fenced only with storage fencing without combining it with power fencing, the administrator must ensure that it joins the cluster again. It is usually done by rebooting or power-cycling the failed node.

The following graphic shows an example of storage fencing by using multipathed Fibre Channel storage.

Figure 6.2: Storage fencing with multipathed Fibre Channel storage and a fiber switch

It is possible to combine fencing methods. When a node needs to be fenced, one fence device can cut off Fibre Channel by blocking ports on a Fibre Channel switch, and an iLO card can then power-cycle the offending machine. Multiple fencing methods can act as a backup for each other. For example, the cluster nodes are first fenced by power fencing, and if that method fails, with storage fencing.

Fencing is a requirement for every operational cluster. When setting up fencing for the cluster, the first step is to set up the hardware or software device that performs the actual fencing. The Red Hat Enterprise Linux High Availability Add-On provides various fencing agents to use with different fence devices. You can install the most-used fencing agents that Red Hat supports with the fence-agents-all package. The pcs stonith list provides a list with the name and description of all the installed fencing agents:

[root@node ~]# pcs stonith list
fence_amt_ws - Fence agent for AMT (WS)
fence_apc - Fence agent for APC over telnet/ssh
fence_apc_snmp - Fence agent for APC, Tripplite PDU over SNMP
fence_bladecenter - Fence agent for IBM BladeCenter
...output omitted...

Depending on the fence device and fencing agent in use, the required parameters are different. Fencing of a cluster node is successful only if communication occurs between the fencing agent and the fencing device. Moreover, the fencing agent must pass the required set of parameters to the fencing device.

A man page, which describes the fencing agent parameters, is available in the system for every shipped fencing agent. It is also possible to list the parameters for a specific fencing agent by executing the pcs stonith describe fence_agent command.

[root@node ~]# pcs stonith describe fence_rhevm
fence_rhevm - Fence agent for RHEV-M REST API

You can use the fence_rhevm I/O fencing agent with RHEV-M REST API to fence virtual machines.

Stonith options:
ip (required): IP address or hostname of fencing device
ipport: TCP/UDP port to use for connection with device
...output omitted...

In the preceding example, the description shows that the fence_rhevm fence agent is appropriate to fence virtual machines by using the Red Hat Virtualization Manager REST API. This fence agent requires the ip parameter, with an optional ipport parameter.

Distinct fencing devices require different hardware and software configurations. The hardware must always be set up and configured. For the software configuration, it is necessary to document various configuration parameters. The fencing agent uses these documented parameters. The following examples are of different fencing devices, and their hardware and software configurations.

Fencing is crucial for an operational cluster. A cluster administrator must test the fencing setup thoroughly. It is possible to test the fencing device setup by calling fencing agents from the command line. All fencing agents reside in /usr/sbin/fence_*. The fencing agents typically take a -h option to show all available options. You can also use pcs stonith describe fence_agent to investigate possible options. The required options to test fencing differ from agent to agent.

For example, to show all available options for the fence_ipmilan fencing agent, you can use the following command:

[root@node ~]# fence_ipmilan -h
Usage:
        fence_ipmilan [options]
Options:
-a, --ip=[ip]
-l, --username=[name]
-p, --password=[password]
-P, --lanplus
-A, --auth=[auth]
...output omitted...

Then, for a fencing device at the 192.168.100.101 IP address, you can use the fence_ipmilan command to fence the node that is connected to that fence device:

[root@node ~]# fence_ipmilan --ip=192.168.100.101 \
> --username=admin --password=password
Success: Rebooted

A system administrator can create a fence device in the cluster with the pcs stonith create command:

[root@node ~]# pcs stonith create name fencing_agent [fencing_parameters]

The command requires additional arguments:

As stated in the previous section, you must select the appropriate fencing agent for your fence device. The pcs stonith list command provides a list with the name and a description of all the installed fencing agents.

All the fencing agents that are shipped with Red Hat Enterprise Linux High Availability Add-On have generic properties:

pcmk_reboot_timeout

This setting defines the time to wait for fencing to complete in seconds. The default value is 60 seconds, but you can define a device-specific timeout with the pcmk_reboot_timeout property. If a fencing action takes longer than this timeout, then the cluster considers that the fence operation failed. You can set this value at the cluster level with pcs property set stonith-timeout=180s.

pcmk_host_list

This parameter provides a space-separated list of nodes that the fencing device controls. It is required if pcmk_host_check is set to static-list.

pcmk_host_map

You usually use this parameter with fence agents that you can use to fence multiple nodes. For example, with the fence_rhevm agent, you create only one STONITH resource to fence all the cluster nodes. You list the nodes in pcmk_host_map and associate each one with its Red Hat Virtualization VM name. The list is a semicolon-separated list of nodename:port mappings, such as node1.example.com:1;node2.example.com:2.

pcmk_host_check

This parameter defines how the cluster determines the nodes that might be controlled from the fencing device. The following values are possible:

The default setting for pcmk_host_check is dynamic-list, but it switches to static-list whenever the pcmk_host_map or pcmk_host_list option is used.

If you do not set any of the pcmk_host_* options, then the cluster defaults to querying the fence device for a list of ports. If a port name matches the name of a cluster node, then the cluster uses that port to fence that device. You can also create a fencing device for a single node by setting the pcmk_host_list parameter to the name of the node to fence.

In addition to the previously listed generic fencing properties, fencing agent-specific properties exist. The pcs stonith describe fence_agent command shows all required and optional parameters that you can set for a particular fence device:

[root@node ~]# pcs stonith describe fence_apc
Stonith options for: fence_apc
ip (required): IP address or hostname of fencing device
username (required): Login name
password: Login password or passphrase
...output omitted...

To create a fencing device by using the fence_apc fencing agent, you can use the following command:

[root@node ~]# pcs stonith create fence_node1 fence_apc ip=10.168.0.101 \
> username=admin password=s3cr3t pcmk_host_list="node1.example.com"

[root@node ~]# pcs stonith status
* fence_node1 (stonith:fence_apc):
* fence_node2 (stonith:fence_apc):
* fence_node3 (stonith:fence_apc):
Started node1.example.com
Started node2.example.com
Started node3.example.com

[root@node ~]# pcs stonith config fence_node1
Resource: fence_node1 (class=stonith type=fence_apc)
Attributes: ip=10.168.0.101 password=s3cr3t pcmk_host_list=node1.example.com
username=admin
Operations: monitor interval=60s (fence_node1-monitor-interval-60s)

[root@node ~]# pcs stonith update fence_node2 pcmk_host_list=node2.example.com

[root@node ~]# pcs stonith delete fence_node4
Attempting to stop: fence_node4...Stopped
Deleting Resource - fence_node4