DO380 - ch03s07

Bookmark this page

Lab: Cluster Partitioning

Apply operating system settings to cluster nodes with the machine configuration operator, and with higher-level operators instead of the low-level machine configuration operator.

Outcomes

Create machine configuration pools (MCPs) to apply custom configurations to some nodes in the cluster.
Use the node tuning operator (NTO) to enable non-uniform memory access (NUMA) balancing.
Use the machine configuration operator (MCO) to enable a real-time kernel.

As the student user on the workstation machine, use the lab command to prepare your environment for this exercise, and to ensure that all required resources are available.

[student@workstation ~]$ lab start nodes-review

Instructions

Your company has a workload that requires many CPUs and much memory. Thus, the bandwidth between the CPUs and the memory is limiting the performance of the workload. In your cluster, the nodes with the numa role have NUMA hardware systems. These nodes perform best when the threads of their processes are accessing memory on the same NUMA node as where the threads are scheduled. Thus, your company requires you to enable the NUMA balancing in the kernel through the NTO for the nodes with that role. In this classroom environment, only one node has the numa role due to the classroom size. However, no nodes in the cluster have real NUMA hardware systems.

Your company also needs to deploy a workload for a 5G core user-plane function. This workload requires a real-time kernel to ensure low latency. Although using a PerformanceProfile custom resource (CR) is the recommended approach for configuring low-latency workloads which require real-time kernels, in this exercise you must use the MCO instead to practice by using its syntax. The NTO creates, among other resources, a similar machine configuration (MC) to the one that you create manually in the exercise. Although in production environments, this application type is deployed in multiple nodes and the kernel configuration must be applied to all of them, in this exercise you apply the configuration only to one node due to the classroom size.

Use the admin user with redhatocp as the password. For the resources that you must create, you can use the incomplete CR YAML files in the ~/DO380/labs/nodes-review directory.

List the labels for the nodes in the cluster. Verify that the worker01 node has the numa role. Create the numa-bal MCP to include in the pool the nodes with the numa role. This MCP must select MCs with the numa-bal and worker labels. You can find an incomplete example for the MCP CR in the ~/DO380/labs/nodes-review/numa-mcp.yml file.

Connect to the OpenShift cluster as the admin user with redhatocp as the password.

[student@workstation ~]$ oc login -u admin -p redhatocp \
  https://api.ocp4.example.com:6443
Login successful.
...output omitted...

List the labels for the nodes. Verify that the worker01 node has the numa role.

[student@workstation ~]$ oc get nodes
NAME       STATUS  ROLES                  AGE    VERSION
master01   Ready   control-plane,master   49d    v1.25.7+eab9cc9
master02   Ready   control-plane,master   49d    v1.25.7+eab9cc9
master03   Ready   control-plane,master   49d    v1.25.7+eab9cc9
worker01   Ready   numa,worker            2d4h   v1.25.7+eab9cc9
worker02   Ready   worker                 2d4h   v1.25.7+eab9cc9
worker03   Ready   worker                 2d4h   v1.25.7+eab9cc9

Change to the ~/DO380/labs/nodes-review directory.

[student@workstation ~]$ cd ~/DO380/labs/nodes-review

Create the MCP CR YAML file. You can find an incomplete example for the CR in the ~/DO380/labs/nodes-review/numa-mcp.yml file.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: numa-bal
spec:
  machineConfigSelector:
    matchExpressions:
      - key: machineconfiguration.openshift.io/role
        operator: In
        values: [worker,numa-bal]
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/numa: ""

Apply the configuration for the MCP CR.

[student@workstation nodes-review]$ oc create -f numa-mcp.yml
machineconfigpool.machineconfiguration.openshift.io/numa-bal created

Verify that the numa-bal MCP is correctly created. Wait until the MCO updates the node and marks the UPDATED field as True. You might have to repeat this command many times.

[student@workstation nodes-review]$ oc get mcp
NAME      CONFIG                UPDATED  UPDATING  DEGRADED  MACHINECOUNT  ...
master    rendered-master-...   True     False     False     3             ...
numa-bal  rendered-numa-ba...   True     False     False     1             ...
worker    rendered-worker-...   True     False     False     2             ...

Verify that the worker01 node is assigned for the numa-bal MCP, and that the MCO applies the rendered MC to the node.

[student@workstation nodes-review]$ oc get events \
  -n openshift-machine-config-operator \
  --sort-by='{.lastTimestamp}' \
  --field-selector involvedObject.name=numa-bal
...  REASON                   OBJECT                       MESSAGE
...  RenderedConfigGenerated  machineconfigpool/numa-bal   rendered-numa-bal-2f56...81d0 successfully generated ...
...  SetDesiredConfig         machineconfigpool/numa-bal   Targeted node worker01 to MachineConfig rendered-numa-bal-2f56...81d0
...  RenderedConfigGenerated  machineconfigpool/numa-bal   rendered-numa-bal-2f56...81d0 successfully generated ...
...  SetDesiredConfig         machineconfigpool/numa-bal   Targeted node worker01 to MachineConfig rendered-numa-bal-2f56...81d0

Create the kernel-numabal TuneD profile CR to enable NUMA balancing. The TuneD profile CR applies to all the nodes in the numa-bal MCP, and includes the default openshift-node TuneD profile. You can find an incomplete example for the TuneD profile CR in the ~/DO380/labs/nodes-review/numa-tuned.yml file. Verify that the NUMA balancing kernel parameter is enabled in the node. You can verify the NUMA balancing by running the sysctl command on the node and checking the kernel.numa_balancing parameter. Verify that the NUMA balancing feature is disabled on other nodes in the cluster.

Create the TuneD profile CR YAML file. You can find an incomplete example for the CR in the ~/DO380/labs/nodes-review/numa-tuned.yml file.

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: kernel-numabal
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=Custom NUMA balancing profile
      include=openshift-node
...output omitted...
  recommend:
  - machineConfigLabels:
      machineconfiguration.openshift.io/role: "numa-bal"
    priority: 30
    profile: kernel-numabal

Apply the configuration for the TuneD CR.

[student@workstation nodes-review]$ oc create -f numa-tuned.yml
tuned.tuned.openshift.io/kernel-numabal created

View the current TuneD profiles in use for each cluster node.

[student@workstation nodes-review]$ oc get profile \
  -n openshift-cluster-node-tuning-operator
NAME      TUNED                    APPLIED  DEGRADED  AGE
master01  openshift-control-plane  True     False     49d
master02  openshift-control-plane  True     False     49d
master03  openshift-control-plane  True     False     49d
worker01  kernel-numabal           True     False     13m
worker02  openshift-node           True     False     2d4h
worker03  openshift-node           True     False     2d4h

Create a debug pod for the worker01 node.

[student@workstation nodes-review]$ oc debug node/worker01
...output omitted...
sh-4.4#

Run the sysctl command and verify that the kernel.numa_balancing parameter is enabled.
```
sh-4.4# sysctl -n kernel.numa_balancing
1
```
Remove the debug pod for the worker01 node.
```
sh-4.4# exit
...output omitted...
```

Create a debug pod for the worker02 node.

[student@workstation nodes-review]$ oc debug node/worker02
...output omitted...
sh-4.4#

Run the sysctl command and verify that the kernel.numa_balancing parameter is disabled.
```
sh-4.4# sysctl -n kernel.numa_balancing
0
```
Remove the debug pod for the worker02 node.
```
sh-4.4# exit
...output omitted...
```

Label the worker02 node with the kernel-rt role. Create the kernel-rt MCP to include the nodes with the kernel-rt role in the pool. You can find an incomplete example for the MCP CR in the ~/DO380/labs/nodes-review/kernel-mcp.yml file.

Add the kernel-rt role to the worker02 node.

[student@workstation nodes-review]$ oc label node/worker02 \
  node-role.kubernetes.io/kernel-rt=

List the labels for the nodes. Verify that the worker02 node has the kernel-rt role.

[student@workstation ~]$ oc get nodes
NAME       STATUS  ROLES                  AGE    VERSION
master01   Ready   control-plane,master   49d    v1.25.7+eab9cc9
master02   Ready   control-plane,master   49d    v1.25.7+eab9cc9
master03   Ready   control-plane,master   49d    v1.25.7+eab9cc9
worker01   Ready   numa,worker            2d4h   v1.25.7+eab9cc9
worker02   Ready   kernel-rt,worker       2d4h   v1.25.7+eab9cc9
worker03   Ready   worker                 2d4h   v1.25.7+eab9cc9

Create the MCP CR YAML file. You can find an incomplete example for the CR in the ~/DO380/labs/nodes-review/kernel-mcp.yml file.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: kernel-rt
spec:
  machineConfigSelector:
    matchExpressions:
      - key: machineconfiguration.openshift.io/role
        operator: In
        values: [worker,kernel-rt]
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/kernel-rt: ""

Apply the configuration for the MCP CR.

[student@workstation nodes-review]$ oc create -f kernel-mcp.yml
machineconfigpool.machineconfiguration.openshift.io/kernel-rt created

Verify that the kernel-rt MCP is correctly created. If the UPDATING field is marked as true for the kernel-rt MCP, then wait until the MCO finishes updating the node.

[student@workstation nodes-review]$ oc get mcp
NAME       CONFIG                 UPDATED  UPDATING  DEGRADED  MACHINECOUNT  ...
kernel-rt  rendered-kernel-...    True     False     False     1             ...
master     rendered-master-...    True     False     False     3             ...
numa-bal   rendered-numa-ba...    True     False     False     1             ...
worker     rendered-worker-...    True     False     False     1             ...

Verify that the worker02 node is assigned for the kernel-rt MCP, and that the MCO applies the rendered MC to the node.

[student@workstation nodes-review]$ oc get events \
  -n openshift-machine-config-operator \
  --sort-by='{.lastTimestamp}' \
  --field-selector involvedObject.name=kernel-rt
...  REASON                   OBJECT                       MESSAGE
...output omitted...
...  RenderedConfigGenerated  machineconfigpool/kernel-rt  rendered-kernel-rt-2f56...81d0 successfully generated ...
...  SetDesiredConfig         machineconfigpool/kernel-rt  Targeted node worker02 to MachineConfig rendered-kernel-rt-2f56...81d0

Use the oc explain command to find the MC parameter that changes the default kernel to a real-time kernel. Create the 99-kernel-realtime MC to change the default kernel to a real-time kernel for the nodes in the kernel-rt pool. You can find an incomplete example for the MC CR in the ~/DO380/labs/nodes-review/kernel-mc.yml file. Verify that the kernel for the worker02 node is a real-time kernel. You can verify that the node uses a real-time kernel by using the oc get nodes -o wide command and verifying that the kernel version shows the rt string.

Find the MC parameter that changes the default kernel to a real-time kernel.

[student@workstation nodes-review]$ oc explain mc.spec
KIND:     MachineConfig
VERSION:  machineconfiguration.openshift.io/v1
...output omitted...
FIELDS:
...output omitted...

   kernelType	<string>
     Contains which kernel we want to be running like default (traditional),
     realtime

   osImageURL	<string>
     OSImageURL specifies the remote location that will be used to fetch the OS

Create the YAML file for the MC CR. You can find an incomplete example for the CR in the ~/DO380/labs/nodes-review/kernel-mc.yml file.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-kernel-realtime
  labels:
    machineconfiguration.openshift.io/role: "kernel-rt"
spec:
  kernelType: realtime

Apply the configuration for the MC CR.

[student@workstation nodes-review]$ oc create -f kernel-mc.yml
machineconfig.machineconfiguration.openshift.io/99-kernel-realtime created

Verify that the 99-kernel-realtime MC is correctly created.

[student@workstation nodes-review]$ oc get mc
NAME                            GENERATEDBYCONTROLLER  IGNITIONVERSION  AGE
...output omitted...
01-worker-kubelet               52fe26136643a946ff...  3.2.0            49d
99-infra-chrony-conf-override                          3.2.0            49d
99-kernel-realtime
99-master-chrony-conf-override                         3.2.0            49d
99-master-generated-registries  52fe26136643a946ff...  3.2.0            49d
...output omitted...

Verify that the MCP starts updating the resources.

[student@workstation nodes-review]$ oc get mcp
NAME       CONFIG                 UPDATED  UPDATING  DEGRADED  MACHINECOUNT  ...
kernel-rt  rendered-kernel-rt...  False    True      False     1             ...
master     rendered-master-...    True     False     False     3             ...
numa-bal   rendered-numa-bal...   True     False     False     1             ...
worker     rendered-worker-...    True     False     False     1             ...

Verify that OpenShift applies the rendered MC to the worker02 node.

[student@workstation nodes-review]$ oc get events \
  -n openshift-machine-config-operator \
  --sort-by='{.lastTimestamp}' \
  --field-selector involvedObject.name=kernel-rt
...  REASON                   OBJECT                       MESSAGE
...output omitted...
...  RenderedConfigGenerated   machineconfigpool/kernel-rt   rendered-kernel-rt-2f56...81d0 successfully generated ...
...  SetDesiredConfig          machineconfigpool/kernel-rt   Targeted node worker02 to MachineConfig rendered-kernel-rt-2f56...81d0
...  RenderedConfigGenerated   machineconfigpool/kernel-rt   rendered-kernel-rt-f6ab...234c successfully generated ...
...  SetDesiredConfig          machineconfigpool/kernel-rt   Targeted node worker02 to MachineConfig rendered-kernel-rt-f6ab...234c

Verify that the MCO updates the node. If the UPDATING field is marked as true for the kernel-rt MCP, then wait until the MCO finishes updating the node.

[student@workstation nodes-review]$ oc get mcp
NAME       CONFIG                 UPDATED  UPDATING  DEGRADED  MACHINECOUNT  ...
kernel-rt  rendered-kernel-rt...  True     False     False     1             ...
master     rendered-master-...    True     False     False     3             ...
numa-bal   rendered-numa-bal...   True     False     False     1             ...
worker     rendered-worker-...    True     False     False     1             ...

Verify that the worker02 node is using a real-time kernel. You can do so by verifying that the kernel version contains the rt string.

[student@workstation nodes-review]$ oc get nodes -o wide
NAME      STATUS  ROLES                 ...  KERNEL-VERSION
master01  Ready   control-plane,master  ...  5.14.0-284.41.1.el9_2.x86_64
master02  Ready   control-plane,master  ...  5.14.0-284.41.1.el9_2.x86_64
master03  Ready   control-plane,master  ...  5.14.0-284.41.1.el9_2.x86_64
worker01  Ready   numa,worker           ...  5.14.0-284.41.1.el9_2.x86_64
worker02  Ready   kernel-rt,worker      ...  5.14.0-284.41.1.rt14.326.el9_2.x86_64
worker03  Ready   worker                ...  5.14.0-284.41.1.el9_2.x86_64

Change to the /home/student directory.
```
[student@workstation nodes-review]$ cd
```

Evaluation

As the student user on the workstation machine, use the lab command to grade your work. Correct any reported failures and rerun the command until successful.

[student@workstation ~]$ lab grade nodes-review

Finish

As the student user on the workstation machine, use the lab command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises.

[student@workstation ~]$ lab finish nodes-review

Discuss Red Hat OpenShift Administration III: Scaling Deployments in the Enterprise

Go to community

Welcome to Red Hat OpenShift Administration III: Scaling Kubernetes Deployments in the Enterprise!

Syed

12 wrz 2023

We are excited to launch a space dedicated to the Red Hat Training course Red Hat OpenShift Administration III: Scaling Kubernetes Deployments in the Enterprise! To gain the most value from this group - click the "Join Group" button in the upper right hand corner of the group home page.We encourage group members to collaborate in this group to discuss topics, ask questions, share best practices and tips, provide course feedback, and share their accomplishments as it relates to DO378.Read more about Red Hat OpenShift Administration III: Scaling Kubernetes Deployments in the Enterprise here.

Revision: do380-4.14-397a507