DO380 - ch03s04

Bookmark this page

Guided Exercise: Node Configuration with the Machine Configuration Operator

Label a cluster node and apply a machine configuration to it, without affecting other cluster nodes.

Outcomes

Use the machine configuration operator (MCO) to modify the Network Time Protocol (NTP) client configuration.
Create a machine configuration pool (MCP) and a machine configuration (MC).
Observe the status of MC updates.

As the student user on the workstation machine, use the lab command to prepare your environment for this exercise, and to ensure that all required resources are available.

[student@workstation ~]$ lab start nodes-mco

Instructions

Your company requires you to configure an NTP server for the nodes in the cluster to synchronize services among them. The new NTP server is 0.rhel.pool.ntp.org.

In production environments, it is common to change the default NTP configuration to refer to NTP servers in the corporate LAN, instead of to public NTP servers from the internet, to prevent clock skew. However, in this exercise, you apply a new NTP configuration only to the worker01 node, to eliminate waiting time for node reboots when applying MCs, and to show how to apply MCs only to a subset of nodes by using labels.

To configure the NTP server in the worker01 node, create the custom MCP. The custom MCP must apply the MCs with the worker and custom labels to the nodes with the custom role. Then, create an MC for the NTP configuration with the custom label.

Connect to the OpenShift cluster and verify the roles for the nodes in the cluster. List the MCs that apply to the worker and custom roles.

Connect to the OpenShift cluster as the admin user with redhatocp as the password.

[student@workstation ~]$ oc login -u admin -p redhatocp \
  https://api.ocp4.example.com:6443
Login successful.
...output omitted...

Change to the nodes-mco project.

[student@workstation ~]$ oc project nodes-mco
Now using project "nodes-mco" on server "https://api.ocp4.example.com:6443".

Verify the nodes in the cluster and their roles.

[student@workstation ~]$ oc get nodes
NAME       STATUS   ROLES                  AGE    VERSION
master01   Ready    control-plane,master   109d   v1.25.7+eab9cc9
master02   Ready    control-plane,master   109d   v1.25.7+eab9cc9
master03   Ready    control-plane,master   109d   v1.25.7+eab9cc9
worker01   Ready    worker                 21d    v1.25.7+eab9cc9
worker02   Ready    worker                 21d    v1.25.7+eab9cc9
worker03   Ready    worker                 21d    v1.25.7+eab9cc9

List the MCs that apply to the worker and custom roles.

[student@workstation ~]$ oc get machineconfig \
  -l machineconfiguration.openshift.io/role=worker
NAME                            GENERATEDBYCONTROLLER  IGNITIONVERSION  AGE
00-worker                       52fe2...ebf97          3.4.0            12d
01-worker-container-runtime     52fe2...ebf97          3.4.0            12d
01-worker-kubelet               52fe2...ebf97          3.4.0            12d
97-worker-generated-kubelet     52fe2...ebf97          3.4.0            12d
98-worker-generated-kubelet     52fe2...ebf97          3.4.0            12d
99-worker-chrony-conf-override                         3.2.0            12d
99-worker-generated-registries  52fe2...ebf97          3.4.0            12d
99-worker-ssh                                          3.2.0            12d

[student@workstation ~]$ oc get machineconfig \
  -l machineconfiguration.openshift.io/role=custom
No resources found

Label the worker01 node with the node-role.kubernetes.io/custom label. Verify that the node has the correct label.

Apply the node-role.kubernetes.io/custom label to the worker01 node.

[student@workstation ~]$ oc label node/worker01 node-role.kubernetes.io/custom=
node/worker01 labeled

Verify that the node has the correct label.

[student@workstation ~]$ oc get nodes
NAME       STATUS   ROLES           AGE    VERSION
master01   Ready    control-plane,master   33d   v1.25.7+eab9cc9
master02   Ready    control-plane,master   33d   v1.25.7+eab9cc9
master03   Ready    control-plane,master   33d   v1.25.7+eab9cc9
worker01   Ready    custom,worker          12d   v1.25.7+eab9cc9
worker02   Ready    worker                 12d   v1.25.7+eab9cc9
worker03   Ready    worker                 12d   v1.25.7+eab9cc9

Create an MCP for the node with the custom label. The MCP must include the MCs for the worker and custom labels.

Change to the /home/student/DO380/labs/nodes-mco directory.
```
[student@workstation ~]$ cd ~/DO380/labs/nodes-mco/
```

Create the MCP custom resource (CR) YAML file. You can find an example for the CR in the /home/student/DO380/labs/nodes-mco/custom-mcp.yml file.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: custom
spec:
  machineConfigSelector:
    matchExpressions:
      - key: machineconfiguration.openshift.io/role
        operator: In
        values: [worker,custom]
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/custom: ""

Apply the configuration for the MCP CR.

[student@workstation nodes-mco]$ oc create -f custom-mcp.yml
machineconfigpool.machineconfiguration.openshift.io/custom created

Verify that the custom MCP is correctly created. If the UPDATING field is marked as true for the custom MCP, then wait until the MCO updates the node.

[student@workstation nodes-mco]$ oc get mcp
NAME      CONFIG               UPDATED  UPDATING  DEGRADED  MACHINECOUNT  ...
custom    rendered-custom-...  True     False     False     1             ...
master    rendered-master-...  True     False     False     3             ...
worker    rendered-worker-...  True     False     False     2             ...

Verify that the worker01 node is assigned for the custom MCP, and that the MCO applies the rendered MC to the node.

[student@workstation nodes-mco]$ oc get events -n \
  openshift-machine-config-operator \
  --sort-by='{.lastTimestamp}' \
  --field-selector involvedObject.name=custom
...  REASON                   OBJECT                    MESSAGE
...  RenderedConfigGenerated  machineconfigpool/custom  rendered-custom-2f56....81d0 successfully generated ...
...  SetDesiredConfig     machineconfigpool/custom  Targeted node worker01 to MachineConfig rendered-custom-2f56...81d0

Because an OpenShift preconfigured MC configures the NTP client, verify the name for the MCs that modify the chrony configuration for the nodes with the worker label. Create an MC for the chrony configuration that applies to the custom label, with higher precedence than the preconfigured MC. The name for the MC must be 99-zcustom-chrony to have higher precedence.

Verify the names for the MCs that modify the chrony configuration for the nodes with the worker label.

[student@workstation nodes-mco]$ oc get mc --selector \
  machineconfiguration.openshift.io/role=worker \
  -o jsonpath="{range .items[*]}{.metadata.name} \
  {': '}{range .spec.config.storage.files[*]}{.path} \
  {' '}{end}{'\n'}{end}" | grep chrony
99-worker-chrony-conf-override: /etc/chrony.conf

Create a chrony configuration file. You can find an example for the CR in the /home/student/DO380/labs/nodes-mco/chrony-mod.conf file.
```
pool 0.rhel.pool.ntp.org iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
logdir /var/log/chrony
```
Encode the chrony configuration file in Base64 format. You use the encoded configuration file in a later step. The encoded configuration file would differ on your system.
```
[student@workstation nodes-mco]$ base64 -w0 chrony-mod.conf; echo
cG9vbCAwLnJoZWwucG9...C92YXIvbG9nL2Nocm9ueQo=
```

Create the 99-zcustom-chrony MC CR YAML file. Use the encoded chrony configuration file from a previous step. You can find an example for the CR in the /home/student/DO380/labs/nodes-mco/99-zcustom-chrony.yml file.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: custom
  name: 99-zcustom-chrony
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - contents:
            compression: ""
            source: data:;base64,cG9vbCAwLnJoZWwucG9...C92YXIvbG9nL2Nocm9ueQo=
          mode: 420
          overwrite: true
          path: /etc/chrony.conf

Apply the configuration for the MC CR.

[student@workstation nodes-mco]$ oc create -f 99-zcustom-chrony.yml
machineconfig.machineconfiguration.openshift.io/99-zcustom-chrony created

Verify that the 99-zcustom-chrony MC is correctly created.

[student@workstation nodes-mco]$ oc get mc
NAME                            GENERATEDBYCONTROLLER  IGNITIONVERSION  AGE
...output omitted...
99-master-generated-registries  52fe26136643a946ff...  3.2.0            109d
99-master-ssh                                          3.2.0            109d
99-worker-chrony-conf-override                         3.2.0            109d
99-worker-generated-registries  52fe26136643a946ff...  3.2.0            109d
99-worker-ssh                                          3.2.0            109d
99-zcustom-chrony                                      3.2.0            2m5s
...output omitted...

Verify that the MCP starts updating the resources.

[student@workstation nodes-mco]$ oc get mcp
NAME      CONFIG               UPDATED  UPDATING  DEGRADED  MACHINECOUNT  ...
custom    rendered-custom-...  False    True      False     1             ...
master    rendered-master-...  True     False     False     3             ...
worker    rendered-worker-...  True     False     False     2             ...

Verify that the MCO applies the rendered MC to the worker01 node.

[student@workstation nodes-mco]$ oc get events \
  -n openshift-machine-config-operator \
  --sort-by='{.lastTimestamp}' \
  --field-selector involvedObject.name=custom
...  REASON                   OBJECT                    MESSAGE
...  RenderedConfigGenerated  machineconfigpool/custom  rendered-custom-2f56...81d0 successfully generated ...
...  SetDesiredConfig         machineconfigpool/custom  Targeted node worker01 to MachineConfig rendered-custom-2f56...81d0
...  RenderedConfigGenerated  machineconfigpool/custom  rendered-custom-009f...3e5d successfully generated ...
...  SetDesiredConfig         machineconfigpool/custom  Targeted node worker01 to MachineConfig rendered-custom-009f...3e5d

Wait a few minutes until the MCO finishes the update.

[student@workstation nodes-mco]$ oc get mcp
NAME      CONFIG               UPDATED  UPDATING  DEGRADED  MACHINECOUNT  ...
custom    rendered-custom-...  True     False     False     1             ...
master    rendered-master-...  True     False     False     3             ...
worker    rendered-worker-...  True     False     False     2             ...

Verify that the MCO applies the chrony configuration to the worker01 node, and that the worker02 and worker03 nodes remain unaltered.

Create a debug pod for the worker01 node.

[student@workstation nodes-mco]$ oc debug node/worker01
Temporary namespace openshift-debug-vlrjg is created for debugging node...
Starting pod/worker01-debug ...
To use host binaries, run chroot /host
Pod IP: 192.168.50.13
If you don't see a command prompt, try pressing enter.
sh-4.4#

Run the chroot /host command to use host binaries.
```
sh-4.4# chroot /host
```

Verify the content for the chrony configuration file. The MC correctly applies the changes to the /etc/chrony.conf file.

sh-4.4# cat /etc/chrony.conf
pool 0.rhel.pool.ntp.org iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
logdir /var/log/chrony

Remove the debug pod for the worker01 node.

sh-4.4# exit
exit
sh-4.4# exit
exit

Removing debug pod ...
Temporary namespace openshift-debug-vlrjg was removed.

Create a debug pod for the worker02 node.

[student@workstation nodes-mco]$ oc debug node/worker02
...output omitted...

Run the chroot /host command to use host binaries.
```
sh-4.4# chroot /host
```

Verify the content for the chrony configuration file. The MC does not change the content in the /etc/chrony.conf file.

sh-4.4# cat /etc/chrony.conf
# The Machine Config Operator manages this file.
server classroom.example.com iburst

stratumweight 0
driftfile /var/lib/chrony/drift
rtcsync
makestep 10 3
bindcmdaddress 127.0.0.1
bindcmdaddress ::1
...output omitted...

Remove the debug pod for the worker02 node.

sh-4.4# exit
exit
sh-4.4# exit
exit
...output omitted...

Manually modify the /etc/chrony.conf file on the worker01 node, and verify that the MCO detects a mismatch in the chrony configuration file.

Create a debug pod for the worker01 node.

[student@workstation nodes-mco]$ oc debug node/worker01
...output omitted...

Run the chroot /host command to use host binaries.
```
sh-4.4# chroot /host
```
Modify the content for the /etc/chrony.conf configuration file as follows. If the vi editor does not show the first line, then resize the window to show it.
```
pool 1.rhel.pool.ntp.org iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
logdir /var/log/chrony
```

Remove the debug pod for the worker01 node.

sh-4.4# exit
exit
sh-4.4# exit
exit
...output omitted...

Verify that the MCO marks the worker01 node with the degraded state.

[student@workstation nodes-mco]$ oc get mcp
NAME      CONFIG               UPDATED  UPDATING  DEGRADED  MACHINECOUNT  ...
custom    rendered-custom-...  False    True      True      1             ...
master    rendered-master-...  True     False     False     3             ...
worker    rendered-worker-...  True     False     False     2             ...

Verify the information for the custom MCP, and that the MCO detects a mismatch in the chrony configuration file.

[student@workstation nodes-mco]$ oc describe mcp custom
...output omitted...
Status:
  Conditions:
...output omitted...
    Last Transition Time:  2023-09-20T11:03:53Z
    Message:               Node worker01 is reporting: "unexpected on-disk state validating against rendered-custom-009f7b72211eb0d37a8ba1d03a343e5d: content mismatch for file \"/etc/chrony.conf\""
    Reason:                1 nodes are reporting degraded status on sync
    Status:                True
    Type:                  NodeDegraded
    Last Transition Time:  2023-09-20T11:03:53Z
    Message:
    Reason:
    Status:                True
    Type:                  Degraded
...output omitted...

Review the logs for the MCO daemon for more information about the configuration drift. The name for the daemon pod might be different on your system.

[student@workstation nodes-mco]$ oc get pod -n openshift-machine-config-operator \
  --field-selector spec.nodeName=worker01
NAME                          READY   STATUS    RESTARTS   AGE
machine-config-daemon-jsrzm   2/2     Running   4          16d

[student@workstation nodes-mco]$ oc logs machine-config-daemon-jsrzm \
  -n openshift-machine-config-operator
...output omitted...
W0929 09:23:20.539413    2520 daemon.go:1763] current+desiredConfig is rendered-custom-11677dc3a3e84870c3a359ab1be4eafc but state is Degraded
I0929 09:23:20.858668    2520 rpm-ostree.go:394] Running captured: rpm-ostree kargs
E0929 09:23:20.905408    2520 on_disk_validation.go:207] content mismatch for file "/etc/chrony.conf" (-want +got):
  bytes.Join({
  	"pool ",
- 	"0",
+ 	"1",
  	".rhel.pool.ntp.org iburst\ndriftfile /var/lib/chrony/drift\nmakest",
  	"ep 1.0 3\nrtcsync\nlogdir /var/log/chrony\n",
  }, "")
E0929 09:23:20.905438    2520 daemon.go:589] Preflight config drift check failed: content mismatch for file "/etc/chrony.conf"
E0929 09:23:20.905457    2520 writer.go:200] Marking Degraded due to: content mismatch for file "/etc/chrony.conf"

Create a debug pod for the worker01 node and fix the /etc/chrony.conf file.

[student@workstation nodes-mco]$ oc debug node/worker01
...output omitted...

sh-4.4# chroot /host

pool 0.rhel.pool.ntp.org iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
logdir /var/log/chrony

sh-4.4# exit
exit
sh-4.4# exit
exit
...output omitted...

Verify that the MCO marks the node with the ready state, which might take a few minutes.

[student@workstation nodes-mco]$ oc get mcp
NAME      CONFIG               UPDATED  UPDATING  DEGRADED  MACHINECOUNT  ...
custom    rendered-custom-...  True     False     False     1             ...
master    rendered-master-...  True     False     False     3             ...
worker    rendered-worker-...  True     False     False     2             ...

Remove the node-role.kubernetes.io/custom label for the worker01 node. After the MCO updates the node with the new configuration, delete the 99-zcustom-chrony MC and the custom-ntp MCP.

Remove the node-role.kubernetes.io/custom label from the worker01 node.

[student@workstation nodes-mco]$ oc label node/worker01 \
  node-role.kubernetes.io/custom-
node/worker01 unlabeled

Verify that no node uses the node-role.kubernetes.io/custom label.

[student@workstation nodes-mco]$ oc get nodes -l node-role.kubernetes.io/custom=
No resources found

Verify that the MCO applies the changes to the worker01 node. Wait until the MCO updates the node. The MCO finishes updating the node when the desiredConfig and currentConfig annotations point to the same rendered MC, and the state annotation is Done.

[student@workstation nodes-mco]$ oc describe node worker01 \
  | grep machineconfiguration
  machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable
  machineconfiguration.openshift.io/currentConfig: rendered-custom-1167...eafc
  machineconfiguration.openshift.io/desiredConfig: rendered-worker-d4f4...6774
  machineconfiguration.openshift.io/desiredDrain: drain-rendered-worker-...
  machineconfiguration.openshift.io/lastAppliedDrain: drain-rendered-worker-...
  machineconfiguration.openshift.io/reason:
  machineconfiguration.openshift.io/state: Working

[student@workstation nodes-mco]$ oc get mcp
NAME      CONFIG               UPDATED  UPDATING  DEGRADED  MACHINECOUNT  ...
custom    rendered-custom-...  True     False     False     0             ...
master    rendered-master-...  True     False     False     3             ...
worker    rendered-worker-...  True     False     False     3             ...

Delete the 99-zcustom-chrony MC.

[student@workstation nodes-mco]$ oc delete mc 99-zcustom-chrony
machineconfig.machineconfiguration.openshift.io "99-zcustom-chrony" deleted

Delete the custom MCP.
Warning
Remove the custom MCP only after the MCO applies the changes to the worker01 node. If you remove the custom MCP before the MCO applies the changes to the node, then the MCP gets stuck in a degraded state, because the currentConfig MC of the node does not exist.
```
[student@workstation nodes-mco]$ oc delete mcp custom
machineconfigpool.machineconfiguration.openshift.io "custom" deleted
```
Change to the /home/student directory.
```
[student@workstation nodes-mco]$ cd
```

Finish

On the workstation machine, use the lab command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises.

[student@workstation ~]$ lab finish nodes-mco

Discuss Red Hat OpenShift Administration III: Scaling Deployments in the Enterprise

Go to community

Welcome to Red Hat OpenShift Administration III: Scaling Kubernetes Deployments in the Enterprise!

Syed

12 wrz 2023

We are excited to launch a space dedicated to the Red Hat Training course Red Hat OpenShift Administration III: Scaling Kubernetes Deployments in the Enterprise! To gain the most value from this group - click the "Join Group" button in the upper right hand corner of the group home page.We encourage group members to collaborate in this group to discuss topics, ask questions, share best practices and tips, provide course feedback, and share their accomplishments as it relates to DO378.Read more about Red Hat OpenShift Administration III: Scaling Kubernetes Deployments in the Enterprise here.

Revision: do380-4.14-397a507

Red Hat OpenShift Administration III: Scaling Deployments in the Enterprise

Guided Exercise: Node Configuration with the Machine Configuration Operator

Warning