Red Hat OpenShift Administration III: Scaling Deployments in the Enterprise

To ensure that the OpenShift scheduler finds a node that matches the target labels, first label your nodes. You can add labels directly to a node resource or indirectly by using a machine set resource.

If your cluster is installed in a way that supports machine set resources, then you must use machine sets to set labels on nodes. In that case, do not label nodes directly. If your cluster does not support machine set resources, because for example it is installed by using the user-provisioned infrastructure method, then you must label nodes directly.

You can add one or more labels to a node by using the following command:

[user@host ~]$ oc label node node1 key1=value1 ... keyn=valuen
node/node1 labeled

You can remove labels from a node by using the following command:

[user@host ~]$ oc label node node1 key1- ... keyn-
node/node1 unlabeled

Use node selectors on a pod to specify target node labels where the OpenShift scheduler tries to place the pod.

You can add one or more node selectors to a pod by using the following syntax:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
...output omitted...
spec:
  nodeSelector:
    key1: value1
    key2: value2
...output omitted...

In the previous example, the OpenShift scheduler places the pod only in nodes with both the key1=value1 and key2=value2 labels.

You can read the node selectors for a pod by reviewing the Node-Selectors parameter as follows:

[user@host ~]$ oc describe pod myapp-95664666d-5l22c
...output omitted...
Node-Selectors:     key1:value1
                    key2:value2
...output omitted...

By default, the OpenShift scheduler tries to place the pods onto any available nodes in your cluster. However, you can modify the OpenShift scheduler default behavior so it includes default node selectors to any new pods that are created in the cluster. By using cluster-wide node selectors, you specify the default node selectors that OpenShift adds to the pods. You require administrative privileges to specify cluster-wide node selectors.

To define cluster-wide node selectors, you can edit the OpenShift scheduler CR as follows:

[user@host ~]$ oc edit scheduler cluster

apiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
  name: cluster
...output omitted...
spec:
  defaultNodeSelector: key1=value1,key2=value2
...output omitted...

After you edit the OpenShift scheduler CR, wait for the pods in the openshift-kube-apiserver project to redeploy.

In the previous example, any new pods that you create in the cluster have the key1=value1 and key2=value2 node selectors, by default.

You can specify node selectors for a project, so OpenShift includes those node selectors to any new pods that are created in the project. You require administrative privileges to specify project node selectors.

To define project node selectors, you can create a YAML file for an OpenShift project CR, and define the node selectors in the metadata.annotations section, as follows:

apiVersion: v1
kind: Namespace
metadata:
  name: myproject
  annotations:
    openshift.io/node-selector: key3=value3,key4=value4
...output omitted...

In the previous example, any new pods that you create in the project have by default the key3=value3 and key4=value4 node selectors.

You can define node selectors at the following levels: pod, project, and cluster-wide.

Project node selectors take precedence over cluster-wide node selectors. Thus, if you define both project and cluster-wide node selectors, then OpenShift applies only the project node selectors to the new pods but not the cluster-wide node selectors.

Pod node selectors are additive to cluster-wide and project node selectors. Thus, if you define both pod and project node selectors, then OpenShift creates the pod with both the pod node selectors and the project node selectors. If you define both pod and cluster-wide node selectors, then OpenShift creates the pod with both the pod node selectors and the cluster-wide node selectors.

Taints consist of a key, a value, and an effect.

You can apply a taint to a node in the node specification YAML file by setting the spec.taints parameter or by using the following command:

[user@host ~]$ oc adm taint nodes node1 key1=value1:NoSchedule
node/node1 tainted

After you apply a taint to a node, you can verify the taint by using the following command:

[user@host ~]$ oc describe nodes node1
Name:               worker01
Roles:              worker
...output omitted...
Taints:             key1=value1:NoSchedule
...output omitted...

To remove a taint from a node, use the following command:

[user@host ~]$ oc adm taint nodes node01 key1=value1:NoSchedule-
node/node01 untainted

Tolerations consist of a key, a value, and an effect, similar to the node taints. Tolerations also include the operator parameter. The following values for the operator parameter are possible:

When you define a toleration with the NoExecute effect, you can also define the tolerationSeconds parameter. The tolerationSeconds parameter defines the time that a pod stays within a node before the node evicts it.

You can apply a toleration to a pod in the pod specification YAML file by setting the spec.tolerations parameter as follows:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
...output omitted...
spec:
  tolerations:
  - key: "key1"
    operator: "Equal"
    value: "value1"
    effect: "NoExecute"
    tolerationSeconds: 3600
...output omitted...

In the previous example, OpenShift can schedule the pod in nodes with the key1=value1:NoExecute taint. If OpenShift schedules the pod in a node with the taint, then the pod runs in the node for 3600 seconds before the node evicts it and OpenShift reschedules the pod to another node.

By default, OpenShift taints and evicts nodes by condition. For example, OpenShift automatically taints nodes under some conditions, such as memory or disk pressure.

The following taints are built into OpenShift:

If a node reports one of the conditions from the previous table, then OpenShift applies the taint to the node until the condition clears.

For example, you can verify how OpenShift automatically applies the node.kubernetes.io/unschedulable taint to an unschedulable pod by following the next example. First, verify the taints for a node by using the following command:

[user@host ~]$ oc describe nodes node1 | grep Taints
Taints:     <none>

Then, mark the node as unschedulable.

[user@host ~]$ oc adm cordon node1
node/node1 cordoned

Finally, verify that OpenShift automatically applies the node.kubernetes.io/unschedulable taint with the NoSchedule effect to the node.

[user@host ~]$ oc describe nodes node1 | grep Taints
Taints:     node.kubernetes.io/unschedulable:NoSchedule

OpenShift applies the taints to the nodes from the previous table with the NoSchedule effect, except for the node.kubernetes.io/not-ready and node.kubernetes.io/unreachable taints where OpenShift uses the NoExecute effect.

For the taints with the NoSchedule effect, the OpenShift scheduler schedules the pod to the node if the pod has a matching toleration, or if the node returns to a normal state, but running pods remain unaltered.

For the taints with the NoExecute effect, the OpenShift scheduler schedules the pod to the node if the pod has a matching toleration, or if the node returns to a normal state, and starts evicting and rescheduling running pods to different nodes. For pods that do not tolerate the taint and that are running on the node, OpenShift evicts them immediately. For pods that match the toleration, OpenShift evicts them only if the tolerationSeconds parameter is defined on the pod. The tolerationSeconds parameter defines the time in seconds that a pod remains in a node until the node evicts it.

The tolerationSeconds parameter is useful in certain scenarios, such as when an application must create local files when running for the first time. In this case, waiting for a few seconds for the pod to recover might be preferable to directly re-creating the pod on another node. If you define the tolerationSeconds parameter for an application, then the node waits the specified time before evicting the pods.

You can specify tolerations for a project, so OpenShift includes those tolerations to any new pods that are created in the project. You require administrative privileges to specify project tolerations. If you define project and pod tolerations, then OpenShift creates the pod with both of these tolerations.

To define project tolerations, you can create a YAML file for an OpenShift project CR and define the toleration in the metadata.annotations section as follows:

kind: Project
apiVersion: project.openshift.io/v1
metadata:
  name: myproject
  annotations:
    scheduler.alpha.kubernetes.io/defaultTolerations: >-
      [{"operator":"Equal","effect":"NoSchedule","key":"key1","value":"value1"}]

In the previous example, any new pods that you create in the project have by default the key1=value1:NoSchedule toleration.

You can configure a pod to tolerate all the node taints by using the operator: "Exists" toleration with no key or value parameters.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
...output omitted...
spec:
  tolerations:
  - operator: "Exists"

OpenShift does not remove the pods with the operator: "Exists" toleration from a node that has taints.