DO280 - ch06s02

Bookmark this page

Guided Exercise: Project and Cluster Quotas

Configure quotas for a project so that applications cannot scale to consume all capacity of a cluster node.

Outcomes

Verify that requesting resources in one namespace can prevent creation of workloads in different namespaces.
Set a quota to prevent workloads in a namespace from requesting excessive resources.
Verify that you can continue to create workloads in different namespaces.

As the student user on the workstation machine, use the lab command to prepare your system for this exercise.

This command ensures that the cluster API is reachable and deletes the namespaces that you use in this exercise.

[student@workstation ~]$ lab start selfservice-quotas

Instructions

[student@workstation ~]$ oc login -u developer -p developer \
    https://api.ocp4.example.com:6443
Login successful.

...output omitted...

Create a selfservice-quotas project.

Use the oc new-project command to create the project.

[student@workstation ~]$ oc new-project selfservice-quotas
Now using project "selfservice-quotas" on server "https://api.ocp4.example.com:6443".
...output omitted...

Create a deployment with a container that requests one CPU.

Use the oc create command to create the deployment.

[student@workstation ~]$ oc create deployment test \
  --image registry.ocp4.example.com:8443/redhattraining/hello-world-nginx
deployment.apps/test created

Use the oc set resources command to request one CPU in the container specification.

[student@workstation ~]$ oc set resources deployment test --requests=cpu=1
deployment.apps/test resource requirements updated

Use the oc get command to ensure that the deployment starts a pod correctly.

[student@workstation ~]$ oc get pod,deployment
NAME                       READY   STATUS    RESTARTS   AGE
pod/test-8b9fdfbd9-bltlc   1/1     Running   0          13s

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/test   1/1     1            1           49s

Execute the command until the deployment and the pod are ready.

Try to scale the deployment to eight replicas.

Use the oc scale command to scale the deployment.

[student@workstation ~]$ oc scale deployment test --replicas=8
deployment.apps/test scaled

Use the oc get command to view pods and deployments.

[student@workstation ~]$ oc get pod,deployment
NAME                        READY   STATUS    RESTARTS   AGE
pod/test-6c66b55cb5-2kclt   1/1     Running   0          48m
pod/test-6c66b55cb5-5n58r   0/1     Pending   0          5s
pod/test-6c66b55cb5-8x929   0/1     Pending   0          5s
pod/test-6c66b55cb5-blgms   0/1     Pending   0          5s
pod/test-6c66b55cb5-d6z42   1/1     Running   0          6s
pod/test-6c66b55cb5-fc8bk   0/1     Pending   0          5s
pod/test-6c66b55cb5-t29dh   0/1     Pending   0          6s
pod/test-6c66b55cb5-xqr66   0/1     Pending   0          6s

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/test   2/8     8            2           54m

Out of eight pods that the deployment creates, only some of them change to Running status. The other pods stay in Pending status. Not all replicas of the deployment are ready and available.

Use the oc get command to list events. Sort the events by their creation timestamp.

[student@workstation ~]$ oc get event --sort-by .metadata.creationTimestamp
LAST SEEN   TYPE      REASON              OBJECT                       MESSAGE
...output omitted...
3m58s       Normal    ScalingReplicaSet   deployment/test              Scaled up replica set test-6c66b55cb5 to 8
3m58s       Normal    Scheduled           pod/test-6c66b55cb5-d6z42    Successfully assigned selfservice-quotas/test-6c66b55cb5-d6z42 to master01
3m57s       Warning   FailedScheduling    pod/test-6c66b55cb5-5n58r    0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..
...output omitted...

Replicas fail to schedule, because the cluster has insufficient CPU.

Examine the cluster as an administrator.

[student@workstation ~]$ oc login -u admin -p redhatocp
Login successful.

...output omitted...

Use the oc adm top command to display the resource usage of nodes.

[student@workstation ~]$ oc adm top node
NAME       CPU(cores)  CPU%   MEMORY(bytes)   MEMORY%
master01   772m        14%    10185Mi         68%

The cluster does not show high CPU usage.

Use the oc describe command to view the node details.

[student@workstation ~]$ oc describe node/master01
Name:               master01
...output omitted...
Capacity:
  cpu:                6
...output omitted...
Allocatable:
  cpu:                5500m
...output omitted...
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests       Limits
  --------           --------       ------
  cpu                4627m (84%)  0 (0%)
  memory             12102Mi (81%)  0 (0%)
  ephemeral-storage  0 (0%)         0 (0%)
  hugepages-1Gi      0 (0%)         0 (0%)
  hugepages-2Mi      0 (0%)         0 (0%)
...output omitted...

The node has a capacity of six CPUs, and has more than five allocatable CPUs. However, over five CPUs are requested, so less than one CPU is available for new workloads.

Create a test project as an administrator, and verify that you cannot create new workloads that request a CPU.

Use the oc new-project command to create the project.

[student@workstation ~]$ oc new-project test
Now using project "test" on server "https://api.ocp4.example.com:6443".
...output omitted...

Use the oc create command to create the deployment.

[student@workstation ~]$ oc create deployment test \
  --image registry.ocp4.example.com:8443/redhattraining/hello-world-nginx
deployment.apps/test created

Use the oc set resources command to request one CPU in the container specification.

[student@workstation ~]$ oc set resources deployment test --requests=cpu=1
deployment.apps/test resource requirements updated

Use the oc get command to review the pods and deployments in the test namespace.
```
[student@workstation ~]$ oc get pod,deployment
NAME                       READY   STATUS    RESTARTS   AGE
pod/test-8b9fdfbd9-rrn7t   0/1     Pending   0          8s
pod/test-c454765f-vkt96    1/1     Running   0          100s

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/test   1/1     1            1           100s
```
The deployment created one pod before adding the CPU request. When you updated the deployment to request a CPU, the deployment tried to replace the pod to add the CPU request. The new pod is in the Pending state, because the cluster has less than one CPU available to request.
The workload in the selfservice-quotas namespace prevents the creation of workloads in other namespaces.

Use the oc delete command to delete the test namespace.

[student@workstation ~]$ oc delete namespace test
namespace "test" deleted

As an administrator, scale the deployment to one replica.

Use the oc project command to switch to the selfservice-quotas project.

[student@workstation ~]$ oc project selfservice-quotas
Now using project "selfservice-quotas" on server "https://api.ocp4.example.com:6443".

Use the oc scale command to scale the test deployment to one replica.

[student@workstation ~]$ oc scale deployment test --replicas=1
deployment.apps/test scaled

Create a quota to prevent workloads in the selfservice-quotas namespace from requesting more than one CPU.

Use the oc create command to create the quota.

[student@workstation ~]$ oc create quota one-cpu --hard=requests.cpu=1
resourcequota/one-cpu created

Use the oc get command to verify the quota.

[student@workstation ~]$ oc get quota one-cpu -o yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  creationTimestamp: "2024-01-30T18:26:49Z"
  name: one-cpu
  namespace: selfservice-quotas
...output omitted...
spec:
  hard:
    requests.cpu: "1"
status:
  hard:
    requests.cpu: "1"
  used:
    requests.cpu: "1"

The test deployment already requests one CPU.

Try to scale the deployment to eight replicas and to create a second deployment.

Use the oc scale command to scale the deployment.

[student@workstation ~]$ oc scale deployment test --replicas=8
deployment.apps/test scaled

Use the oc create command to create a second deployment.

[student@workstation ~]$ oc create deployment test2 \
  --image registry.ocp4.example.com:8443/redhattraining/hello-world-nginx
deployment.apps/test2 created

Use the oc get command to review pods and deployments.

[student@workstation ~]$ oc get pod,deployment
NAME                        READY   STATUS    RESTARTS   AGE
pod/test-6c66b55cb5-mdxjl   1/1     Running   0          2m58s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/test    1/8     1            1           3m20s
deployment.apps/test2   0/1     0            0           14s

The test deployment creates only two pods. The second deployment does not create any pods.

Use the oc get command to examine the quota status.

[student@workstation ~]$ oc get quota one-cpu -o yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: one-cpu
  namespace: selfservice-quotas
...output omitted...
spec:
  hard:
    requests.cpu: "1"
status:
  hard:
    requests.cpu: "1"
  used:
    requests.cpu: "1"

The used status is kept at 1 because the test2 deployment can't request more resources in the quota.

Use the oc get command to list events. Sort the events by their creation timestamp.

[student@workstation ~]$ oc get event --sort-by .metadata.creationTimestamp
LAST SEEN   TYPE      REASON              OBJECT                       MESSAGE
...output omitted...
4m42s       Warning   FailedCreate        replicaset/test-6c66b55cb5    (combined from similar events): Error creating: pods "`test`-6c66b55cb5-djrr9" is forbidden: exceeded quota: one-cpu, requested: requests.cpu=1, used: requests.cpu=2, limited: requests.cpu=2
9m3s        Warning   FailedCreate        replicaset/test2-7b9df44445   Error creating: pods "test2-7b9df44445-98wxp" is forbidden: failed quota: one-cpu: must specify requests.cpu for: hello-world-nginx
...output omitted...

The test deployment cannot create further pods, because the new pods would exceed the quota. The test2 deployment cannot create pods, because the deployment does not set a CPU request.

Create a test project to verify that you can create new workloads in other namespaces that request CPU resources.

Use the oc new-project command to create the project.

[student@workstation ~]$ oc new-project test
Now using project "test" on server "https://api.ocp4.example.com:6443".
...output omitted...

Use the oc create command to create the deployment.

[student@workstation ~]$ oc create deployment test --image \
  registry.ocp4.example.com:8443/redhattraining/hello-world-nginx
deployment.apps/test created

Use the oc set resources command to request one CPU in the container specification.

[student@workstation ~]$ oc set resources deployment test --requests=cpu=1
deployment.apps/test resource requirements updated

Use the oc get command to review the pods and deployments in the test namespace.

[student@workstation ~]$ oc get pod,deployment
NAME                       READY   STATUS    RESTARTS   AGE
pod/test-8b9fdfbd9-447w9   1/1     Running   0          21s

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/test   1/1     1            1           51s

Even though you cannot create further workloads in the selfservice-quotas namespace, you can create workloads that request CPUs in other namespaces when the node has CPUs and memory available.

Finish

On the workstation machine, use the lab command to complete this exercise. This step is important to ensure that resources from previous exercises do not impact upcoming exercises.

[student@workstation ~]$ lab finish selfservice-quotas

Discuss Red Hat OpenShift Administration II: Configuring a Production Cluster

Go to community

Red Hat OpenShift Administration II: Operating a Production Kubernetes Cluster (DO280)

Haley_Ruccio

26 lip 2023

Configure, manage, and troubleshoot OpenShift clusters and containerized applicationsRed Hat OpenShift Administration II: Operating a Production Kubernetes Cluster (DO280) prepares OpenShift Cluster Administrators to perform daily administration tasks on clusters that host applications provided by internal teams and external vendors, enable self-service for cluster users with different roles, and deploy applications that require special permissions such as such as CI/CD tooling, performance monitoring, and security scanners. This course focuses on configuring multi-tenancy and security features of OpenShift as well as managing OpenShift add-ons based on operators.The skills you learn in this course can be applied using all versions of OpenShift, including Red Hat OpenShift on AWS (ROSA), Azure Red Hat OpenShift, and OpenShift Container Platform.This course is based on OpenShift Container Platform 4.12.5-10 Course TopicsDeploying packaged applications using manifests, templates, kustomize, and helm.Configuring authentication and authorization for users and applications.Protecting network traffic with network policies and exposing applications with proper network access.Deploying and managing applications using resources manifests.Enabling developer self-service of application projects.Managing OpenShift cluster updates and Kubernetes operator updates.Target AudienceSystem Administrators and Platform Operators interested in the ongoing management of OpenShift clusters, applications, users, and add-ons.Site Reliability Engineers interested in the ongoing maintenance and troubleshooting of Kubernetes clusters.System and Software Architects interested in understanding the security of an OpenShift cluster.Recommended TrainingTake our free assessment to gauge whether this offering is the best fit for your skills.Prerequisite: Red Hat OpenShift I: Containers & Kubernetes (DO180 v4.12), or equivalent skills deploying and managing Kubernetes applications using the OpenShift web console and command-line interfaces.Technology considerationsThis course requires internet access to access the cloud-based classroom environment that provides an OpenShift cluster and a remote administrator’s workstation.

476

About pod security standards and warnings

alexcorcoles

17 kwi 2023

(This material has been created with the inputs of instructors and other people. We are trying to get this material published as soon as possible to address some concerns with DO180 and DO280.)With the default configuration, any namespace that you create on OpenShift 4.12 has the following labels: pod-security.kubernetes.io/audit: restrictedpod-security.kubernetes.io/audit-version: v1.24pod-security.kubernetes.io/warn: restrictedpod-security.kubernetes.io/warn-version: v1.24 These labels activate auditing and warning of the restricted pod security standard. The Kubernetes documentation describes the restricted pod security standard. Pod security standards such as restricted are not directly related to security context constraints such as the restricted SCC.Kubernetes uses an admission controller to reject pods that do not comply with pod security standards. An admission controller is a Kubernetes component that inspects every Kubernetes resource before its creation, and acts before Kubernetes creates the resource. This action can include rejecting the creation of the resource, or modifying the resource.The pod security standards are policies that include properties that workloads must follow. For example, the restricted standard requires that containers set the securityContext.allowPrivilegeEscalation field to false.Pod security standards apply both to workloads, such as deployments and jobs, and to pods.When a pod is created, the admission controller inspects the container specifications and enumerates the fields that violate the pod security standard of the namespace. The admission controller inspects the pod-security labels in the namespace to decide which pod security standard to apply (restricted, baseline, or privileged), and which action to take on violations. Namespaces can audit, warn, or enforce pod security standard violations. When the pod-security.kubernetes.io/audit annotation is set to restricted, the admission controller prevents the creation of pods that violate the security standard. When the pod-security.kubernetes.io/audit and pod-security.kubernetes.io/warn annotations are defined in the project, the admission controller does not block the creation of pods, but registers audit events or warnings.The admission controller does not enforce pod security standards for workloads. If the namespace is configured to enforce a pod security standard, then you can create a violating workload. The admission controller enforces only pod security policies on pods.Cluster administrators can use pod security standards to ensure that pods and workloads in namespaces limit their privileges. By limiting privileges, cluster administrators can mitigate what malicious pods can do.The default configuration of user namespaces in OpenShift does not restrict any workload; it only notifies users when they create workloads that do not follow pod security standards that might be enforced in future OpenShift versions.Security Context ConstraintsOpenShift introduced security context constraints (SCC) before the introduction of pod security standards in Kubernetes. SCCs automatically make security modifications to pods when they are created. By default, the system:openshift:scc:restricted-v2 cluster role binding applies to all authenticated users, so regular user workloads use the restricted-v2 SCC. For example, when you execute the following command to create a workload without any security configuration, the SCC adds security configuration to the pods: [user@host ~]$ oc create deployment test --image registry.ocp4.example.com:8443/redhattraining/hello-world-nginx (You can add the --dry-run=client -o yaml options to view the deployment manifest. The deployment manifest does not contain the security properties, which are described later.)If you inspect the pods that the deployment created, the SCC makes the following changes: [user@host ~]$ oc get pod test-b8d5658b6-7bxfs -o yamlapiVersion: v1kind: Podmetadata: annotations:... output omitted... openshift.io/scc: restricted-v2 seccomp.security.alpha.kubernetes.io/pod: runtime/default...output omitted...spec: containers: - image: registry.ocp4.example.com:8443/redhattraining/hello-world-nginx imagePullPolicy: Always name: hello-world-nginx resources: {} securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000690000... output omitted... securityContext: fsGroup: 1000690000 seLinuxOptions: level: s0:c26,c20 seccompProfile: type: RuntimeDefault... output omitted... These restrictions satisfy the restricted pod security policy.The result of these defaults means that when creating a workload (such as a deployment) in a namespace without any customization, the restricted pod security standard applies, but the admission controller only warns and audits. If the workload does not satisfy the pod security standard, then the admission controller allows the workload to be created, but prints a message such as the following warning: Warning: would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (container "example" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "example" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "database" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "example" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") However, by default Kubernetes creates the deployment pods with the restricted-v2 SCC, with the restrictions that the previous warning included. Due to the following factors, workloads generate only a warning, and the pod security standard admission controller allows their pods:The pod security standard admission controller never enforces pod security standards on workloads (only on pods).By default, namespaces do not enforce pod security standards on pods.By default, regular workloads create pods by using the restricted-v2 SCC, which satisfies the restricted pod security standard.Even if future versions of OpenShift change the default namespace configuration to enforce the restricted pod security standard, users will continue to be able to create workloads without explicitly declaring the restrictions that the pod security standard requires, because the restricted-v2 SCC will apply such restrictions automatically.You can take the following approaches to handling pod security:For most workloads, the restricted-v2 SCC limits pod privileges adequately. When creating a workload, you receive the warning, but the workload creates pods and works correctly.You can also create your workloads by providing a full specification of the security constraints to apply, without relying on the SCC to provide the configuration. By following this approach, privileges are more explicit, and you have greater control, so you can adjust more precisely the specific privileges that are granted to workloads.When the restricted-v2 SCC and the default restricted pod security standard are not adequate for your workload, because either you require further privileges, or you need further restrictions, you must use an existing SCC, or create a custom SCC, and provide explicit security specifications. You can also adjust the namespace labels to choose a different pod security standard, and decide to audit, log, or enforce the standard.

1473

Welcome to the Red Hat OpenShift Administration II (DO280) group in the Red Hat Learning Community!

Deanna

17 kwi 2023

We are excited to launch a space dedicated to the Red Hat Training course Red Hat OpenShift Administration II: Operating a Production Kubernetes Cluster!To gain the most value from this group - click the "Join Group" button in the upper right hand corner.We encourage group members to collaborate in this group to discuss topics, ask questions, share best practices and tips, provide course feedback, and share their accomplishments as it relates to DO280.Read more about Red Hat OpenShift Administration II here.Thanks!

1334

Revision: do280-4.14-08d11e1