Kubernetes clusters can run heterogeneous workloads across many compute nodes. By using Kubernetes role-based access control (RBAC), cluster administrators can allow users to create workloads on their own. Although RBAC can limit the kinds of resources that users can create, administrators might want further measures to ensure correct operation of the cluster.

Clusters have limited resources, such as CPU, RAM, and storage. If workloads on a cluster exceed the available resources, then workloads might not work correctly. A cluster that is configured to autoscale might also incur unwanted economic costs if the cluster scales to accommodate unexpected workloads.

To help with this issue, Kubernetes workloads can reserve resources and declare resource limits. Workloads can specify the following properties:

These measures prevent workloads from affecting other workloads. However, cluster administrators might need to prevent other risks.

For example, users might mistakenly create unwanted workloads. The resource requests of those unwanted workloads can prevent legitimate workloads from executing.

By dividing workloads into namespaces, Kubernetes can offer enhanced protection features. The namespace structure often mirrors the organization that runs the cluster. Kubernetes introduces resource quotas to limit resource usage by the combined workloads in a namespace.

Kubernetes administrators can create resources of the ResourceQuota type in a namespace for this purpose. When a resource quota exists in a namespace, Kubernetes prevents the creation of workloads that exceed the quota.

Whereas quota features in other systems often act on users or groups of users, Kubernetes resource quotas act on namespaces.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: memory
  namespace: example
spec:
  hard: 
    limits.memory: 4Gi
    requests.memory: 2Gi
  scopes: {} 
  scopeSelector: {} 

The following sections describe the compute and object count quotas that you can include in the hard key. Other components can define other quotas and enforce them.

[user@host ~]$ oc api-resources --api-group=""  --namespaced=true
NAME                     SHORTNAMES   APIVERSION   NAMESPACED   KIND
bindings                              v1           true         Binding
...output omitted...

Navigate to Administration → ResourceQuotas to create a resource quota from the web console. The YAML editor loads an example resource quota that you can edit for your needs.

You can also use the oc command to create a resource quota. The oc command can create resource quotas without requiring a complete resource definition. Execute the oc create resourcequota --help command to display examples and help for creating resource quotas without a complete resource definition.

For example, execute the following command to create a resource quota that limits the number of pods in a namespace:

[user@host ~]$ oc create resourcequota example --hard=count/pods=1
resourcequota/example created

The previous command is equivalent to creating a resource quota with the following definition:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: example
spec:
  hard:
    count/pods: "1"

After creating a resource quota, the status key in the resource describes the current values and limits in the quota.

[user@host ~]$ oc get quota example -o yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  creationTimestamp: "2024-01-30T17:59:52Z"
  name: example
  namespace: default
  resourceVersion: "193658"
  uid: df12b484-4e78-4920-acb4-e04ab286a4a1
spec:
  hard:
    count/pods: "1"
status:
  hard:
    count/pods: "1"
  used:
    count/pods: "0"

The oc get and oc describe commands show resource quota information in a custom format. The oc get command displays the status of the quota in resource lists:

[user@host ~]$ oc get quota
NAME      AGE     REQUEST           LIMIT
example   9m54s   count/pods: 1/1

Resource quotas generate the kube_resourcequota metric. You can examine this metric for planning and trend analysis.

Figure 6.1:

The kube_resourcequota metric

Because resource quotas are extensible, Kubernetes cannot verify that a resource quota is correct. For example, the following command creates a resource quota that has no effect:

[user@host ~]$ oc create resourcequota example --hard=count/deployment=1
resourcequota/example created

The correct syntax for limiting the number of deployments is count/deployments.apps.

To ensure that a resource quota is correct, you can use the following procedures:

Exceeding a quota often produces an error immediately. For example, if you create a deployment that exceeds the deployment quota, then the deployment creation fails.

[user@host ~]$ oc create deployment --image=nginx hello
error: failed to create deployment: deployments.apps "hello" is forbidden: exceeded quota: example, requested: count/deployments.apps=1, used: count/deployments.apps=1, limited: count/deployments.apps=1

However, some quotas do not cause operations to fail immediately. For example, if you set a resource quota for pods, then creating a deployment appears to succeed, but the deployment never becomes available. When a resource quota is acting indirectly, namespace events might provide further information.

[user@host ~]$ oc get event --sort-by .metadata.creationTimestamp
LAST SEEN   TYPE      REASON              OBJECT                        MESSAGE
...output omitted...
10s         Normal    ScalingReplicaSet   deployment/hello              Scaled up replica set hello-5cdfd9c858 to 1
9s          Warning   FailedCreate        replicaset/hello-5cdfd9c858   Error creating: pods "hello-5cdfd9c858-zsgn9" is forbidden: exceeded quota: example, requested: count/pods=1, used: count/pods=1, limited: count/pods=1
5s          Warning   FailedCreate        replicaset/hello-5cdfd9c858   (combined from similar events): Error creating: pods "hello-5cdfd9c858-h2dv4" is forbidden: exceeded quota: example, requested: count/pods=1, used: count/pods=1, limited: count/pods=1

The web console also shows quota information. Navigate to Administration → ResourceQuotas to view resource quotas and their status. The project pages on both the developer and administrator perspectives also show the quotas that apply to a specific project.

Cluster administrators can use resource quotas to apply restrictions to namespaces.

Resource restrictions often follow organization structure. Although namespaces often reflect organization structure, cluster administrators might apply restrictions to resources without being limited to a single namespace.

For example, a group of developers manages many namespaces. Namespace quotas can limit RAM usage per namespace. However, a cluster administrator cannot limit total RAM usage by all workloads that the group of developers manages.

OpenShift introduces cluster resource quotas for those scenarios.

Cluster resource quotas follow a similar structure to namespace resource quotas. However, cluster resource quotas use selectors to choose which namespaces the quota applies to.

Cluster resource quotas selectors use set-based requirements.

The following example shows a cluster resource quota:

apiVersion: quota.openshift.io/v1
kind: ClusterResourceQuota
metadata:
  name: example
spec:
  quota: 
    hard:
      limits.cpu: 4
  selector: 
    annotations: {}
    labels:
      matchLabels:
        kubernetes.io/metadata.name: example

Navigate to Administration → CustomResourceDefinitions to create a cluster resource quota with the web console.

You can also use the oc command to create a cluster quota. The oc command can create quotas without requiring a complete resource definition. Execute the oc create clusterresourcequota --help command to display examples and help about creating cluster resource quotas without a complete resource definition.

For example, execute the following command to create a resource quota that limits total CPU requests. The quota limits the total CPU requests on namespaces that have the group label with the dev value.

[user@host ~]$ oc create clusterresourcequota example --project-label-selector=group=dev --hard=requests.cpu=10
clusterresourcequota/example created

Cluster resource quotas collect total resource usage across namespaces and enforce the limits. The following example shows the status of the previous cluster resource quota:

apiVersion: quota.openshift.io/v1
kind: ClusterResourceQuota
metadata:
  name: example
spec:
  quota:
    hard:
      requests.cpu: "10"
  selector:
    annotations: null
    labels:
      matchLabels:
        group: dev
status:
  namespaces: 
  - namespace: example-3
    status:
      hard:
        requests.cpu: "10"
      used:
        requests.cpu: 500m
  - namespace: example-2
    status:
      hard:
        requests.cpu: "10"
      used:
        requests.cpu: 250m
_...output omitted..._
  total: 
    hard:
      requests.cpu: "10"
    used:
      requests.cpu: 2250m

Users might not have read access to cluster resource quotas. OpenShift creates resources of the AppliedClusterResourceQuota type in namespaces that are affected by cluster resource quotas. Project administrators can review quota usage by reviewing the AppliedClusterResourceQuota resources. For example, use the oc describe command to view the cluster resource quotas that apply to a specific namespace:

[user@host ~]$ oc describe AppliedClusterResourceQuota -n example-2
Name:		example
Created:	9 minutes ago
Labels:		<none>
Annotations:	<none>
Namespace Selector: ["example-3" "example-2" "example-4" "example-1"]
Label Selector: group=dev
AnnotationSelector: map[]
Resource	Used	Hard
--------	----	----
requests.cpu	2250m	10

Navigate to Administration → ResourceQuotas to view quotas and their status. This page displays cluster quotas along with namespace quotas. Although you can view resources of the ClusterResourceQuota type and create resources of the ResourceQuota type in the ResourceQuotas page, you cannot create objects of the ClusterResourceQuota in this page.

The project pages on both the developer and administrator perspectives also show the cluster quotas that apply to a specific project.