DO280 - ch01s03

Bookmark this page

Kustomize Overlays

Objectives

Deploy and update applications from resource manifests that are augmented by Kustomize.

Kustomize

When using Kubernetes, multiple teams use multiple environments, such as development, staging, testing, and production, to deploy applications. These environments use applications with minor configuration changes.

Many organizations deploy a single application to multiple data centers for multiple teams and regions. Depending on the load, the organization needs a different number of replicas for every region. The organization might need various configurations that are specific to a data center or team.

All these use cases require a single set of manifests with multiple customizations at multiple levels. Kustomize can support such use cases.

Kustomize is a configuration management tool to make declarative changes to application configurations and components and preserve the original base YAML files. You group in a directory the Kubernetes resources that constitute your application, and then use Kustomize to copy and adapt these resource files to your environments and clusters. The kubectl command integrates the kustomization tool.

Kustomize File Structure

Kustomize works on directories that contain a kustomization.yaml file at the root. Kustomize supports compositions and customization of different resources such as deployment, service, and secret. You can use patches to apply customization to different resources. Kustomize has a concept of base and overlays.

Base

A base directory contains a kustomization.yaml file. The kustomization.yaml file has a list resource field to include all resource files. As the name implies, all resources in the base directory are a common resource set. You can create a base application by composing all common resources from the base directory.

The following diagram shows the structure of a base directory:

base
├── configmap.yaml
├── deployment.yaml
├── secret.yaml
├── service.yaml
├── route.yaml
└── kustomization.yaml

The base directory has YAML files to create configuration map, deployment, service, secret, and route resources. The base directory also has a kustomization.yaml file, such as the following example:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- configmap.yaml
- deployment.yaml
- secret.yaml
- service.yaml
- route.yaml

The kustomization.yaml file lists all resource files.

Overlays

Kustomize overlays declarative YAML artifacts, or patches, that override the general settings without modifying the original files. The overlay directory contains a kustomization.yaml file. The kustomization.yaml file can refer to one or more directories as bases. Multiple overlays can use a common base kustomization directory.

The following diagram shows the structure of all Kustomize directories:

Figure 1.4: Kustomize file structure

The following example shows the directory structure of the frontend-app directory containing the base and overlay directories:

[user@host frontend-app]$ tree
base
├── configmap.yaml
├── deployment.yaml
├── secret.yaml
├── service.yaml
├── route.yaml
└── kustomization.yaml
overlay
└── development
    └── kustomization.yaml
└── testing
    └── kustomization.yaml
└── production
    ├── kustomization.yaml
    └── patch.yaml

The following example shows a kustomization.yaml file in the overlays/development directory:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: dev-env
resources:
- ../../base

The frontend-app/overlay/development/kustomization.yaml file uses the base kustomization file at ../../base to create all the application resources in the dev-env namespace.

Kustomize provides fields to set values for all resources in the kustomization file:

Field	Description
`namespace`	Set a specific namespace for all resources.
`namePrefix`	Add a prefix to the name of all resources.
`nameSuffix`	Add a suffix to the name of all resources.
`commonLabels`	Add labels to all resources and selectors.
`commonAnnotations`	Add annotations to all resources and selectors.

You can customize for multiple environments by using overlays and patching. The patches mechanism has two elements: patch and target.

Previously, Kustomize used the PatchesJson6902 and PatchesStrategicMerge keys to add resource patches. These keys are deprecated in Kustomize version 5 and are replaced with a single key. However, the content of the patches key continues to use the same patch formats.

You can use JSON Patch and strategic merge patches. See the references section for further information about both patch formats.

The following is an example of a kustomization.yaml file in the overlays/testing directory:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: test-env
patches: 
- patch: |-
    - op: replace 
      path: /metadata/name
      value: frontend-test
  target: 
    kind: Deployment
    name: frontend
- patch: |- 
    - op: replace
      path: /spec/replicas
      value: 15
  target:
    kind: Deployment
    name: frontend
resources: 
- ../../base
commonLabels: 
  env: test

	The `patches` field contains a list of patches.
	The `patch` field defines operation, path, and value keys. In this example, the name changes to `frontend-test`.
	The `target` field specifies the kind and name of the resource to apply the patch. In this example, you are changing the `frontend` deployment name to `frontend-test`.
	This patch updates the number of replicas of the `frontend` deployment.
	The `frontend-app/overlay/testing/kustomization.yaml` file uses the base kustomization file at `../../base` to create an application.
	The `commonLabels` field adds the `env: test` label to all resources.

The patches mechanism also provides an option to include patches from a separate YAML file by using the path key.

The following example shows a kustomization.yaml file that uses a patch.yaml file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: prod-env
patches: 
- path: patch.yaml 
  target: 
    kind: Deployment
    name: frontend
  options:
    allowNameChange: true 
resources: 
- ../../base
commonLabels: 
  env: prod

	The `patches` field lists the patches that are applied by using a production kustomization file.
	The `path` field specifies the name of the patching YAML file.
	The `target` field specifies the kind and name of the resource to apply the patch. In this example, you are targeting the `frontend` deployment.
	The `allowNameChange` field enables kustomization to update the name by using a patch YAML file.
	The `frontend-app/overlay/production/kustomization.yaml` file uses the base kustomization file at `../../base` to create an application.
	The `commonLabels` field adds an `env: prod` label to all resources.

The patch.yaml file has the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend-prod 
spec:
  replicas: 5

	The `metadata.name` field in the patch file updates the `frontend` deployment name to `frontend-prod` if the `allowNameChange` field is set to `true` in the kustomization YAML file.
	The `spec/replicas` field in the patch file updates the number of replicas of the `frontend-prod` deployment.

View and Deploy Resources by Using Kustomize

Run the kubectl kustomize kustomization-directory command to render the manifests without applying them to the cluster.

[user@host frontend-app]$ kubectl kustomize overlay/production
...output omitted...
kind: Deployment
metadata:
  labels:
    app: frontend
    env: prod
  name: frontend-prod
...output omitted...
spec:
  replicas: 5
  selector:
    matchLabels:
      app: frontend
      env: prod
...output omitted...

The kubectl apply command applies configurations to the resources in the cluster. If resources are not available, then the kubectl apply command creates resources. The kubectl apply command applies a kustomization with the -k flag.

[user@host frontend-app]$ kubectl apply -k overlay/production
deployment.apps/frontend-prod created
...output omitted...

Delete Resources by Using Kustomize

Run the oc delete -k kustomization-directory command to delete the resources that were deployed by using Kustomize.

[user@host frontend-app]$ oc delete -k overlay/production
configmap "database" deleted
secret "database" deleted
service "database" deleted
deployment.apps "database" deleted

Kustomize Generators

Configuration maps hold non-confidential data by using a key-value pair. Secrets are similar to configuration maps, but secrets hold confidential information such as usernames and passwords. Kustomize has configMapGenerator and secretGenerator fields that generate configuration map and secret resources.

The configuration map and secret generators can include content from external files in the generated resources. By keeping the content of the generated resources outside the resource definitions, you can use files that other tools generated, or that are stored in different systems. Generators help to manage the content of configuration maps and secrets, by taking care of encoding and including content from other sources.

Configuration Map Generator

Kustomize provides a configMapGenerator field to create a configuration map. The configuration map that a configMapGenerator field creates behaves differently. In this method, Kustomize appends a hash to the name, and any change in the configuration map triggers a rolling update.

The following example adds a configuration map by using the configMapGenerator field in the staging kustomization file. The hello application deployment has two environment variables to refer to the hello-app-configmap configuration map.

The kustomization.yaml file has the following content:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: hello-stage
resources:
- ../../base
configMapGenerator:
- name: hello-app-configmap
  literals:
    - msg="Welcome!"
    - enable="true"

The deployment.yaml file has the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  labels:
    app: hello
    name: hello
spec:
...output omitted...
    spec:
      containers:
      - name: hello
        image: quay.io/hello-app:v1.0
        env:
        - name: MY_MESSAGE
          valueFrom:
            configMapKeyRef:
              name: hello-app-configmap
              key: msg
        - name: MSG_ENABLE
          valueFrom:
            configMapKeyRef:
              name: hello-app-configmap
              key: enable

You can view and deploy all resources and customizations that the kustomization YAML file defines, in the development directory.

[user@host hello-app]$ kubectl kustomize overlays/staging
apiVersion: v1
data:
  enable: "true"
  msg: Welcome!
kind: ConfigMap
metadata:
  name: hello-app-configmap-9tcmf95d77
  namespace: hello-stage
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: hello
    name: hello
  name: hello
  namespace: hello-stage
spec:
...output omitted...
    spec:
      containers:
      - env:
        - name: MY_MESSAGE
          valueFrom:
            configMapKeyRef:
              key: msg
              name: hello-app-configmap-9tcmf95d77
        - name: MSG_ENABLE
          valueFrom:
            configMapKeyRef:
              key: enable
              name: hello-app-configmap-9tcmf95d77
...output omitted...

[user@host hello-app]$ kubectl apply -k overlays/staging
configmap/hello-app-configmap-9tcmf95d77 created
deployment.apps/hello created

[user@host hello-app]$ oc get all
NAME                         READY   STATUS    RESTARTS   AGE
pod/hello-75dc9cfc87-jh62k   1/1     Running   0          97s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello   1/1     1            1           97s

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/hello-75dc9cfc87   1         1         1       97s

The kubectl apply -k command creates a hello-app-configmap-9tcmf95d77 configuration map and a hello deployment. Update the kustomization.yaml file with the configuration map values.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: hello-stage
resources:
- ../../base
configMapGenerator:
- name: hello-app-configmap
  literals:
    - msg="Welcome Back!"
    - enable="true"

Then, apply the overlay with the kubectl apply command.

[user@host hello-app]$ kubectl apply -k overlays/staging
configmap/hello-app-configmap-696dm8h728 created
deployment.apps/hello configured

[user@host hello-app]$ oc get all
NAME                        READY   STATUS    RESTARTS   AGE
pod/hello-55bc55ff9-hrszh   1/1     Running   0          3s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello   1/1     1            1           5m5s

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/hello-55bc55ff9    1         1         1       3s
replicaset.apps/hello-75dc9cfc87   0         0         0       5m5s

The kubectl apply -k command applies kustomization. Kustomize appends a new hash to the configuration map name, which creates a hello-app-configmap-696dm8h728 configuration map. The new configuration map triggers the generation of a new hello-55bc55ff9-hrszh pod.

You can generate a configuration map by using the files key from the .properties file or from the .env file by using the envs key with the file name as the value. You can also create a configuration map from a literal key-value pair by using the literals key.

The following example shows a kustomization.yaml file with the configMapGenerator field.

...output omitted...
configMapGenerator:
- name: configmap-1  
  files:
    - application.properties
- name: configmap-2  
  envs:
    - configmap-2.env
- name: configmap-3  
  literals:
    - name="configmap-3"
    - description="literal key-value pair"

	The `configmap-1` key is using the `application.properties` file.
	The `configmap-2` key is using the `configmap-2.env` file.
	The `configmap-3` key is using a literal key-value pair.

The following example shows the application.properties file that is referenced in the configmap-1 key.

Day=Monday
Enable=True

The following example shows the configmap-2.env file that is referenced in the configmap-2 key.

Greet=Welcome
Enable=True

Run the kubectl kustomize command to view details of resources and customizations that the kustomization YAML file defines:

[user@host base]$ kubectl kustomize .
apiVersion: v1
data:
  application.properties: |
    Day=Monday
    Enable=True
kind: ConfigMap
metadata:
  name: configmap-1-5g2mh569b5 
---
apiVersion: v1
data:
  Enable: "True"
  Greet: Welcome
kind: ConfigMap
metadata:
  name: configmap-2-92m84tg9kt 
---
apiVersion: v1
data:
  description: literal key-value pair
  name: configmap-3
kind: ConfigMap
metadata:
  name: configmap-3-k7g7d5bffd 
---
...output omitted...

	The `configMapGenerator` field appends a hash to all ConfigMap resources. The `configmap-1-5g2mh569b5` configuration map is generated from the `application.properties` file, and the data field has a single key with the `application.properties` value.
	The `configmap-2-92m84tg9kt` configuration map is generated from the `configmap-2.env` file, and the data field has separate keys for each listed variable in the `configmap-2.env` file.
	The `configmap-3-k7g7d5bffd` configuration map is generated from a literal key-value pair.

Secret Generator

A secret resource has sensitive data such as a username and a password. You can generate the secret by using the secretGenerator field. The secretGenerator field works similarly to the configMapGenerator field. However, the secretGenerator field also performs the base64 encoding that secret resources require.

The following example shows a kustomization.yaml file with the secretGenerator field:

...output omitted...
secretGenerator:
- name: secret-1  
  files:
    - password.txt
- name: secret-2  
  envs:
    - secret-mysql.env
- name: secret-3  
  literals:
    - MYSQL_DB=mysql
    - MYSQL_PASS=root

	The `secret-1` key is using the `password.txt` file.
	The `secret-2` key is using the `secret-mysql.env` file.
	The `secret-3` key is using literal key-value pairs.

Generator Options

Kustomize provides a generatorOptions field to alter the default behavior of Kustomize generators. The configMapGenerator and secretGenerator fields append a hash suffix to the name of the generated resources.

Workload resources such as deployments do not detect any content changes to configuration maps and secrets. Any changes to a configuration map or secret do not apply automatically.

Because the generators append a hash, when you update the configuration map or secret, the resource name changes. This change triggers a rollout.

In some cases, the hash is not needed. Some operators observe the contents of the configuration maps and secrets that they use, and apply changes immediately. For example, the OpenShift OAuth operator applies changes to htpasswd secrets automatically. You can disable this feature with the generatorOptions field.

You can also add labels and annotations to the generated resources by using the generatorOptions field.

The following example shows the use of the generatorOptions field.

...output omitted...
configMapGenerator:
- name: my-configmap
  literals:
    - name="configmap-3"
    - description="literal key-value pair"
generatorOptions:
  disableNameSuffixHash: true
  labels:
    type: generated-disabled-suffix
  annotations:
    note: generated-disabled-suffix

You can use the kubectl kustomize command to render the changes to verify their effect.

[user@host base]$ kubectl kustomize .
apiVersion: v1
data:
  description: literal key-value pair
  name: configmap-3
kind: ConfigMap
metadata:
  annotations:
    note: generated-disabled-suffix
  labels:
    type: generated-disabled-suffix
  name: my-configmap

The my-configmap configuration map is without a hash suffix, and has a label and annotations that are defined in the kustomization file.

References

Declarative Management of Kubernetes Objects Using Kustomize

Your Guide to Continuous Delivery with OpenShift GitOps and Kustomize

Customization of Kubernetes YAML Configurations

JavaScript Object Notation (JSON) Patch

Notes on the Strategic Merge Patch

Discuss Red Hat OpenShift Administration II: Configuring a Production Cluster

Go to community

Red Hat OpenShift Administration II: Operating a Production Kubernetes Cluster (DO280)

Haley_Ruccio

26 lip 2023

Configure, manage, and troubleshoot OpenShift clusters and containerized applicationsRed Hat OpenShift Administration II: Operating a Production Kubernetes Cluster (DO280) prepares OpenShift Cluster Administrators to perform daily administration tasks on clusters that host applications provided by internal teams and external vendors, enable self-service for cluster users with different roles, and deploy applications that require special permissions such as such as CI/CD tooling, performance monitoring, and security scanners. This course focuses on configuring multi-tenancy and security features of OpenShift as well as managing OpenShift add-ons based on operators.The skills you learn in this course can be applied using all versions of OpenShift, including Red Hat OpenShift on AWS (ROSA), Azure Red Hat OpenShift, and OpenShift Container Platform.This course is based on OpenShift Container Platform 4.12.5-10 Course TopicsDeploying packaged applications using manifests, templates, kustomize, and helm.Configuring authentication and authorization for users and applications.Protecting network traffic with network policies and exposing applications with proper network access.Deploying and managing applications using resources manifests.Enabling developer self-service of application projects.Managing OpenShift cluster updates and Kubernetes operator updates.Target AudienceSystem Administrators and Platform Operators interested in the ongoing management of OpenShift clusters, applications, users, and add-ons.Site Reliability Engineers interested in the ongoing maintenance and troubleshooting of Kubernetes clusters.System and Software Architects interested in understanding the security of an OpenShift cluster.Recommended TrainingTake our free assessment to gauge whether this offering is the best fit for your skills.Prerequisite: Red Hat OpenShift I: Containers & Kubernetes (DO180 v4.12), or equivalent skills deploying and managing Kubernetes applications using the OpenShift web console and command-line interfaces.Technology considerationsThis course requires internet access to access the cloud-based classroom environment that provides an OpenShift cluster and a remote administrator’s workstation.

476

About pod security standards and warnings

alexcorcoles

17 kwi 2023

(This material has been created with the inputs of instructors and other people. We are trying to get this material published as soon as possible to address some concerns with DO180 and DO280.)With the default configuration, any namespace that you create on OpenShift 4.12 has the following labels: pod-security.kubernetes.io/audit: restrictedpod-security.kubernetes.io/audit-version: v1.24pod-security.kubernetes.io/warn: restrictedpod-security.kubernetes.io/warn-version: v1.24 These labels activate auditing and warning of the restricted pod security standard. The Kubernetes documentation describes the restricted pod security standard. Pod security standards such as restricted are not directly related to security context constraints such as the restricted SCC.Kubernetes uses an admission controller to reject pods that do not comply with pod security standards. An admission controller is a Kubernetes component that inspects every Kubernetes resource before its creation, and acts before Kubernetes creates the resource. This action can include rejecting the creation of the resource, or modifying the resource.The pod security standards are policies that include properties that workloads must follow. For example, the restricted standard requires that containers set the securityContext.allowPrivilegeEscalation field to false.Pod security standards apply both to workloads, such as deployments and jobs, and to pods.When a pod is created, the admission controller inspects the container specifications and enumerates the fields that violate the pod security standard of the namespace. The admission controller inspects the pod-security labels in the namespace to decide which pod security standard to apply (restricted, baseline, or privileged), and which action to take on violations. Namespaces can audit, warn, or enforce pod security standard violations. When the pod-security.kubernetes.io/audit annotation is set to restricted, the admission controller prevents the creation of pods that violate the security standard. When the pod-security.kubernetes.io/audit and pod-security.kubernetes.io/warn annotations are defined in the project, the admission controller does not block the creation of pods, but registers audit events or warnings.The admission controller does not enforce pod security standards for workloads. If the namespace is configured to enforce a pod security standard, then you can create a violating workload. The admission controller enforces only pod security policies on pods.Cluster administrators can use pod security standards to ensure that pods and workloads in namespaces limit their privileges. By limiting privileges, cluster administrators can mitigate what malicious pods can do.The default configuration of user namespaces in OpenShift does not restrict any workload; it only notifies users when they create workloads that do not follow pod security standards that might be enforced in future OpenShift versions.Security Context ConstraintsOpenShift introduced security context constraints (SCC) before the introduction of pod security standards in Kubernetes. SCCs automatically make security modifications to pods when they are created. By default, the system:openshift:scc:restricted-v2 cluster role binding applies to all authenticated users, so regular user workloads use the restricted-v2 SCC. For example, when you execute the following command to create a workload without any security configuration, the SCC adds security configuration to the pods: [user@host ~]$ oc create deployment test --image registry.ocp4.example.com:8443/redhattraining/hello-world-nginx (You can add the --dry-run=client -o yaml options to view the deployment manifest. The deployment manifest does not contain the security properties, which are described later.)If you inspect the pods that the deployment created, the SCC makes the following changes: [user@host ~]$ oc get pod test-b8d5658b6-7bxfs -o yamlapiVersion: v1kind: Podmetadata: annotations:... output omitted... openshift.io/scc: restricted-v2 seccomp.security.alpha.kubernetes.io/pod: runtime/default...output omitted...spec: containers: - image: registry.ocp4.example.com:8443/redhattraining/hello-world-nginx imagePullPolicy: Always name: hello-world-nginx resources: {} securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000690000... output omitted... securityContext: fsGroup: 1000690000 seLinuxOptions: level: s0:c26,c20 seccompProfile: type: RuntimeDefault... output omitted... These restrictions satisfy the restricted pod security policy.The result of these defaults means that when creating a workload (such as a deployment) in a namespace without any customization, the restricted pod security standard applies, but the admission controller only warns and audits. If the workload does not satisfy the pod security standard, then the admission controller allows the workload to be created, but prints a message such as the following warning: Warning: would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (container "example" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "example" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "database" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "example" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") However, by default Kubernetes creates the deployment pods with the restricted-v2 SCC, with the restrictions that the previous warning included. Due to the following factors, workloads generate only a warning, and the pod security standard admission controller allows their pods:The pod security standard admission controller never enforces pod security standards on workloads (only on pods).By default, namespaces do not enforce pod security standards on pods.By default, regular workloads create pods by using the restricted-v2 SCC, which satisfies the restricted pod security standard.Even if future versions of OpenShift change the default namespace configuration to enforce the restricted pod security standard, users will continue to be able to create workloads without explicitly declaring the restrictions that the pod security standard requires, because the restricted-v2 SCC will apply such restrictions automatically.You can take the following approaches to handling pod security:For most workloads, the restricted-v2 SCC limits pod privileges adequately. When creating a workload, you receive the warning, but the workload creates pods and works correctly.You can also create your workloads by providing a full specification of the security constraints to apply, without relying on the SCC to provide the configuration. By following this approach, privileges are more explicit, and you have greater control, so you can adjust more precisely the specific privileges that are granted to workloads.When the restricted-v2 SCC and the default restricted pod security standard are not adequate for your workload, because either you require further privileges, or you need further restrictions, you must use an existing SCC, or create a custom SCC, and provide explicit security specifications. You can also adjust the namespace labels to choose a different pod security standard, and decide to audit, log, or enforce the standard.

1473

Welcome to the Red Hat OpenShift Administration II (DO280) group in the Red Hat Learning Community!

Deanna

17 kwi 2023

We are excited to launch a space dedicated to the Red Hat Training course Red Hat OpenShift Administration II: Operating a Production Kubernetes Cluster!To gain the most value from this group - click the "Join Group" button in the upper right hand corner.We encourage group members to collaborate in this group to discuss topics, ask questions, share best practices and tips, provide course feedback, and share their accomplishments as it relates to DO280.Read more about Red Hat OpenShift Administration II here.Thanks!

1334

Revision: do280-4.14-08d11e1