DO280 - ch04s05

Bookmark this page

Protect Internal Traffic with TLS

Objectives

Configure and use automatic service certificates.

Zero-trust Environments

Zero-trust environments assume that every interaction begins in an untrusted state. Users can access only files or objects that are specifically allowed; communication must be encrypted; and client applications must verify the authenticity of servers.

By default, OpenShift encrypts network traffic between nodes and the control plane, and prevents external entities from reading internal traffic. This encryption provides stronger security than default Kubernetes, which does not automatically encrypt internal traffic. Although the control plane traffic is encrypted, applications in OpenShift do not necessarily verify the authenticity of other applications or encrypt application traffic.

Zero-trust environments require that a trusted certificate authority (CA) signs the certificates that are used to encrypt traffic. By referencing the CA certificate, an application can cryptographically verify the authenticity of another application with a signed certificate.

Service Certificates

OpenShift provides the service-ca controller to generate and sign service certificates for internal traffic. The service-ca controller creates a secret that it populates with a signed certificate and key. A deployment can mount this secret as a volume to use the signed certificate. Additionally, client applications need to trust the service-ca controller CA.

Service Certificate Creation

To generate a certificate and key pair, apply the service.beta.openshift.io/serving-cert-secret-name=your-secret annotation to a service. The service-ca controller creates the your-secret secret in the same namespace if it does not exist, and populates it with a signed certificate and key pair for the service.

[user@host ~]$ oc annotate service hello \ 
    service.beta.openshift.io/serving-cert-secret-name=hello-secret 
service/hello annotated

	The `hello` service is annotated.
	The secret that contains the certificate and key pair is named `hello-secret`.

After OpenShift generates the secret, you must mount the secret in the application deployment. The location to place the certificate and key is application-dependent. The following YAML patch is for an NGINX deployment:

spec:
  template:
    spec:
      containers:
        - name: hello
          volumeMounts:
            - name: hello-volume 
              mountPath: /etc/pki/nginx/ 
      volumes:
        - name: hello-volume 
          secret:
            defaultMode: 420 
            secretName: hello-secret 
            items:
              - key: tls.crt 
                path: server.crt 
              - key: tls.key 
                path: private/server.key

	Defining the volume as `hello-volume`.
	The application-specific mount path.
	The read-write permissions that the application recommends.
	The secret that the earlier annotation defined.
	The secret has `tls.crt` as the signed certificate and `tls.key` as the key.
	The application-specific destinations for the certificate and key.

After mounting the secret to the application container, the application can use the signed certificate for TLS traffic.

Client Service Application Configuration

For a client service application to verify the validity of a certificate, the application needs the CA bundle that signed that certificate. The service-ca controller injects the CA bundle when you apply the service.beta.openshift.io/inject-cabundle=true annotation to an object. You can apply the annotation to configuration maps, API services, custom resource definitions (CRD), mutating webhooks, and validating webhooks.

Configuration Maps: Apply the service.beta.openshift.io/inject-cabundle=true annotation to a configuration map to inject the CA bundle into the data: { service-ca.crt } field. The service-ca controller replaces all data in the selected configuration map with the CA bundle. You must therefore use a dedicated configuration map to prevent overwriting existing data.

[user@host ~]$ oc annotate configmap ca-bundle \
    service.beta.openshift.io/inject-cabundle=true
configmap/ca-bundle annotated

API service: Applying the annotation to an API service injects the CA bundle into the spec.caBundle field.
CRD: Applying the annotation to a CRD injects the CA bundle into the spec.conversion.webhook.clientConfig.caBundle field.
Mutating or validating webhook: Applying the annotation to a mutating webhook or validating webhook injects the CA bundle into the clientConfig.caBundle field.

Key Rotation

The service CA certificate is valid for 26 months by default and is automatically rotated after 13 months. After rotation is a 13-month grace period where the original CA certificate is still valid. During this grace period, each pod that is configured to trust the original CA certificate must be restarted in some way. A service restart automatically injects the new CA bundle.

You can also manually rotate the certificate for the service CA and for generated service certificates. To rotate a generated service certificate, delete the existing secret, and the service-ca controller automatically generates a new one.

[user@host ~]$ oc delete secret certificate-secret
secret/certificate-secret deleted

To manually rotate the service CA certificate, delete the signing-key secret in the openshift-service-ca namespace.

[user@host ~]$ oc delete secret/signing-key -n openshift-service-ca
secret/signing-key deleted

This process immediately invalidates the former service CA certificate. You must restart all pods that use it, for TLS to function.

Alternatives to Service Certificates

Other options can handle TLS encryption inside an OpenShift cluster, such as a service mesh or the certmanager operator.

You can use the certmanager operator to delegate the certificate signing process to a trusted external service, and also to renew a certificate.

You can also use Red Hat OpenShift Service Mesh for encrypted service-to-service communication and for other advanced features. Service mesh is an advanced topic and is not covered in the course.

Patching Kubernetes Resources

You can modify objects in OpenShift in a repeatable way with the oc patch command. The oc patch command updates or adds fields in an existing object from a provided JSON or YAML snippet or file. A software developer might distribute a patch file or snippet to fix problems before a full update is available.

To patch an object from a snippet, use the oc patch command with the -p option and the snippet. The following example updates the hello deployment to have a CPU resource request of 100m with a JSON snippet:

[user@host ~]$ oc patch deployment hello -p \
    '{"spec":{"template":{"spec":{"resources":{"requests":{"cpu": "100m"}}}}}}'
deployment/hello patched

To patch an object from a patch file, use the oc patch command with the --patch-file option and the location of the patch file. The following example updates the hello deployment to include the content of the ~/volume-mount.yaml patch file:

[user@host ~]$ oc patch deployment hello --patch-file ~/volume-mount.yaml
deployment.apps/hello patched

The contents of the patch file describe mounting a persistent volume claim as a volume:

spec:
  template:
    spec:
      containers:
        - name: hello
          volumeMounts:
            - name: www
              mountPath: /usr/share/nginx/html/
      volumes:
        - name: www
          persistentVolumeClaim:
            claimName: nginx-www

This patch results in the following manifest for the hello deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  ...output omitted...
spec:
  ...output omitted...
  template:
    ...output omitted...
    spec:
      containers:
        ...output omitted...
        name: server
        ...output omitted...
        volumeMounts:
        - mountPath: /usr/share/nginx/html/
          name: www
        - mountPath: /etc/nginx/conf.d/
          name: tls-conf
      ...output omitted...
      volumes:
      - configMap:
          defaultMode: 420
          name: tls-conf
        name: tls-conf
      - persistentVolumeClaim:
          claimName: nginx-www
        name: www
...output omitted...

The patch applies to the hello deployment regardless of whether the www volume mount exists. The oc patch command modifies existing fields in the object that are specified in the patch. If the beginning state of the hello deployment already contains data as follows, then the end result is the same as if the fields do not exist:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  ...output omitted...
spec:
  ...output omitted...
  template:
    ...output omitted...
    spec:
      containers:
        ...output omitted...
        name: server
        ...output omitted...
        volumeMounts:
        - mountPath: /usr/share/nginx/www/ 
          name: www
        - mountPath: /etc/nginx/conf.d/
          name: tls-conf
      ...output omitted...
      volumes:
      - configMap:
          defaultMode: 420
          name: tls-conf
        name: tls-conf
      - persistentVolumeClaim:
          claimName: deprecated-www 
        name: www
...output omitted...

The www volume already exists. The patch replaces the existing data with the new data.

References

For more information about service certificates, refer to the Securing Service Traffic Using Service Serving Certificate Secrets section in the Configuring Certificates chapter in the Red Hat OpenShift Container Platform 4.14 Security and Compliance documentation at https://access.redhat.com/documentation/en-us/openshift_container_platform/4.14/html-single/security_and_compliance#add-service-serving

For more information about service mesh, refer to the About OpenShift Service Mesh section in the Service Mesh 2.x chapter in the Red Hat OpenShift Container Platform 4.14 Service Mesh documentation at https://access.redhat.com/documentation/en-us/openshift_container_platform/4.14/html-single/service_mesh#ossm-about

For more information about the cert-manager operator, refer to the cert-manager Operator for Red Hat OpenShift chapter in the Red Hat OpenShift Container Platform 4.14 Security and Compliance documentation at https://access.redhat.com/documentation/en-us/openshift_container_platform/4.14/html-single/security_and_compliance#cert-manager-operator-for-red-hat-openshift

For more information about the oc patch command, refer to the oc patch section in the OpenShift CLI Developer Command Reference chapter in the Red Hat OpenShift Container Platform 4.14 CLI Tools documentation at https://access.redhat.com/documentation/en-us/openshift_container_platform/4.14/html-single/cli_tools/index#oc-patch

Red Hat Topics - What Is Zero Trust?

Discuss Red Hat OpenShift Administration II: Configuring a Production Cluster

Go to community

Red Hat OpenShift Administration II: Operating a Production Kubernetes Cluster (DO280)

Haley_Ruccio

26 lip 2023

Configure, manage, and troubleshoot OpenShift clusters and containerized applicationsRed Hat OpenShift Administration II: Operating a Production Kubernetes Cluster (DO280) prepares OpenShift Cluster Administrators to perform daily administration tasks on clusters that host applications provided by internal teams and external vendors, enable self-service for cluster users with different roles, and deploy applications that require special permissions such as such as CI/CD tooling, performance monitoring, and security scanners. This course focuses on configuring multi-tenancy and security features of OpenShift as well as managing OpenShift add-ons based on operators.The skills you learn in this course can be applied using all versions of OpenShift, including Red Hat OpenShift on AWS (ROSA), Azure Red Hat OpenShift, and OpenShift Container Platform.This course is based on OpenShift Container Platform 4.12.5-10 Course TopicsDeploying packaged applications using manifests, templates, kustomize, and helm.Configuring authentication and authorization for users and applications.Protecting network traffic with network policies and exposing applications with proper network access.Deploying and managing applications using resources manifests.Enabling developer self-service of application projects.Managing OpenShift cluster updates and Kubernetes operator updates.Target AudienceSystem Administrators and Platform Operators interested in the ongoing management of OpenShift clusters, applications, users, and add-ons.Site Reliability Engineers interested in the ongoing maintenance and troubleshooting of Kubernetes clusters.System and Software Architects interested in understanding the security of an OpenShift cluster.Recommended TrainingTake our free assessment to gauge whether this offering is the best fit for your skills.Prerequisite: Red Hat OpenShift I: Containers & Kubernetes (DO180 v4.12), or equivalent skills deploying and managing Kubernetes applications using the OpenShift web console and command-line interfaces.Technology considerationsThis course requires internet access to access the cloud-based classroom environment that provides an OpenShift cluster and a remote administrator’s workstation.

476

About pod security standards and warnings

alexcorcoles

17 kwi 2023

(This material has been created with the inputs of instructors and other people. We are trying to get this material published as soon as possible to address some concerns with DO180 and DO280.)With the default configuration, any namespace that you create on OpenShift 4.12 has the following labels: pod-security.kubernetes.io/audit: restrictedpod-security.kubernetes.io/audit-version: v1.24pod-security.kubernetes.io/warn: restrictedpod-security.kubernetes.io/warn-version: v1.24 These labels activate auditing and warning of the restricted pod security standard. The Kubernetes documentation describes the restricted pod security standard. Pod security standards such as restricted are not directly related to security context constraints such as the restricted SCC.Kubernetes uses an admission controller to reject pods that do not comply with pod security standards. An admission controller is a Kubernetes component that inspects every Kubernetes resource before its creation, and acts before Kubernetes creates the resource. This action can include rejecting the creation of the resource, or modifying the resource.The pod security standards are policies that include properties that workloads must follow. For example, the restricted standard requires that containers set the securityContext.allowPrivilegeEscalation field to false.Pod security standards apply both to workloads, such as deployments and jobs, and to pods.When a pod is created, the admission controller inspects the container specifications and enumerates the fields that violate the pod security standard of the namespace. The admission controller inspects the pod-security labels in the namespace to decide which pod security standard to apply (restricted, baseline, or privileged), and which action to take on violations. Namespaces can audit, warn, or enforce pod security standard violations. When the pod-security.kubernetes.io/audit annotation is set to restricted, the admission controller prevents the creation of pods that violate the security standard. When the pod-security.kubernetes.io/audit and pod-security.kubernetes.io/warn annotations are defined in the project, the admission controller does not block the creation of pods, but registers audit events or warnings.The admission controller does not enforce pod security standards for workloads. If the namespace is configured to enforce a pod security standard, then you can create a violating workload. The admission controller enforces only pod security policies on pods.Cluster administrators can use pod security standards to ensure that pods and workloads in namespaces limit their privileges. By limiting privileges, cluster administrators can mitigate what malicious pods can do.The default configuration of user namespaces in OpenShift does not restrict any workload; it only notifies users when they create workloads that do not follow pod security standards that might be enforced in future OpenShift versions.Security Context ConstraintsOpenShift introduced security context constraints (SCC) before the introduction of pod security standards in Kubernetes. SCCs automatically make security modifications to pods when they are created. By default, the system:openshift:scc:restricted-v2 cluster role binding applies to all authenticated users, so regular user workloads use the restricted-v2 SCC. For example, when you execute the following command to create a workload without any security configuration, the SCC adds security configuration to the pods: [user@host ~]$ oc create deployment test --image registry.ocp4.example.com:8443/redhattraining/hello-world-nginx (You can add the --dry-run=client -o yaml options to view the deployment manifest. The deployment manifest does not contain the security properties, which are described later.)If you inspect the pods that the deployment created, the SCC makes the following changes: [user@host ~]$ oc get pod test-b8d5658b6-7bxfs -o yamlapiVersion: v1kind: Podmetadata: annotations:... output omitted... openshift.io/scc: restricted-v2 seccomp.security.alpha.kubernetes.io/pod: runtime/default...output omitted...spec: containers: - image: registry.ocp4.example.com:8443/redhattraining/hello-world-nginx imagePullPolicy: Always name: hello-world-nginx resources: {} securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000690000... output omitted... securityContext: fsGroup: 1000690000 seLinuxOptions: level: s0:c26,c20 seccompProfile: type: RuntimeDefault... output omitted... These restrictions satisfy the restricted pod security policy.The result of these defaults means that when creating a workload (such as a deployment) in a namespace without any customization, the restricted pod security standard applies, but the admission controller only warns and audits. If the workload does not satisfy the pod security standard, then the admission controller allows the workload to be created, but prints a message such as the following warning: Warning: would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (container "example" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "example" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "database" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "example" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost") However, by default Kubernetes creates the deployment pods with the restricted-v2 SCC, with the restrictions that the previous warning included. Due to the following factors, workloads generate only a warning, and the pod security standard admission controller allows their pods:The pod security standard admission controller never enforces pod security standards on workloads (only on pods).By default, namespaces do not enforce pod security standards on pods.By default, regular workloads create pods by using the restricted-v2 SCC, which satisfies the restricted pod security standard.Even if future versions of OpenShift change the default namespace configuration to enforce the restricted pod security standard, users will continue to be able to create workloads without explicitly declaring the restrictions that the pod security standard requires, because the restricted-v2 SCC will apply such restrictions automatically.You can take the following approaches to handling pod security:For most workloads, the restricted-v2 SCC limits pod privileges adequately. When creating a workload, you receive the warning, but the workload creates pods and works correctly.You can also create your workloads by providing a full specification of the security constraints to apply, without relying on the SCC to provide the configuration. By following this approach, privileges are more explicit, and you have greater control, so you can adjust more precisely the specific privileges that are granted to workloads.When the restricted-v2 SCC and the default restricted pod security standard are not adequate for your workload, because either you require further privileges, or you need further restrictions, you must use an existing SCC, or create a custom SCC, and provide explicit security specifications. You can also adjust the namespace labels to choose a different pod security standard, and decide to audit, log, or enforce the standard.

1473

Welcome to the Red Hat OpenShift Administration II (DO280) group in the Red Hat Learning Community!

Deanna

17 kwi 2023

We are excited to launch a space dedicated to the Red Hat Training course Red Hat OpenShift Administration II: Operating a Production Kubernetes Cluster!To gain the most value from this group - click the "Join Group" button in the upper right hand corner.We encourage group members to collaborate in this group to discuss topics, ask questions, share best practices and tips, provide course feedback, and share their accomplishments as it relates to DO280.Read more about Red Hat OpenShift Administration II here.Thanks!

1334

Revision: do280-4.14-08d11e1