Configure a project that requires custom settings.
Secure applications by encrypting and restricting network traffic.
Automate cluster maintenance tasks.
Outcomes
Create a project quota.
Create a limit range.
Use role-based access control to grant permissions to service accounts and groups.
Encrypt the traffic end-to-end with TLS by using a signed certificate.
Restrict cluster internal traffic to pods by using network policies.
Grant application access to Kubernetes APIs.
Configure a cluster maintenance application to run periodically.
As the student user on the workstation machine, use the lab command to prepare your system for this exercise.
[student@workstation ~]$ lab start compreview-apps
The lab command copies the exercise files into the ~/DO280/labs/compreview-apps directory and creates the workshop-support group with the do280-support user.
The lab command also restores the project template configuration from the previous exercise.
The goal, as a cluster administrator, is to prepare the workshop-support namespace for the support team.
Create a namespace instead of a project to avoid using the project template.
The project template applies a default configuration for workshop projects, and does not apply the configuration to the workshop-support namespace.
Then, as a support team member, you configure and deploy the applications that maintain the cluster and support the workshop experience.
You must set up an application that automatically deletes completed workshops, and set up a social media API that attendees from all workshops use.
Specifications
Create the workshop-support namespace with the category: support label.
Grant to the workshop-support group the admin role in the cluster.
Workloads from the workshop-support namespace must enforce the following constraints:
The project uses up to 4 CPUs.
The project uses up to 4 Gi of RAM.
The project requests up to 3.5 CPUs.
The project requests up to 3 Gi of RAM.
Define the default resource specification for workloads:
A default limit of 300m CPUs.
A default limit of 400 Mi of RAM.
A default request of 100m CPUs.
A default request of 250 Mi of RAM.
Any quota or limit range must have the workshop-support name for grading purposes.
As the do280-support user, deploy the project-cleaner application from the project-cleaner/example-pod.yaml file to the workshop-support namespace by using a project-cleaner cron job that runs every minute.
The project cleaner deletes projects with the workshop label that exist for more than 10 seconds.
This short expiration time is deliberate for this lab.
You must create a project-cleaner-sa service account to use in the project cleaner application.
The role that the project cleaner needs is defined in the project-cleaner/cluster-role.yaml file.
Deploy the beeper-db database in the beeper-api/beeper-db.yaml file to the workshop-support namespace.
Deploy the beeper-api application in the beeper-api/deployment.yaml file to the workshop-support namespace.
You must configure this application to use TLS end-to-end by using the following specification:
Use the beeper-api.pem certificate and the beeper-api.key in the certs directory.
Configure the /etc/pki/beeper-api/ path as the mount point for the certificate and key.
Set the TLS_ENABLED environment variable to the true value.
Update the startup, readiness, and liveness probes to use TLS.
Create a passthrough route with the beeper-api.apps.ocp4.example.com hostname.
The database pods, which are pods in the workshop-support namespace with the app=beeper-db label, must accept only TCP traffic from the beeper-api pods in the workshop-support namespace on the 5432 port.
You can use the category=support label to identify the pods that belong to the workshop-support namespace.
Configure the cluster network so that the workshop-support namespace accepts only external ingress traffic to pods that listen on the 8080 port, and blocks traffic from other projects.
Change to the ~/DO280/labs/compreview-apps directory and log in to the cluster as the admin user.
Open a terminal window and change to the lab directory.
[student@workstation ~]$ cd ~/DO280/labs/compreview-appsLog in as the admin user with the redhatocp password.
[student@workstation compreview-apps]$ oc login -u admin -p redhatocp \
https://api.ocp4.example.com:6443
Login successful.
...output omitted...Create and prepare the workshop-support namespace with the following actions:
Add the category=support label.
Grant the admin cluster role to the workshop-support group.
Create the workshop-support namespace.
[student@workstation compreview-apps]$ oc create namespace workshop-support
namespace/workshop-support createdUse the oc label command to add the category=support label to the workshop-support namespace.
[student@workstation beeper-api]$ oc label namespace \
workshop-support category=support
namespace/workshop-support labeledChange to the workshop-support namespace by using the oc project command.
[student@workstation beeper-api]$ oc project workshop-support
Now using project "workshop-support" on server...Create a cluster role binding to assign the admin cluster role to the workshop-support group.
[student@workstation compreview-apps]$ oc adm policy \
add-cluster-role-to-group admin workshop-support
clusterrole.rbac.authorization.k8s.io/admin added: "workshop-support"Create the resource quota for the workshop-support namespace with the following specification.
| Quota | Value |
|---|---|
limits.cpu
|
4
|
limits.memory
|
4Gi
|
requests.cpu
|
3500m
|
requests.memory
|
3Gi
|
Run the oc create quota command to create the quota.
[student@workstation compreview-apps]$ oc create quota workshop-support \
--hard=limits.cpu=4,limits.memory=4Gi,requests.cpu=3500m,requests.memory=3Gi
resourcequota/workshop-support createdCreate the workshop limit range with the following specification.
| Limit type | Value |
|---|---|
default.cpu
|
300m
|
default.memory
|
400Mi
|
defaulRequest.cpu
|
100m
|
defaulRequest.memory
|
250Mi
|
Edit the limitrange.yaml file and replace the CHANGE_ME label to match the following definition.
apiVersion: v1
kind: LimitRange
metadata:
name: workshop-support
namespace: workshop-support
spec:
limits:
- default:
cpu: 300m
memory: 400Mi
defaultRequest:
cpu: 100m
memory: 250Mi
type: ContainerUse the oc apply command to create the limit range in the workshop-support project.
[student@workstation compreview-apps]$ oc apply -f limitrange.yaml
limitrange/workshop-support createdCreate the project-cleaner-sa service account in the workshop-support namespace.
Then, assign the role from the project-cleaner/cluster-role.yaml file to the project-cleaner-sa service account.
Create the project-cleaner-sa service account.
[student@workstation compreview-apps]$ oc create sa project-cleaner-sa
serviceaccount/project-cleaner-sa createdChange to the ~/DO280/labs/compreview-apps/project-cleaner directory to access the application files.
[student@workstation compreview-apps]$ cd \
~/DO280/labs/compreview-apps/project-cleanerCreate the project-cleaner cluster role by applying the cluster-role.yaml manifest file.
[student@workstation project-cleaner]$ oc apply -f cluster-role.yaml
clusterrole.rbac.authorization.k8s.io/project-cleaner createdUse the oc adm policy add-cluster-role-to-user command to add the project-cleaner role to the project-cleaner-sa service account.
[student@workstation project-cleaner]$ oc adm policy add-cluster-role-to-user \
project-cleaner -z project-cleaner-sa
clusterrole.rbac.authorization.k8s.io/project-cleaner added: "project-cleaner-sa"As the do280-support user, create the project-cleaner cron job by editing the cron-job.yaml file and by using the example-pod.yaml pod manifest as the job template.
Configure the cron job to run every minute.
Log in as the do280-support user with the redhat password.
[student@workstation project-cleaner]$ oc login -u do280-support -p redhat
Login successful.
...output omitted...Edit the cron-job.yaml file:
Replace the CHANGE_ME label with the "*/1 * * * *" schedule to execute the job every minute.
Replace the CHANGE_ME label in the jobTemplate definition with the spec definition from the example-pod.yaml pod manifest.
Replace the CHANGE_ME label in the serviceAccountName key with the project-cleaner-sa service account.
Although the long image name might show across two lines, you must add it as one line.
A solution file is in the ~/DO280/solutions/compreview-apps/project-cleaner/cron-job.yaml path.
apiVersion: batch/v1 kind: CronJob metadata: name: project-cleaner namespace: workshop-support spec: schedule:"*/1 * * * *"concurrencyPolicy: Forbid jobTemplate: spec: template: spec:restartPolicy: Never serviceAccountName: project-cleaner-sa containers: - name: project-cleaner image: registry.ocp4.example.com:8443/redhattraining/do280-project-cleaner:v1.1 imagePullPolicy: Always env: - name: "PROJECT_TAG" value: "workshop" - name: "EXPIRATION_SECONDS" value: "10" resources: limits: cpu: 100m memory: 200Mi
Create the cron job.
[student@workstation project-cleaner]$ oc apply -f cron-job.yaml
cronjob.batch/project-cleaner createdIt is safe to ignore pod security warnings for exercises in this course. OpenShift uses the Security Context Constraints controller to provide safe defaults for pod security.
Verify that the project cleaner application is deployed correctly, by creating a clean-test project.
[student@workstation project-cleaner]$ oc new-project clean-test
Now using project "clean-test" on server...
...output omitted...Change to the workshop-support namespace.
[student@workstation project-cleaner]$ oc project workshop-support
Now using project "workshop-support" on server...Wait for a successful job run. Then, get the pod name from the last job run.
[student@workstation project-cleaner]$oc get jobs,podsNAME COMPLETIONS DURATION AGE job.batch/project-cleaner-27949859 1/1 7s 2m40s job.batch/project-cleaner-27949860 1/1 7s 100s job.batch/project-cleaner-27949861 1/1 6s 40s NAME READY STATUS RESTARTS AGE pod/project-cleaner-27949859-f98vj 0/1 Completed 0 2m40s pod/project-cleaner-27949860-j8td5 0/1 Completed 0 100spod/project-cleaner-27949861-p262t 0/1 Completed 0 40s
Read the logs of the pod that completed the job.
[student@workstation project-cleaner]$ oc logs \
pod/project-cleaner-27949861-p262t
Listing namespaces with label workshop:
- namespace: clean-test, created 55.327453 seconds ago...
Deleting namespaces: clean-test
Namespace 'clean-test' deletedYou might see deleted projects from other exercises in the course.
Verify that the cron job deletes the clean-test project, by using the oc get project command.
[student@workstation project-cleaner]$ oc get project clean-test
Error from server (NotFound): namespaces "clean-test" not foundCreate the beeper database by applying the beeper-api/beeper-db.yaml file.
Change to the ~/DO280/labs/compreview/beeper-api directory to access the application files.
[student@workstation project-cleaner]$ cd ~/DO280/labs/compreview-apps/beeper-apiUse the oc apply command to create the database in the workshop-support namespace.
[student@workstation beeper-api]$ oc apply -f beeper-db.yaml
secret/beeper-db created
service/beeper-db created
persistentvolumeclaim/beeper-db created
deployment.apps/beeper-db createdVerify that the database pod is running by using the oc get pod command to get the pods with the app=beeper-db label.
[student@workstation beeper-api]$ oc get pod -l app=beeper-db
NAME READY STATUS RESTARTS AGE
beeper-db-688756744f-rgxpg 1/1 Running 0 3m51sConfigure TLS on the beeper-api deployment by using a signed certificate by a corporate CA to accept TLS connections from outside the cluster.
You have the CA certificate and the signed certificate for the beeper-api.apps.ocp4.example.com domain in the beeper-api/certs directory of the lab.
Use the following settings in the deployment to configure TLS:
Set the path for the certificate and key to /etc/pki/beeper-api/.
Set the TLS_ENABLED environment variable to the true value.
Update the startup, readiness, and liveness probes to use TLS.
Create the beeper-api-cert secret by using the beeper-api.pem certificate and the beeper-api.key key from the lab directory.
[student@workstation beeper-api]$ oc create secret tls beeper-api-cert \
--cert certs/beeper-api.pem --key certs/beeper-api.key
secret/beeper-api-cert createdEdit the beeper-api deployment in the deployment.yaml file to mount the beeper-api-cert secret on the /etc/pki/beeper-api/ path.
apiVersion: apps/v1
kind: Deployment
metadata:
name: beeper-api
namespace: workshop-support
spec:
...output omitted...
spec:
containers:
- name: beeper-api
...output omitted...
env:
- name: TLS_ENABLED
value: "false"
volumeMounts:
- name: beeper-api-cert
mountPath: /etc/pki/beeper-api/
volumes:
- name: beeper-api-cert
secret:
defaultMode: 420
secretName: beeper-api-certEdit the beeper-api deployment in the deployment.yaml file to configure TLS for the application and for the startup, readiness, and liveness probes.
apiVersion: apps/v1 kind: Deployment metadata: name: beeper-api namespace: workshop-support spec: ...output omitted... spec: containers: - name: beeper-api ...output omitted... ports: - containerPort: 8080 readinessProbe: httpGet: port: 8080 path: /readyzscheme: HTTPSlivenessProbe: httpGet: port: 8080 path: /livezscheme: HTTPSstartupProbe: httpGet: path: /readyz port: 8080scheme: HTTPSfailureThreshold: 30 periodSeconds: 3 env: - name: TLS_ENABLEDvalue: "true"...output omitted...
Use the oc apply command to create the beeper-api deployment.
[student@workstation beeper-api]$ oc apply -f deployment.yaml
deployment.apps/beeper-api createdEdit the service.yaml file to configure the beeper-api service to listen on the standard HTTPS 443 port and to forward connections to pods with the app: beeper-api label on port 8080.
apiVersion: v1
kind: Service
metadata:
name: beeper-api
namespace: workshop-support
spec:
selector:
app: beeper-api
ports:
- port: 443
targetPort: 8080
name: httpsUse the oc apply command to create the beeper-api service.
[student@workstation beeper-api]$ oc apply -f service.yaml
service/beeper-api createdExpose the beeper API to outer cluster access by using the FQDN in the signed certificate by the corporate CA.
Create a passthrough route for the beeper-api service by using the beeper-api.apps.ocp4.example.com hostname.
[student@workstation beeper-api]$ oc create route \
passthrough beeper-api-https \
--service beeper-api \
--hostname beeper-api.apps.ocp4.example.com
route.route.openshift.io/beeper-api-https createdUse the curl command to the https://beeper-api.apps.ocp4.example.com/api/beeps URL to verify that the beeper API is accessible from outside the cluster.
Add the --cacert option to accept the certs/ca.pem CA.
[student@workstation beeper-api]$ curl -s --cacert \
certs/ca.pem https://beeper-api.apps.ocp4.example.com/api/beeps; echo
[]Optionally, open a web browser and verify that you can access the API by navigating to the https://beeper-api.apps.ocp4.example.com/swagger-ui.html URL.
When you see the warning about the security risk, click and then click .
Configure network policies to allow only TCP ingress traffic on port 5432 to database pods from the beeper-api pods.
Verify that you can access the beeper-db service from the workshop-support namespace by testing TCP connectivity to the database service.
Use the oc debug command to create a pod with the nc command with the -z option to test TCP access.
[student@workstation beeper-api]$oc debug --to-namespace="workshop-support" -- \ nc -z -v beeper-db.workshop-support.svc.cluster.local 5432Starting pod/image-debug ... Ncat: Version 7.70 ( https://nmap.org/ncat ) Ncat:Connected to 172.30.219.94:5432.Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds. Removing debug pod ...
Create an entry in the database by using the following curl command.
[student@workstation beeper-api]$ curl -s --cacert certs/ca.pem -X 'POST' \
'https://beeper-api.apps.ocp4.example.com/api/beep' \
-H 'Content-Type: application/json' \
-d '{ "username": "user1", "content": "first message" }'Edit the db-networkpolicy.yaml file so that only pods with the app: beeper-api label can connect to database pods.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: database-policy
namespace: workshop-support
spec:
podSelector:
matchLabels:
app: beeper-db
ingress:
- from:
- namespaceSelector:
matchLabels:
category: support
podSelector:
matchLabels:
app: beeper-api
ports:
- protocol: TCP
port: 5432Create the network policy.
[student@workstation beeper-api]$ oc apply -f db-networkpolicy.yaml
networkpolicy.networking.k8s.io/beeper-api-ingresspolicy createdVerify that you cannot connect to the database, by running the previous nc command.
[student@workstation beeper-api]$oc debug --to-namespace="workshop-support" -- \ nc -z -v beeper-db.workshop-support.svc.cluster.local 5432Starting pod/image-debug ... Ncat: Version 7.70 ( https://nmap.org/ncat ) Ncat:Connection timed out.Removing debug pod ...
Verify that the API pods have access to the database pods, by running the curl command to query the API by using the external route.
[student@workstation beeper-api]$ curl -s --cacert \
certs/ca.pem https://beeper-api.apps.ocp4.example.com/api/beeps; echo
[{"id":1,"username":"user1","content":"first message","votes":0}]Configure network policies in the workshop-support namespace to accept only ingress connections from the OpenShift router pods to port 8080.
Verify that you can access the API service from the workshop-support namespace by testing TCP connectivity.
Use the oc debug command to create a pod with the nc command with the -z option to test TCP access.
[student@workstation beeper-api]$ oc debug --to-namespace="workshop-support" -- \
nc -z -v beeper-api.workshop-support.svc.cluster.local 443
Starting pod/image-debug ...
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 172.30.32.28:443.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.
Removing debug pod ...Edit the beeper-api-ingresspolicy.yaml file to accept ingress connections from router pods by adding a namespace selector with the policy-group.network.openshift.io/ingress label.
apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: beeper-api-ingresspolicy namespace: workshop-support spec: podSelector:{}ingress: - from:- namespaceSelector: matchLabels: policy-group.network.openshift.io/ingress: ""ports:- protocol: TCP port: 8080
Create the network policy.
[student@workstation beeper-api]$ oc apply -f beeper-api-ingresspolicy.yaml
networkpolicy.networking.k8s.io/beeper-api createdVerify that you cannot access the API service from the workshop-support namespace.
Use the oc debug command to create a pod with the nc command with the -z option to test TCP access.
[student@workstation beeper-api]$oc debug --to-namespace="workshop-support" -- \ nc -z -v beeper-api.workshop-support.svc.cluster.local 443Starting pod/image-debug ... Ncat: Version 7.70 ( https://nmap.org/ncat ) Ncat:Connection timed out.Removing debug pod ...
Verify that the API pods are accessible from outside the cluster by running the curl command to query the API external route.
[student@workstation beeper-api]$ curl -s --cacert \
certs/ca.pem https://beeper-api.apps.ocp4.example.com/livez; echo
{"status":"UP"}Change to the home directory.
[student@workstation appsec-review]$ cd