The Kubernetes CLI uses both imperative and declarative commands. Imperative commands perform an action that is based on a command, and use command names that closely reflect the action. In contrast, declarative commands use a resource manifest file to declare the intended state of a resource.

A Kubernetes manifest is a YAML- or JSON-formatted file with declaration statements for Kubernetes resources such as deployments, pods, or services. Instead of using imperative commands to create Kubernetes resources, manifest files provide all the details for the resource in a single file. Working with manifest files enables the use of more reproducible processes. Instead of reproducing sequences of imperative commands, manifest files contain the entire definition of resources and can be applied in a single step. Using manifest files is also useful for tracking system configuration changes in a source code management system.

Given a new or updated resource manifest, Kubernetes provides commands that compare the intended state that is specified in the resource manifest to the current state of the resource. These commands then apply transformations to the current state to match the intended state.

An imperative workflow is useful for developing and testing. The following example uses the kubectl create deployment imperative command, to create a deployment for a MYSQL database.

[user@host ~]$ kubectl create deployment db-pod --port 3306 \
  --image registry.ocp4.example.com:8443/rhel8/mysql-80
deployment.apps/db-pod created

In addition to using verbs that reflect the action of the command, imperative commands use options to provide the details. The example command uses the --port and the --image options to provide the required details to create the deployment.

The use of imperative commands affects applying changes to live resources. For example, the pod from the previous deployment would fail to start due to missing environment variables. The following kubectl set env deployment imperative command resolves the problem by adding the required environment variables to the deployment:

[user@host ~]$ kubectl set env deployment/db-pod \
  MYSQL_USER='user1' \
  MYSQL_PASSWORD='mypa55w0rd' \
  MYSQL_DATABASE='items'
deployment.apps/db-pod updated

Executing this kubectl set env deployment command changes the deployment resource named db-pod, and provides the extra needed variables to start the container. A developer can continue building out the application, by using imperative commands to add components, such as services, routes, volume mounts, and persistent volume claims. With the addition of each component, the developer can run tests to ensure that the component correctly executes the intended function.

Imperative commands are useful for developing and experimenting. With imperative commands, a developer can build up an application one component at a time. When a component is added, the Kubernetes cluster provides error messages that are specific to the component. The process is analogous to using a debugger to step through code execution one line at a time. Using imperative commands usually provides clearer error messages, because an error occurs after adding a specific component.

However, long command lines and a fragmented application deployment are not ideal for deploying an application in production. With imperative commands, changes are a sequence of commands that must be maintained to reflect the intended state of the resources. The sequence of commands must be tracked and kept up to date.

Instead of tracking a sequence of commands, a manifest file captures the intended state of the sequence. In contrast to using imperative commands, declarative commands use a manifest file, or a set of manifest files, to combine all the details for creating those components into YAML files that can be applied in a single command. Future changes to the manifest files require only reapplying the manifests. Instead of tracking a sequence of complex commands, version control systems can track changes to the manifest file.

Although manifest files can also use the JSON syntax, YAML is generally preferred and is more popular. To continue the debugging analogy, debugging an application that is deployed from manifests is similar to trying to debug a full, completed running application. It can take more effort to find the source of the error, especially when the error is not a result of manifest errors.

Creating manifest files from scratch can take time. You can use the following techniques to provide a starting point for your manifest files:

The kubectl explain command provides the details for any field in the manifest. For example, use the kubectl explain deployment.spec.template.spec command to view field descriptions that specify a pod object within a deployment manifest.

To create a starter deployment manifest, use the kubectl create deployment command to generate a manifest by using the --dry-run=client option:

[user@host ~]$ kubectl create deployment hello-openshift -o yaml \
  --image registry.ocp4.example.com:8443/redhattraining/hello-world-nginx:v1.0 \
  --save-config \ 
  --dry-run=client \ 
  > ~/my-app/example-deployment.yaml

The following example shows a minimal deployment manifest file, not production-ready, for the hello-openshift deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
      ...output omitted...
  creationTimestamp: null
  labels:
    app: hello-openshift
  name: hello-openshift
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-openshift
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: hello-openshift
    spec:
      containers:
      - image: quay.io/redhattraining/hello-world-nginx:v1.0
        name: hello-world-nginx
        resources: {}
status: {}

When using imperative commands to create manifests, the resulting manifests might contain fields that are not necessary for creating a resource. For example, the following example changes the manifest by removing the empty and null fields. Removing unnecessary fields can significantly reduce the length of the manifests, and in turn reduce the overhead to work with them.

Additionally, you might need to further customize the manifests. For example, in a deployment, you might customize the number of replicas, or declare the ports that the deployment provides. The following notes explain the additional changes:

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: resource-manifests 
  labels:
    app: hello-openshift
  name: hello-openshift
spec:
  replicas: 2 
  selector:
    matchLabels:
      app: hello-openshift
  template:
    metadata:
      labels:
        app: hello-openshift
    spec:
      containers:
      - image: quay.io/redhattraining/hello-world-nginx:v1.0
        name: hello-world-nginx
        ports:
        - containerPort: 8080 
          protocol: TCP

You can create a manifest file for each resource that you manage. Alternatively, add each of the manifests to a single multi-part YAML file, and use a --- line to separate the manifests.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: resource-manifests
  annotations:
  ...output omitted...
---
apiVersion: v1
kind: Service
metadata:
  namespace: resource-manifests
  labels:
    app: hello-openshift
  name: hello-openshift
spec:
  ...output omitted...

Using a single file with multiple manifests versus using manifests that are defined in multiple manifest files is a matter of organizational preference. The single file approach has the advantage of keeping together related manifests. With the single file approach, it can be more convenient to change a resource that must be reflected across multiple manifests. In contrast, keeping manifests in multiple files can be more convenient for sharing resource definitions with others.

After creating manifests, you can test them in a non-production environment, or proceed to deploy the manifests. Validate the resource manifests before deploying applications in the production environment.

Declarative commands use a resource manifest instead of adding the details to many options on the command line. To create a resource, use the kubectl create -f resource.yaml command. Instead of a file name, you can pass a directory to the command to process all the resource files in a directory. Add the --recursive=true or -R option to recursively process resource files that are provided in multiple subdirectories.

The following example creates the resources from the manifests in the my-app directory. In this example, the my-app directory contains the example-deployment.yaml and service/example-service.yaml files from previously.

[user@host ~]$ tree my-app
my-app
├── example_deployment.yaml
└── service
    └── example_service.yaml

[user@host ~]$ kubectl create -R -f ~/my-app
deployment.apps/hello-openshift created
service/hello-openshift created

The command also accepts a URL:

[user@host ~]$ kubectl create -f \
  https://example.com/example-apps/deployment.yaml
deployment.apps/hello-openshift created

The kubectl apply command can also create resources with the same -f option that is illustrated with the kubectl create command. However, the kubectl apply command can also update a resource.

Updating resources is more complex than creating resources. The kubectl apply command implements several techniques to apply the updates without causing issues.

The kubectl apply command writes the contents of the configuration file to the kubectl.kubernetes.io/last-applied-configuration annotation. The kubectl create command can also generate this annotation by using the --save-config option. The kubectl apply command uses the last-applied-configuration annotation to identify fields that are removed from the configuration file and that must be cleared from the live configuration.

Although the kubectl create -f command can create resources from a manifest, the command is imperative and thus does not account for the current state of a live resource. Executing kubectl create -f against a manifest for a live resource gives an error. In contrast, the kubectl apply -f command is declarative, and considers the difference between the current resource state in the cluster and the intended resource state that is expressed in the manifest.

For example, to update the container's image from version v1.0 to latest, first update the YAML resource manifest to specify the new tag on the image. Then, use the kubectl apply command to instruct Kubernetes to create a version of the deployment resource by using the updated image version that is specified in the manifest.

Before applying the changes to the resource, use the --dry-run=server and the --validate=true flags to inspect the file for errors.

Any syntax errors in the YAML are included in the output. Most importantly, the --dry-run=server option prevents applying any changes to the Kubernetes runtime.

[user@host ~]$ kubectl apply -f ~/my-app/example-deployment.yaml \
  --dry-run=server --validate=true
deployment.apps/hello-openshift created (server dry-run) 

Use the kubectl diff command to review differences between live objects and manifests. When updating resource manifests, you can track differences in the changed files. However, many manifest changes, when applied, do not change the state of the cluster resources. A text-based diff tool would show all such differences, and result in a noisy output.

In contrast, using the kubectl diff command might be more convenient to preview changes. The kubectl diff command emphasizes the significant changes for the Kubernetes cluster. Review the differences to validate that manifest changes have the intended effect.

[user@host ~]$ kubectl diff -f example-deployment.yaml
...output omitted...
diff -u -N /tmp/LIVE-2647853521/apps.v1.Deployment.resource...
--- /tmp/LIVE-2647853521/apps.v1.Deployment.resource-manife...
+++ /tmp/MERGED-2640652736/apps.v1.Deployment.resource-mani...
@@ -6,7 +6,7 @@
     kubectl.kubernetes.io/last-applied-configuration: |
       ...output omitted...
   creationTimestamp: "2023-04-27T16:07:47Z"
-  generation: 1 
+  generation: 2
   labels:
     app: hello-openshift
   name: hello-openshift
@@ -32,7 +32,7 @@
         app: hello-openshift
     spec:
       containers:
-      - image: registry.ocp4.example.com:8443/.../hello-world-nginx:v1.0 
+      - image: registry.ocp4.example.com:8443/.../hello-world-nginx:latest
         imagePullPolicy: IfNotPresent
         name: hello-openshift
         ports:

Kubernetes resource controllers automatically add annotations and attributes to the live resource that make the output of other text-based diff tools misleading, by reporting many differences that have no impact on the resource configuration. Extracting manifests from live resources and making comparisons with tools such as the diff command reports many differences of no value. Using the kubectl diff command confirms that a live resource matches a resource configuration that a manifest provides. GitOps tools depend on the kubectl diff command to determine whether anyone changed resources outside the GitOps workflow. Because the tools themselves cannot know all details about how any controllers might change a resource, the tools defer to the cluster to determine whether a change is meaningful.

When using the oc diff command, recognize when applying a manifest change does not generate new pods. For example, if an updated manifest changes only values in secret or a configuration map, then applying the updated manifest does not generate new pods that use those values. Because pods read secret and configuration maps at startup, in this case applying the updated manifest leaves the pods in a vulnerable state, with stale values that are not synchronized with the updated secret or with the configuration map.

As a solution, use the oc rollout restart deployment deployment-name command to force a restart of the pods that are associated with the deployment. The forced restart generates pods that use the new values from the updated secret or configuration map.

In deployments with a single replica, you can also resolve the problem by deleting the pod. Kubernetes responds by automatically creating a pod to replace the deleted pod. However, for multiple replicas, using the oc rollout command to restart the pods is preferred, because the pods are stopped and replaced in a smart manner that minimizes downtime.

This course covers other resource management mechanisms that can automate or eliminate some of these challenges.

The kubectl create command attempts to create the specified resources in the manifest file. Using the kubectl create command generates an error if the targeted resources are already live in the cluster. In contrast, the kubectl apply command compares three sources to determine how to process the request and to apply changes.

If the specified resource in the manifest file does not exist, then the kubectl apply command creates the resource. If any fields in the last-applied-configuration annotation of the live resource are not present in the manifest, then the command removes those fields from the live configuration. After applying changes to the live resource, the kubectl apply command updates the last-applied-configuration annotation of the live resource to account for the change.

When creating a resource, the --save-config option of the kubectl create command produces the required annotations for future kubectl apply commands to operate.