Bookmark this page

Detect Deprecated Kubernetes API Usage

Objectives

  • Identify applications that use deprecated Kubernetes APIs.

OpenShift Versions

Kubernetes is an open source container orchestration engine for automating the deployment, scaling, and management of containerized applications. The OpenShift Container Platform foundation is based on Kubernetes and therefore shares the underlying technology. The following table lists the OpenShift version and the Kubernetes version that it is based on:

OpenShift versionKubernetes version
4.121.25
4.131.26
4.141.27

Kubernetes API Deprecation Policy

The Kubernetes API versions are categorized based on feature maturity (experimental, pre-release, and stable).

API versionCategoryDescription
v1alpha1AlphaExperimental features
v1beta1BetaPre-release features
v1StableStable features, generally available

Use the following command to view the current version of a resource:

[user@host ~]$ oc api-resources | egrep '^NAME|cronjobs'
NAME      SHORTNAMES  APIVERSION  NAMESPACED  KIND
cronjobs  cj          batch/v1    true        CronJob

When a stable version of a feature is released, the beta versions are marked as deprecated and are removed after three Kubernetes releases. If a request uses a deprecated API version, then the API server returns a deprecation warning that includes the name of the current version of the cluster.

[user@host ~]$ egrep 'kind|apiVersion' cronjob-beta.yaml
kind: CronJob
apiVersion: batch/v1beta1

[user@host ~]$ oc create -f cronjob-beta.yaml
Warning: batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
cronjob.batch/hello created

If a request uses an API version that Kubernetes removed, then the API server returns an error, because that API version is not supported in the cluster.

[user@host ~]$ egrep 'kind|apiVersion' cronjob-alpha.yaml
apiVersion: batch/v1alpha1
kind: CronJob

[user@host ~]$ oc create -f cronjob-alpha.yaml
error: resource mapping not found for name: "hello" namespace: "" from "cronjob-alpha.yaml": no matches for kind "CronJob" in version "batch/v1beta1"
ensure CRDs are installed first

Deprecated and Removed Features in Kubernetes

The Kubernetes 1.27 release stopped serving some API versions that were marked as deprecated in previous releases. The following table contains a short list of the deprecated and removed API versions.

ResourceRemoved API GroupCurrent API Group
CSIStorageCapacitystorage.k8s.io/​v1beta1storage.k8s.io/​v1

Note

For more information about the API versions that are deprecated and removed in Kubernetes, consult Kubernetes Deprecated API Migration Guide in the references section.

Identifying Deprecated APIs

You can identify from the API request count whether a workload uses a deprecated API version. The API request count output contains four columns. A value in the REMOVEDINRELEASE column indicates that the API version is deprecated and specifies the Kubernetes version that will remove it.

[user@host ~]$ oc get apirequestcounts | awk '{if(NF==4){print $0}}'
NAME
                    REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
...output omitted...
cronjobs.v1beta1.batch
                    1.25               15                      44
horizontalpodautoscalers.v2beta2.autoscaling
                    1.26               6                       30
podsecuritypolicies.v1beta1.policy
                    1.25               28                      77
...output omitted...

If the REMOVEDINRELEASE column is blank, then it indicates that the current API version is not deprecated, and that version will be kept in future releases.

Note

You can use a JSONPath filter to retrieve the results. The FILTER variable is written on a single line.

[user@host ~]$ FILTER='{range .items[?(@.status.removedInRelease!="")]}{.status.removedInRelease}{"\t"}{.status.requestCount}{"\t"}{.metadata.name}{"\n"}{end}'
[user@host ~]$ oc get apirequestcounts -o jsonpath="${FILTER}" | \
  column -t -N "RemovedInRelease,RequestCount,Name"
RemovedInRelease  RequestCount  Name
1.25              44            cronjobs.v1beta1.batch
...output omitted...

If the command does not retrieve any information, then it indicates that none of the installed APIs are deprecated.

You can use a JSONPath filter for a list of actions for that resource and who did them.

[user@host ~]$ FILTER='{range .status.currentHour..byUser[*]}{..byVerb[*].verb}{","}{.username}{","}{.userAgent}{"\n"}{end}'
[user@host ~]$ TYPE=apirequestcount.apiserver.openshift.io/cronjobs.v1.batch
[user@host ~]$ echo ${TYPE} ; oc get ${TYPE} -o jsonpath="${FILTER}" | \
  column -t -s ',' -N "Verbs,Username,UserAgent"

apirequestcount.apiserver.openshift.io/cronjobs.v1.batch
Verbs        Username    				  UserAgent
get update   system:serviceaccount:kube-system:cronj...   kube-controller-manager/v1...
watch        system:kube-controller-manager               kube-controller-manager/v1...
...output omitted...

Deprecated and Removed Features in OpenShift

Red Hat OpenShift Container Platform (RHOCP) is a set of modular components and services that are built on top of a Kubernetes container infrastructure.

Some features that were available in previous OpenShift releases are deprecated or removed. A deprecated feature is not recommended for new deployments, because a future release will remove it. The following table contains a short list of the deprecated and removed features in OpenShift.

OpenShift 4.12 OpenShift 4.13 OpenShift 4.14 Feature
General AvailabilityGeneral AvailabilityDeprecatedOperator lifecycle and development deprecated
DeprecatedDeprecatedDeprecatedCoreDNS wildcard queries for the cluster.local domain
DeprecatedDeprecatedDeprecatedPersistent storage that uses FlexVolume
Not AvailableGeneral Availability Removed --include-local-oci-catalogs parameter for oc-mirror
General AvailabilityGeneral Availability Deprecated DeploymentConfig objects

Note

For more information about the deprecated and removed API versions in Kubernetes, consult the OpenShift Container Platform 4.14 release notes in the references section.

Deprecated API Alerts in OpenShift

OpenShift includes two alerts that are triggered when a workload uses a deprecated API version:

APIRemovedInNextReleaseInUse

This alert is triggered for APIs that OpenShift Container Platform will remove in the next release.

APIRemovedInNextEUSReleaseInUse

This alert is triggered for APIs that OpenShift Container Platform Extended Update Support (EUS) will remove in the next release.

The alert describes the situation with context to identify the affected workload.

Figure 9.8: Deprecated API alert

You can extract the alerts in JSON format from the Prometheus stateful set, and then filter the result to retrieve the deprecated API alerts.

[user@host ~]$ oc exec -it statefulset/prometheus-k8s -c prometheus \
  -n openshift-monitoring -- \
    curl -fsSL 'http://localhost:9090/api/v1/alerts' | jq . > alerts.json

[user@host ~]$ jq '[.data.alerts[] |
  select(.labels.alertname=="APIRemovedInNextReleaseInUse" or
         .labels.alertname=="APIRemovedInNextEUSReleaseInUse")]' < alerts.json
[
  {
    "labels": {
      "alertname": "APIRemovedInNextReleaseInUse",
...output omitted...
    },
    "state": "firing",
...output omitted...
  },
  {
    "labels": {
      "alertname": "APIRemovedInNextEUSReleaseInUse",
...output omitted...
    },
    "state": "firing",
...output omitted...
  }
]

Note

If the output of the jq command is an empty JSON array [], then the alerts were not reported.

Explicit Acknowledgment Before Cluster Updates

OpenShift Container Platform 4.14 uses Kubernetes 1.27, which removed deprecated v1beta1 APIs.

OpenShift Container Platform requires an administrator to provide a manual acknowledgment before the cluster can be upgraded from version 4.13 to 4.14. This requirement helps to prevent issues after upgrading to OpenShift Container Platform 4.14, where workloads, tools, or other components that run on or interact with the cluster still use removed APIs.

Administrators must evaluate their cluster for workloads that use removed APIs, and migrate the affected components to the appropriate new API version. After migration, the administrator can provide an acknowledgment.

[user@host ~]$ oc patch configmap admin-acks -n openshift-config --type=merge \
  --patch '{"data":{"ack-4.13-kube-1.27-api-removals-in-4.14":"true"}}'
configmap/admin-acks patched

References

For more information about the removed features in OpenShift, refer to the Deprecated and Removed Features section in the Red Hat OpenShift Container Platform 4.14 release notes at https://access.redhat.com/documentation/en-us/openshift_container_platform/4.14/html-single/release_notes/index#ocp-4-14-deprecated-removed-features

For more information about what version of the Kubernetes API is included with each OpenShift 4.x release, visit the following page in the customer portal: https://access.redhat.com/solutions/4870701

For more information about the Kubernetes API deprecations and removals, visit the following page in the customer portal: https://access.redhat.com/articles/6955985

For more information about the deprecated APIs in OpenShift Container Platform 4.14, visit the following page in the customer portal: https://access.redhat.com/articles/6955381

For more information about how to get fired alerts on OpenShift by using the command-line, visit the following page in the customer portal: https://access.redhat.com/solutions/4250221

What's New in Red Hat OpenShift 4.14

Preparing to Update to OpenShift Container Platform 4.14

Kubernetes Deprecation Policy

Kubernetes Deprecated API Migration Guide

Kubernetes Removals and Deprecations in 1.27

Kubernetes 1.27 release announcement

Revision: do280-4.14-08d11e1