Introduction to Red Hat OpenShift Service on AWS (ROSA)

Some workloads require more specialized storage from EBS than the general-purpose types. For example, I/O-intensive workloads, such as database systems, might benefit from the io2 EBS volume type. Batch workloads that compute large cold data sets might use the cheaper sc1 EBS volume type, which uses slower hard disk drives (HDDs).

$ aws ec2 create-volume --dry-run --volume-type io2 --size 20 --iops 1000
  --availability-zone us-east-1a

An error occurred (DryRunOperation) when calling the CreateVolume operation: Request would have succeeded, but DryRun flag is set.

$ aws ec2 create-volume --dry-run --volume-type io2 --size 20 --iops 1000
  --availability-zone sa-east-1a

An error occurred (UnknownVolumeType) when calling the CreateVolume operation: Unsupported volume type 'io2' for volume creation.

To use the EBS volume types with your workloads, you must expose them as storage classes in your cluster. The EBS CSI plug-in supports all the EBS volume types. The following file declares a storage class resource that developers can use to provision io2 EBS volumes:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: io2-csi
provisioner: ebs.csi.aws.com  
volumeBindingMode: WaitForFirstConsumer  
allowVolumeExpansion: true
parameters:
  csi.storage.k8s.io/fstype: ext4  
  type: io2  
  iops: "32000"  
  encrypted: "true"  

You must set the volumeBindingMode parameter to WaitForFirstConsumer. An EBS volume is bound to an availability zone, and is only accessible from nodes in that zone. If OpenShift were to provision the volume before creating the pod, then it would not know on which availability zone the pod will run.

When you create a ROSA cluster, you can configure it to span multiple availability zones. Red Hat recommends this configuration for production clusters. However, EBS volumes are bound to a single availability zone, and Amazon does not replicate them to the other availability zones. In case of the failure of an entire availability zone, OpenShift cannot move the pods that use EBS volumes from the failing zone.

The iops parameter is valid only for io1, io2 and gp3 EBS volume types. Instead of the iops parameter, you can use the iopsPerGB parameter, which specifies the maximum number of I/O operations per second per GiB.

For EBS volumes, storage size and IOPS scale together. In other words, larger volumes get better IOPS performances.

For example, Amazon sets the maximum performance for the io1 volume type to 50 IOPS per GiB. A 4 GiB io1 volume has a limit of 200 IOPS (50 x 4), whereas a 1 TiB volume has a limit of 51,200 IOPS (50 x 1024). In some case, to achieve your required IOPS performance, you might have to assign more space than you need.

Creating storage classes requires cluster administrator permissions. The dedicated administrator permissions are not enough.

The following example declares a PVC that uses the preceding storage class to request a 128 GiB io2 EBS volume:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dbdata
spec:
  storageClassName: io2-csi
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 128Gi

You can use the AWS Management Console or the aws command to inspect the resulting EBS volume that OpenShift creates from this claim. In the volume in EBS, the kubernetes.io/created-for/pvc/name tag is set to the name of the PVC. Add the --filter option to the aws command to filter the EBS volumes by this tag.

The following command displays the parameters of the EBS volume that OpenShift creates from the dbdata PVC:

$ aws ec2 describe-volumes
  --filters "Name=tag:kubernetes.io/created-for/pvc/name,Values=dbdata"
{
    "Volumes": [
        {
...output omitted...
            "Size": 128,
            "SnapshotId": "",
            "State": "in-use",
            "VolumeId": "vol-0b0102de7640bb98a",
            "Iops": 32000,
            "Tags": [
                {
                    "Key": "CSIVolumeName",
                    "Value": "pvc-fd7869ad-8d12-4683-8bd4-87217b306e59"
                },
                {
                    "Key": "kubernetes.io/created-for/pvc/name",
                    "Value": "dbdata"
                },
...output omitted...
            ],
            "VolumeType": "io2",
            "MultiAttachEnabled": false
        }
    ]
}