Introduction to Red Hat OpenShift Service on AWS (ROSA)

Before creating a machine pool, you must choose an EC2 instance type for the nodes in the pool.

Amazon EC2 publishes the list of the instance types and their characteristics at https://aws.amazon.com/ec2/instance-types/. However, not all types are available for ROSA. Some instance types do not have enough memory, or a supported CPU architecture, to work as nodes in an OpenShift cluster. Some other types might not be available in your AWS Region.

To list the available EC2 instance types for your ROSA cluster, use the rosa list instance-types command:

$ rosa list instance-types
ID               CATEGORY               CPU_CORES  MEMORY
dl1.24xlarge     accelerated_computing  96         768.0 GiB
g4dn.12xlarge    accelerated_computing  48         192.0 GiB
p3.2xlarge       accelerated_computing  8          61.0 GiB
...output omitted...
c5.12xlarge      compute_optimized      48         96.0 GiB
c5.18xlarge      compute_optimized      72         144.0 GiB
...output omitted...
m5.12xlarge      general_purpose        48         192.0 GiB
m5.16xlarge      general_purpose        64         256.0 GiB
...output omitted...
r4.16xlarge      memory_optimized       64         488.0 GiB
r4.2xlarge       memory_optimized       8          61.0 GiB
...output omitted...
c5n.18xlarge     network_optimized      72         192.0 GiB
c5n.2xlarge      network_optimized      8          21.0 GiB
...output omitted...
m5d.8xlarge      storage_optimized      32         128.0 GiB
m5d.xlarge       storage_optimized      4          16.0 GiB

For specialized workloads, Amazon EC2 provides optimized instances that are organized into categories. For example, the accelerated computing category groups instance types that provide specific hardware, such as GPU.

You can get all the details of an instance type by using the aws ec2 describe-instance-types command:

$ aws ec2 describe-instance-types --instance-types p3.2xlarge
{
    "InstanceTypes": [
        {
            "InstanceType": "p3.2xlarge",
...output omitted...
            "MemoryInfo": {
                "SizeInMiB": 62464
            },
...output omitted...
            "GpuInfo": {
                "Gpus": [
                    {
                        "Name": "V100",
                        "Manufacturer": "NVIDIA",
                        "Count": 1,
                        "MemoryInfo": {
                            "SizeInMiB": 16384
                        }
                    }
                ],
                "TotalGpuMemoryInMiB": 16384
            },
...output omitted...
        }
    ]
}

Use the rosa create machinepool command to add a machine pool. The following command creates the with-gpu machine pool that deploys three compute nodes of the p3.2xlarge type:

$ rosa create machinepool --cluster mycluster --name with-gpu --replicas 3
  --instance-type p3.2xlarge

You can run the command interactively by using the --interactive option:

$ rosa create machinepool --cluster mycluster --interactive
? Machine pool name: with-gpu
? Enable autoscaling (optional): No
? Replicas: 3
I: Fetching instance types
? Instance type: p3.2xlarge
...output omitted...

Use the rosa list machinepools command to list the machine pools in your cluster. The following command lists the machine pools in the cluster:

$ rosa list machinepools --cluster mycluster
ID       AUTOSCALING  REPLICAS  INSTANCE TYPE LABELS TAINTS AVAILABILITY ZONES ...
Default  No           2         m5.xlarge                   us-east-1a ...
with-gpu No           3         p3.2xlarge                  us-east-1a ...

Use the rosa edit machinepools command to modify some machine pool parameters. You cannot change the instance type of an existing pool. The following command scales up the with-gpu machine pool to six nodes:

$ rosa edit machinepool --cluster mycluster --replicas 6 with-gpu

Use the rosa delete machinepools command to delete a machine pool. The associated compute nodes are also deleted. OpenShift relocates the workloads that are still using the pool. However, if these workloads use a node selector to target the pool, then OpenShift cannot restart them. The following command deletes the with-gpu machine pool:

$ rosa delete machinepool --cluster mycluster with-gpu

To add a label to the nodes of a machine pool, declare the label at the machine pool level. ROSA automatically applies that label to all the nodes in the pool.

Use the rosa edit machinepool command with the --labels label-name=label-value option to declare a node label. The following command adds the workload=IA label to the nodes of the with-gpu machine pool:

$ rosa edit machinepool --cluster mycluster --labels workload=IA with-gpu

You can also add that --labels option to the rosa create machinepool command, when you create the machine pool.

To assign your workload to the pool, use the nodeSelector parameter in the pod definition of your workload. The following example deploys a web server in a compute node with the workload=IA label:

apiVersion: v1
kind: Pod
metadata:
  name: frontweb
spec:
  containers:
    - name: web
      image: registry.redhat.io/rhel9/nginx-120:1-101
  nodeSelector:
    workload: IA

To add a taint to the nodes of a machine pool, use the rosa edit machinepool command with the --taints key=value:effect option. The following command adds the nvidia.com/gpu=true:NoSchedule taint to the nodes of the with-gpu machine pool:

$ rosa edit machinepool --cluster mycluster
  --taints nvidia.com/gpu=true:NoSchedule with-gpu

You can also add that --taints option to the rosa create machinepool command, when you create the machine pool.

At that point, no workload can use the machine pool. For OpenShift to schedule your workload to the pool, add the same toleration to your workload. The following example adds the toleration to the pod resource:

apiVersion: v1
kind: Pod
metadata:
  name: frontweb
spec:
  containers:
    - name: web
      image: registry.redhat.io/rhel9/nginx-120:1-101
  nodeSelector:
    workload: IA
  tolerations:
    - key: "nvidia.com/gpu"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

The effect parameter controls how OpenShift manages the workloads that are already running when you set the taint, and how to schedule new workloads: