In a Distributed Compute Node (DCN) architecture, only compute nodes are deployed at remote sites; control-plane nodes are only deployed at the central site. There are two main roles used to deploy to a remote site in a DCN architecture. Use the DistributedCompute role to provide persistent compute power for up to three nodes, or the DistributedComputeHCI role to deploy three compute nodes combined with Ceph for storage. These roles include the GlanceAPIEdge service to serve image requests locally.

After the initial deployment, use the DistributedComputeScaleOut or DistributedComputeHCIScaleOut roles to deploy additional nodes at the site. These scale-out roles include the HAproxyEdge service to ensure that image requests are served by the local GlanceAPIEdge services.

Note that the round-trip time between the central and remote sites must not exceed 100 ms.

The overcloud is commonly deployed using a single Heat stack. A DCN architecture is only supported using multiple Heat stacks. The stacks can be managed, scaled, and updated separately, creating a smaller window for failure because there are fewer nodes in each stack. If a node fails, then only one stack requires intervention. Not all nodes within a stack need to be located in the same physical location. As long as the locations are routed using layer 3 the nodes can reside in different geographical or physical locations.

For example, one stack might create a control plane with three high-availability OpenStack controllers. A second stack might create a number of DistributedComputeHCI nodes with Ceph for persistent storage. A third stack might create a number of DistributedComputeHCI nodes in another location. There is no limit to the number of stacks an overcloud can have, and although they are all deployed and managed separately, they are all part of the same overcloud.

The openstack stack list command is used to list all available stacks within the overcloud. The following output shows the control plane for the controller nodes, and two separate locations for the DistributedComputeHCI nodes.

(undercloud) [stack@director ~]$ openstack stack list \
> -c 'Stack Name' -c 'Stack Status'
+----------------+-----------------+
| Stack Name     | Stack Status    |
+----------------+-----------------+
| control-plane  | CREATE_COMPLETE |
| europe         | CREATE_COMPLETE |
| apac           | CREATE_COMPLETE |
+----------------+-----------------+

The openstack baremetal node list command is used to list all the bare metal nodes in the overcloud deployment.

(undercloud) [stack@director ~]$ openstack baremetal node list \
> -c Name -c "Provisioning State"
+-----------------------+--------------------+
| Name                  | Provisioning State |
+-----------------------+--------------------+
| controller0           | active             |
| controller1           | active             |
| controller2           | active             |
| europe-hcicompute0    | active             |
| europe-hcicompute1    | active             |
| europe-hcicompute2    | active             |
| apac-hcicompute0      | active             |
| apac-hcicompute1      | active             |
| apac-hcicompute2      | active             |
+-----------------------+--------------------+

Use the openstack server list command to list the overcloud nodes and show their IP addresses on the provisioning network.

(undercloud) [stack@director ~]$ openstack server list \
> -c Name -c Status -c Networks
+---------------------------+--------------------+------------------------+
| Name                      | Provisioning State | Networks               |
+---------------------------+--------------------+------------------------+
| control-plane-controller0 | active             | ctlplane=172.25.249.59 |
| control-plane-controller1 | active             | ctlplane=172.25.249.57 |
| control-plane-controller2 | active             | ctlplane=172.25.249.53 |
| europe-hcicompute0        | active             | ctlplane=172.25.249.60 |
| europe-hcicompute1        | active             | ctlplane=172.25.249.61 |
| europe-hcicompute2        | active             | ctlplane=172.25.249.62 |
| apac-hcicompute0          | active             | ctlplane=172.25.249.70 |
| apac-hcicompute1          | active             | ctlplane=172.25.249.71 |
| apac-hcicompute2          | active             | ctlplane=172.25.249.72 |
+---------------------------+--------------------+------------------------+

A multisite topology can be created using multiple stacks or a single overcloud stack. In the case of a single overcloud stack, availability zones are used to separate the geographical locations. The following diagram shows a single stack overcloud using availability zones.

Figure 9.1: Single stack using availability zones

Redundancy is a major factor when designing a multisite topology. Because the compute nodes can be located in different geographical locations, you need to create redundant paths to ensure network communication between sites. Use redundant physical network connections as well. Network latency requirements must meet certain criteria; for example, the round-trip time between the compute nodes and the control plane should not exceed 100 ms. In some use cases, there is no layer 2 connectivity between the compute nodes and the control plane. In such cases the WAN must operate with high network latency.

Network dropouts can cause serious problems in the overcloud deployment. Dropouts mean that no control plane API or CLI operations can be executed. Operations such as snapshots, creating instances or volumes, deleting images, and many others are impossible if there are network dropouts.

During the planning phase it is essential to plan for potential failures. For example, SLA expectations must be taken into consideration when choosing an SDN solution. The failure of the control plane has the same impact in a single site topology as in a multisite topology, and is much less likely to occur than a network error at a remote site. A failure of the control plane is catastrophic because no API or CLI operations can be executed. It is also a rare event.

Failure of a single compute node should not have any effect on any one site. It is important to plan for compute node failure. For example, if one compute node fails, the remaining compute nodes should be able to host all instances at the remote site.

A region is a division within the overcloud deployment. Regions can have subregions, creating a tree-like hierarchy. Regions are a multisite architecture with a single Identity service supporting all sites, and all sites having a unique region name.

Regions can only be created by the admin user. Users with administrative privileges within a project cannot create regions.

The openstack region list command lists all available regions within an overcloud deployment.

[user@demo ~]$ openstack region list
+-----------+---------------+-------------+
| Region    | Parent Region | Description |
+-----------+---------------+-------------+
| regionOne | None          |             |
+-----------+---------------+-------------+

To create a new region, use the openstack region create command. Use the --parent-region option to create a subregion.

[user@demo ~(admin)]$ openstack region create demo \
> --parent-region regionOne
+---------------+-----------+
| Field         | Value     |
+---------------+-----------+
| description   |           |
| enabled       | True      |
| parent_region | regionOne |
| region        | demo      |
+---------------+-----------+

Use the openstack region delete command to delete a region. You must specify the region ID. Use the openstack region set command to set properties for the region, for example, the parent ID or a description. Use the openstack region show command to display the region details.

Multiple regions can be supported using a single Identity service instance. The openstack endpoint list command lists all the available endpoints. This table tells OpenStack services where to find the correct service. For example, if a user launches a new instance, the Compute service needs to request the image from the Image service by querying the Identity service for endpoints.

(overcloud) [stack@director ~]$ openstack endpoint list \
> -c Region -c "Service Name" -c "Service Type"
+-----------+--------------+----------------+
| Region    | Service Name | Service Type   |
+-----------+--------------+----------------+
| regionOne | octavia      | load-balancer  |
| regionOne | keystone     | identity       |
| regionOne | glance       | image          |
| regionOne | placement    | placement      |
...output omitted...

The default regionOne region is used when regions have not been configured. In a multiregion deployment, the output would look like the following:

+-----------+--------------+----------------+
| Region    | Service Name | Service Type   |
+-----------+--------------+----------------+
| europe    | octavia      | load-balancer  |
| apac      | octavia      | load-balancer  |
| europe    | keystone     | identity       |
| apac      | keystone     | identity       |
| europe    | glance       | image          |
| apac      | glance       | image          |
| europe    | placement    | placement      |
| apac      | placement    | placement      |
...output omitted...

Regions are not necessarily locations, but could be service levels, such as standard storage or SSD storage. The Compute service could have a larger quota for standard storage and a smaller quota for SSD storage. They can also be managed using regions.

+-----------+--------------+----------------+
| Region    | Service Name | Service Type   |
+-----------+--------------+----------------+
| standard  | nova         | compute        |
| ssd       | nova         | compute        |
...output omitted...

Nova cells is a strategy for scaling the Compute service, and is enabled by default. The Compute service database is now separated into a database with global data, a special cell0 database, and one or more cell-specific databases. Each cell is composed of a cell database instance, a message queue instance, and any number of compute nodes. The cell-specific database contains information for each instance launched on the compute nodes belonging to that cell. The database and message queue in the cells that contain hosts only support those hosts, reducing the amount of traffic and load they are subject to.

Deploying a new cell is performed by generating cell information, and then running the overcloud deployment again but also specifying the cell configuration. Cells can be organized by hardware; for example, a cell for bare metal nodes and another cell for SSD storage nodes. This organization makes capacity planning easier, and deployments can be scaled out by adding new cells.

Host aggregates are used to create further partitioning distinctions for nodes within an entire OpenStack deployment, or a region within an OpenStack deployment, or an availability zone. Creating and configuring aggregates requires administrator privileges. Host aggregates are used to assign metadata key-value pairs to groups of nodes. Nodes can be members of multiple host aggregates. The same key-value metadata can be assigned to many aggregates, and aggregates can be configured with as many key-value pairs as needed to define the host selection scenario. When an instance is launched with custom flavor metadata, the metadata is passed to the compute scheduler to perform advanced scheduling selection from available host aggregates. For example, this can be used to define logical groups to control instance deployment or migration by specifying required additional hardware or performance characteristics.

Figure 9.2: Host aggregates

OpenStack users cannot see configured host aggregates, but instead select appropriate flavors to specify the host characteristics to be filtered by the compute scheduler during an instance launch. Administrator privileges are required to configure host aggregates and flavors with matching metadata properties. Set the key-value property on any host aggregate to include the host characteristic property, and then set the same property again on any flavor to be used to request launching to hosts with the required host characteristics.

To create a host aggregate, use the openstack aggregate create command. Hosts can be added to the aggregate using the openstack aggregate add host command. The metadata is added to the aggregate using the openstack aggregate set command.

[user@demo ~]$ openstack aggregate create \
> ssd-aggregate
+-------------------+----------------------------+
| Field             | Value                      |
+-------------------+----------------------------+
| availability_zone | None                       |
| created_at        | 2020-08-05T07:30:57.628686 |
| deleted           | False                      |
| deleted_at        | None                       |
| hosts             | None                       |
| id                | 1                          |
| name              | ssd-aggregate              |
| properties        | None                       |
| updated_at        | None                       |
+-------------------+----------------------------+
[user@demo ~]$ openstack aggregate add host \
> ssd-aggregate compute0.overcloud.example.com
+-------------------+--------------------------------+
| Field             | Value                          |
+-------------------+--------------------------------+
| availability_zone | None                           |
| created_at        | 2020-08-05T07:30:57.000000     |
| deleted           | False                          |
| deleted_at        | None                           |
| hosts             | compute0.overcloud.example.com |
| id                | 1                              |
| name              | ssd-aggregate                  |
| properties        |                                |
| updated_at        | None                           |
+-------------------+--------------------------------+
[user@demo ~]$ openstack aggregate set \
> --property ssd=true ssd-aggregate
[user@demo ~]$ openstack aggregate show ssd-aggregate
+-------------------+--------------------------------+
| Field             | Value                          |
+-------------------+--------------------------------+
| availability_zone | None                           |
| created_at        | 2020-08-05T07:30:57.000000     |
| deleted           | False                          |
| deleted_at        | None                           |
| hosts             | compute0.overcloud.example.com |
| id                | 1                              |
| name              | ssd-aggregate                  |
| properties        | ssd='true'                     |
| updated_at        | None                           |
+-------------------+--------------------------------+

Use the openstack flavor create command to create a new flavor using the aggregate. Users can then use the flavor to ensure that an instance is created on a compute node with SSD storage.

Availability zones, which are commonly used to specify geographical host groupings, are also a type of host aggregate. Availability zones have a special flag set, which allows zones to be seen and selected by normal users to control an instance launch. Administrators can set that flag on any host aggregate to make it visible as an availability zone. One other important difference is that hosts can be members of many aggregates but of only one availability zone.

Each separate site must be configured as an availability zone. Instances deployed to a specific zone will run on the compute nodes in that zone. If the zone contains HCI compute nodes, then the storage will also be contained within the zone. If the compute nodes do not have integrated storage, the instances will use the central storage solution.

To list the controller and compute nodes available in the whole cloud use the openstack compute servicel list command.

(overcloud) [stack@director ~]$ openstack compute service list \
> -c Binary -c Host -c Zone
+----------------+----------------------------------------+----------+
| Binary         | Host                                   | Zone     |
+----------------+----------------------------------------+----------+
| nova-conductor | controller0.overcloud.example.com      | internal |
| nova-conductor | controller1.overcloud.example.com      | internal |
| nova-conductor | controller2.overcloud.example.com      | internal |
| nova-scheduler | controller0.overcloud.example.com      | internal |
| nova-scheduler | controller1.overcloud.example.com      | internal |
| nova-scheduler | controller2.overcloud.example.com      | internal |
| nova-compute   | eu-computehci0.overcloud.example.com   | europe   |
| nova-compute   | eu-computehci0.overcloud.example.com   | europe   |
| nova-compute   | eu-computehci0.overcloud.example.com   | europe   |
| nova-compute   | apac-computehci0.overcloud.example.com | apac     |
| nova-compute   | apac-computehci0.overcloud.example.com | apac     |
| nova-compute   | apac-computehci0.overcloud.example.com | apac     |
+----------------+----------------------------------------+----------+

Services are grouped by aggregate and availability zone per geographical location.

(overcloud) [stack@director ~]$ nova aggregate-list
+----+--------+-------------------+
| Id | Name   | Availability Zone |
+----+--------+-------------------+
| 2  | europe | europe            |
| 8  | apac   | apac              |
+----+--------+-------------------+

(overcloud) [stack@director ~]$ nova availability-zone-list
+----------------------------------------------------+-------------+
| Name                                               | Status      |
+----------------------------------------------------+-------------+
| internal                                           | available   |
| |- control-plane-controller0.overcloud.example.com |             |
| | |- nova-conductor                                | enabled ... |
| | |- nova-scheduler                                | enabled ... |
| |- control-plane-controller1.overcloud.example.com |             |
| | |- nova-conductor                                | enabled ... |
| | |- nova-scheduler                                | enabled ... |
| |- control-plane-controller2.overcloud.example.com |             |
| | |- nova-conductor                                | enabled ... |
| | |- nova-scheduler                                | enabled ... |
| europe                                             | available   |
| |- eu-compute0.overcloud.example.com               |             |
| | |- nova-compute                                  | enabled ... |
| |- eu-compute1.overcloud.example.com               |             |
| | |- nova-compute                                  | enabled ... |
| |- eu-compute2.overcloud.example.com               |             |
| | |- nova-compute                                  | enabled ... |
| apac                                               | available   |
| |- apac-compute0.overcloud.example.com             |             |
| | |- nova-compute                                  | enabled ... |
| |- apac-compute1.overcloud.example.com             |             |
| | |- nova-compute                                  | enabled ... |
| |- apac-compute2.overcloud.example.com             |             |
| | |- nova-compute                                  | enabled ... |
+----------------------------------------------------+-------------+