CL260 - ch08s03

Bookmark this page

Configuring a Multisite Object Storage Deployment

Objectives

After completing this section, you should be able to configure the RADOS Gateway with multisite support to allow objects to be stored in two or more geographically diverse Ceph storage clusters.

RADOS Gateway Multisite Deployment

The Ceph RADOS Gateway supports multisite deployments within a global namespace, which allows the RADOS Gateway to automatically replicate object data between multiple Red Hat Ceph Storage clusters. A common supported use case is active/active replication between geographically separate clusters for disaster recovery purposes.

The latest multisite configuration simplifies failover and failback procedures, supports active/active replication configuration between clusters, and incorporates new features such as a simpler configuration and support for namespaces.

Figure 8.3: RADOS Gateway multisite diagram

Multisite Components

The multisite components and definitions are listed below.

zone: A zone is backed by its own Red Hat Ceph Storage cluster. Each zone has one or more RADOS Gateways associated with it.
zone group: A zone group is a set of one or more zones. Data stored in one zone in the zone group is replicated to all other zones in the zone group. One zone in every zone group is designated as the master zone for that group. The other zones in the zone group are secondary zones.
realm: A realm represents the global namespace for all objects and buckets in the multisite replication space. A realm contains one or more zone groups, each of which contains one or more zones. One zone group in the realm is designated as the master zone group, and the others are secondary zone groups. All RADOS Gateways in the environment pull their configuration from the RADOS Gateway in the master zone group and master zone.

Because the master zone in the master zone group handles all metadata updates, operations such as creating users must occur in the master zone.

Important

You can execute metadata operations in a secondary zone, but it is not recommended because the metadata will not be synchronized over the realm. This behavior can lead to metadata fragmentation and configuration inconsistency between zones.

This architecture can be structured in several ways:

A single zone configuration has one zone group and one zone in the realm. One or more (possibly load-balanced) RADOS Gateways are backed by one Red Hat Ceph Storage cluster.
A multizone configuration has one zone group but multiple zones. Each zone is backed by one or more RADOS Gateways and an independent Red Hat Ceph Storage cluster. Data stored in one zone is replicated to all zones in the zone group. This can be used for disaster recovery if one zone suffers a catastrophic failure.
A multizone group configuration has multiple zone groups, each with one or more zones. You can use a multizone group to manage the geographic location of RADOS Gateways within one or more zones in a region.
A multiregion configuration allows the same hardware to be used to support multiple object namespaces that are common across zone groups and zones.

A minimal RADOS Gateway multisite deployment requires two Red Hat Ceph Storage clusters, and a RADOS Gateway for each cluster. They exist in the same realm and are assigned to the same master zone group. One RADOS Gateway is associated with the master zone in that zone group. The other is associated with a separate secondary zone in that zone group. This is a basic multizone configuration.

Change Coordination with Periods and Epochs

Each realm has an associated period, and each period has an associated epoch. A period is used to track the configuration state of the realm, zone groups, and zones at a particular time. Epochs are version numbers to track configuration changes for a particular realm period. Each period has a unique ID, contains realm configuration, and knows the previous period ID.

When you update the configuration of the master zone, the RADOS Gateway service updates the period. This new period becomes the current period of the realm, and the epoch of this period increases its value by one. For other configuration changes, only the epoch is incremented; the period does not change.

Multisite Synchronization Process

The RADOS Gateway synchronizes metadata and data operations between all master and secondary zone group sets. Metadata operations relate to buckets: creating, deleting, enabling and disabling versioning, and managing users. The meta master, which is in the master zone of the master zone group, manages metadata updates. Data operations are those relevant to objects.

When a multisite configuration is active, the RADOS Gateway performs an initial full synchronization between the master and secondary zones. Subsequent updates are incremental. When the RADOS Gateway writes data to any zone within a zone group, it synchronizes this data between all of the zones in other zone groups. When the RADOS Gateway synchronizes data, all active gateways update the data log and notify the other gateways. When the RADOS Gateway synchronizes metadata because of a bucket or user operation, the master updates the metadata log and notifies the other RADOS Gateways.

Configuring Multisite RGW Deployments

You can deploy, configure, and remove multisite Ceph RADOS Gateway instances by using the Ceph orchestrator command-line interface or by using a service specification file.

Multisite Configuration Example

The following example configures a realm with one zone group containing two zones, one acting as the master zone and the other as a secondary zone. Each zone has one RADOS Gateway instance associated with it.

Configure the Master Zone

These example steps configure the RADOS Gateway instance in the master zone.

Create a realm.

[ceph: root@node01 /]# radosgw-admin realm create --default --rgw-realm=gold

Create a master zone group.

[ceph: root@node01 /]# radosgw-admin zonegroup create --rgw-zonegroup=us \
  --master --default --endpoints=http://node01:80

Create a master zone.

[ceph: root@node01 /]# radosgw-admin zone create --rgw-zone=datacenter01 \
  --master --rgw-zonegroup=us --endpoints=http://node01:80 \
  --access-key=12345 --secret=67890 --default

Create a system user.

[ceph: root@node01 /]# radosgw-admin user create --uid=sysadm \
  --display-name="SysAdmin" --access-key=12345 --secret=67890 --system

Commit the changes.

[ceph: root@node01 /]# radosgw-admin period update --commit

Create the RADOS Gateway service for the master zone.

[ceph: root@node /]# ceph orch apply rgw gold-service --realm=gold \
  --zone=datacenter01 --placement="1 node01"

Update the zone name in the configuration database.

[ceph: root@node01 /]# ceph config set client.rgw rgw_zone datacenter01

Configure the Secondary Zone

These example steps configure the RADOS Gateway instance on the secondary zone.

Pull the realm configuration.

[ceph: root@node02 /]# radosgw-admin realm pull --rgw-realm=gold \
  --url=http://node01:80 --access-key=12345 --secret=67890 --default

Pull the period.

[ceph: root@node02 /]# radosgw-admin period pull --url=http://node01:8000 \
  --access-key=12345 --secret=67890

Create a secondary zone.

[ceph: root@node /]# radosgw-admin zone create --rgw-zone=datacenter02 \
  --rgw-zonegroup=us --endpoints=http://node02:80 \
  --access-key=12345 --secret=67890 --default

Use the --read-only option to set the zone as read-only when adding it to the zone group.

Commit the changes.

[ceph: root@node02 /]# radosgw-admin period update --commit

Create the RADOS Gateway service for the secondary zone.

[ceph: root@node02 /]# ceph orch apply rgw gold-service --realm=gold \
  --zone=datacenter02 --placement="1 node02"

Update the zone name in the configuration database.
```
[ceph: root@node02 /]# ceph config set client.rgw rgw_zone datacenter02
```
Use the radosgw-admin sync status command to verify the synchronization status.

Managing Zone Failover

In a multisite deployment, when the master zone is unavailable, the secondary zones can continue to serve read and write requests. However, because the master zone is not available, you cannot create new buckets and users. If the master zone does not recover immediately, promote one of the secondary zones as a replacement for the master zone.

To promote a secondary zone, modify the zone and zone group and commit the period update.

Point the master zone to the secondary zone (datacenter02).

[ceph: root@node02 /]# radosgw-admin zone modify --master --rgw-zone=datacenter02

Update the master zone group after changing the role of the zone.

[ceph: root@node02 /]# radosgw-admin zonegroup modify --rgw-zonegroup=us \
  --endpoints=http://node02:80

Commit the changes.

[ceph: root@node02 /]# radosgw-admin period update --commit

Metadata Search Capabilities

Very large object stores are becoming common. Users still require smart access to objects so that they can extract additional information from the object metadata. The metadata associated with the objects provides insightful information about the objects. RADOS Gateway supports Elasticsearch to query metadata in an object store, and to index objects based on user-defined custom metadata for objects.

RADOS Gateway supports Elasticsearch as a component of the multimode architecture. Elasticsearch can manage the metadata for all objects in a zone group. A zone group can contain zones that store objects, while other zones store the metadata for those objects.

The following command defines the metadata-zone zone as a metadata zone managed by Elasticsearch:

[ceph: root@node /]# radosgw-admin zone modify --rgw-zone=metadata-zone \
  --tier-type=elasticsearch \
  --tier-config=endpoint=http://node03:9200,num_shards=10,num_replicas=1

The --tier-type option sets the zone type to elasticsearch.
The --tier-config option defines the configuration for the Elasticsearch zone.
The endpoint parameter defines the endpoint to access the Elasticsearch server.
The num_shards parameter defines the number of shards used by Elasticsearch.
The num_replicas parameter specifies the number of replicas used by Elasticsearch.

RADOS Gateway Multisite Monitoring

You can monitor RADOS Gateway performance and usage statistics in the Ceph Storage Dashboard. To integrate the RADOS Gateway with the Ceph Storage Dashboard, you must add the login credentials of an RGW user with the system flag in a JSON format file.

Use the ceph dashboard set-rgw-api-access-key and ceph dashboard set-rgw-api-secret-key commands to provide the access key and secret key, when adding an RGW system user to the dashboard.

[ceph: root@node /]# cat access_key
{'myzone.node.tdncax': 'ACCESS_KEY'}
[ceph: root@node /]# cat secret_key
{'myzone.node.tdncax': 'SECRET_KEY'}
[ceph: root@node /]# ceph dashboard set-rgw-api-access-key -i access_key
Option RGW_API_ACCESS_KEY updated
[ceph: root@node /]# ceph dashboard set-rgw-api-secret-key -i secret_key
Option RGW_API_SECRET_KEY updated

To view details of the Ceph RADOS Gateway service, log in to the Dashboard and click Object Gateway. You are presented with a choice of Daemons, Users, or Buckets.

In the Daemons submenu, the dashboard shows a list of RGW daemons.

Figure 8.4: RADOS Gateway Daemon list

To view details and performance statistics, click a daemon name.

Figure 8.5: RADOS Gateway Daemon Performance

To view the overall performance of the service, click Overall Performance.

Figure 8.6: RADOS Gateway Overall Performance

References

RGW Metadata Search

For more information, refer to the Multi-site Configuration and Administration chapter in the Object Gateway Guide at https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/5/html-single/object_gateway_guide/index#multisite-configuration-and-administration

Discuss Cloud Storage with Red Hat Ceph Storage

Go to community

Red Hat Ceph Storage for OpenStack (CL260)

Haley_Ruccio

3 sie 2023

Build, expand and maintain cloud-scale, clustered storage for your applications with Red Hat Ceph StorageCloud Storage with Red Hat Ceph Storage (CL260) is designed for storage administrators and cloud operators who deploy Red Hat Ceph Storage in a production data center environment or as a component of a Red Hat OpenStack Platform or OpenShift Container Platform infrastructure. Learn how to deploy, manage, and scale a Ceph storage cluster to provide hybrid storage resources, including Amazon S3 and OpenStack Swift-compatible object storage, Ceph-native and iSCSI-based block storage, and shared file storage. This course is based on Red Hat Ceph Storage 5.0.Course summaryDeploy and manage a Red Hat Ceph Storage cluster on commodity servers.Perform common management operations using the web-based management interface.Create, expand, and control access to storage pools provided by the Ceph cluster.Access Red Hat Ceph Storage from clients using object, block, and file-based methods.Analyze and tune Red Hat Ceph Storage performance.Integrate Red Hat OpenStack Platform image, object, block, and file storage with a Red Hat Ceph Storage cluster.Integrate OpenShift Container Platform with a Red Hat Ceph Storage cluster.Target AudienceThis course is intended for storage administrators and cloud operators who want to learn how to deploy and manage Red Hat Ceph Storage on servers in an enterprise data center or within a Red Hat OpenStack Platform or OpenShift Container Platform environment.Developers writing applications that use cloud-based storage will learn the distinctions of various storage types and client access methods.Recommended trainingTake our free assessment to gauge whether this offering is the best fit for your skills.Red Hat Certified System Administrator (RHCSA) certification, or equivalent experience.For candidates that have not earned an RHCSA or equivalent, confirmation of the correct skill set knowledge can be obtained by taking the online skills assessment.Some experience with storage administration is recommended but not required.Technology considerationsThis course does not have any special technical requirements.This course is not intended for BYOD.Internet access is recommended.

Welcome to the Red Hat Ceph Storage for OpenStack (CL260) group in the Red Hat Learning Community!

cschunke

31 lip 2023

We are excited to launch a space dedicated to the Red Hat Training course CL260! To gain the most value from this group - click the "Join Group" button in the upper right hand corner of the group home page.We encourage group members to collaborate in this group to discuss topics, ask questions, share best practices and tips, provide course feedback, and share their accomplishments as it relates to CL260.Read more about Red Hat Ceph Storage for OpenStack here.

381

Revision: cl260-5.0-29d2128