AD482 - ch02s06

Bookmark this page

Guided Exercise: Creating Topics

In this exercise, you will create, alter, and describe a Kafka topic, observe partitions, and verify message ordering of the topics and partitions.

Outcomes

You should be able to:

Create and alter a Kafka topic.
Describe a Kafka topic to see its partition, replication factor, and other configuration details.
Verify the partition distribution, and identify the leader/follower brokers.
Understand the record ordering in topics and partitions.

To perform this exercise, ensure you have the following:

Access to a configured and running Red Hat OpenShift Container Platform cluster.
Access to an installed and running Kafka instance in the OpenShift cluster.
Installed the oc command-line binary.
A configured Python virtual environment, including the grading scripts for this course.
The JDK installed.

From your workspace directory, activate the Python virtual environment.

[user@host AD482]$ source .venv/bin/activate

Important

On Windows, use the Activate.ps1 script to activate your Python virtual environment.

PS C:\Users\user\AD482> ./.venv/Scripts/Activate.ps1

Use the lab command to prepare your system for this exercise.

(.venv) [user@host AD482]$ lab start kafka-topics
...output omitted...
 · Verifying your Kafka settings ...output omitted...
    - Bootstrap Server: my-cluster-kafka-cluster.apps.cluster.example.com
    - Bootstrap Port: 443
    - Cluster Namespace: your-username-kafka-cluster
...output omitted...

Copy the Bootstrap Server, Bootstrap Port, and Cluster Namespace values. You will use these values to connect to the Kafka cluster.

The lab command copies the exercise files, from the AD482-apps/kafka-topics/apps directory, at your local Git repository, into the kafka-topics directory, at the root of your workspace.

Procedure 2.1. Instructions

Create a topic named call-detail-records.

Verify that your project is set to RHT_OCP4_DEV_USER-kafka-cluster.

(.venv) [user@host AD482]$ oc project
Using project "RHT_OCP4_DEV_USER-kafka-cluster" on server ...output omitted...

Navigate to the kafka-topics/resources directory.
Create a call-detail-records-topic.yaml file that creates a Kafka topic with the following parameters:
- Topic name: call-detail-records
- Partition number: 12
- Replication factor: 2
- Cluster name: my-cluster
- Record retention period: 604800000 (default value of 7 days)
- Segment file size: 1073741824 (default value of 1 GiB)
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: call-detail-records
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 12
  replicas: 2
  config:
    retention.ms: 604800000
    segment.bytes: 1073741824
```

Create the topic.

(.venv) [user@host resources]$ oc create -f call-detail-records-topic.yaml
kafkatopic.kafka.strimzi.io/call-detail-records created

Verify the topic is in the Ready state:

(.venv) [user@host resources]$ oc get kafkatopics call-detail-records
NAME                  CLUSTER      PARTITIONS   REPLICATION FACTOR   READY
call-detail-records   my-cluster   12           2                    True

Produce and receive records by using the call-detail-records topic.
Use the kafka-console-consumer.sh and kafka-console-producer.sh scripts. The producer and consumer Kafka scripts are available on every Kafka broker node.
Use the my-cluster-kafka-brokers Kubernetes service on port 9092 to discover the Kafka broker nodes.
1. Run the following command to receive records from the call-detail-records topic.
```
(.venv) [user@host resources]$ oc exec -it my-cluster-kafka-0 \
 -- bin/kafka-console-consumer.sh \
 --bootstrap-server my-cluster-kafka-brokers:9092 \
 --topic call-detail-records
```
  Because the topic is empty, the output should be empty. Leave the terminal window open and the console consumer working.
2. Open a new terminal window and start a producer for the call-detail-records topic. Enter a text after the input indicator >. Then, press Enter to send each line as a separate record.
```
(.venv) [user@host]$ oc exec -it my-cluster-kafka-0 \
 -- bin/kafka-console-producer.sh \
 --bootstrap-server my-cluster-kafka-brokers:9092 \
 --topic call-detail-records
>Call Detail Record 1
>Call Detail Record 2
>Call Detail Record 3
```
  Observe that the consumer receives the records:
```
Call Detail Record 1
Call Detail Record 2
Call Detail Record 3
```
  Terminate the consumer and producer applications. Then, close the extra terminal window.

Alter the call-detail-records topic.

Copy the kafka-topics/resources/call-detail-records-topic.yaml file that you created in the preceding steps and rename it call-detail-records-topic_altered.yaml.
In the call-detail-records-topic_altered.yaml file, change the partition number from 12 to 24 and retention.ms from 604800000 to 2629800000 (1 month).
The final file should have the following content:
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: call-detail-records
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 24
  replicas: 2
  config:
    retention.ms: 2629800000
    segment.bytes: 1073741824
```

Alter the call-detail-records topic by using the oc apply command.

(.venv) [user@host resources]$ oc apply \
 -f call-detail-records-topic_altered.yaml
kafkatopic.kafka.strimzi.io/call-detail-records configured

Use the oc describe command to verify the state of the call-detail-records topic.

(.venv) [user@host resources]$ oc describe kafkatopic call-detail-records
...output omitted...
Spec:
  Config:
    retention.ms:   2629800000
    segment.bytes:  1073741824
  Partitions:       24
  Replicas:         2
Status:
  Conditions:
    Last Transition Time:  2021-08-22.10:36:59.892248Z
    Status:                True
    Type:                  Ready
...output omitted...

Verify that the topic operator modified the topic.

Describe the call-detail-records topic by using the Kafka kafka-topics.sh shell script. Run the following command to see the change details of the topic.

(.venv) [user@host resources]$ oc exec -it my-cluster-kafka-0 \
 -- bin/kafka-topics.sh \
 --bootstrap-server my-cluster-kafka-brokers:9092 \
 --describe --topic call-detail-records
Topic: call-detail-records      PartitionCount: 24      ReplicationFactor: 2    Configs: segment.bytes=1073741824,retention.ms=2629800000,message.format.version=2.8-IV2
        Topic: call-detail-records      Partition: 0    Leader: 1       Replicas: 1,0   Isr: 1,0
        ...output omitted...
        Topic: call-detail-records      Partition: 23   Leader: 0       Replicas: 0,1   Isr: 0,1

Verify that there is no record ordering guarantee in a topic with multiple partitions. Then, verify that the record order is preserved in a one-partition topic.
1. Restart the consumer to receive records from call-detail-records.
  Add the --from-beginning flag to the consumer command to receive already sent records.
```
(.venv) [user@host resources]$ oc exec -it my-cluster-kafka-0 \
 -- bin/kafka-console-consumer.sh \
 --bootstrap-server my-cluster-kafka-brokers:9092 \
 --topic call-detail-records --from-beginning
Call Detail Record 2
Call Detail Record 1
Call Detail Record 3
```
  Note that your order output might be different from the previous output.
  Notice that the consumer does not maintain the producer record order. Re-execute the command multiple times to verify there is no guaranteed order at the topic level.
  Stop the receiver process in the terminal.
2. Create a topic named one-partition-topic. Use the kafka-topics/resources/one-partition-topic.yaml file.
```
(.venv) [user@host resources]$ oc create -f one-partition-topic.yaml
kafkatopic.kafka.strimzi.io/one-partition-topic created
```
3. Send records to the one-partition-topic topic.
```
(.venv) [user@host resources]$ oc exec -it my-cluster-kafka-0 \
 -- bin/kafka-console-producer.sh \
 --bootstrap-server my-cluster-kafka-brokers:9092 \
 --topic one-partition-topic
>Message 1
>Message 2
>Message 3
```
  Terminate the producer.
4. Receive the records from the one-partition-topic topic by using the --from-beginning flag to verify the record ordering.
```
(.venv) [user@host resources]$ oc exec -it my-cluster-kafka-0 \
 -- bin/kafka-console-consumer.sh \
 --bootstrap-server my-cluster-kafka-brokers:9092 \
 --topic one-partition-topic --from-beginning
Message 1
Message 2
Message 3
```
  Run this command again and check that the message order is kept.

Finish

Go back to your workspace. Run the lab command to complete this exercise. This is important to ensure that resources from previous exercises do not impact upcoming exercises.

(.venv) [user@host AD482]$ lab finish kafka-topics

This concludes the guided exercise.

Discuss Developing Event-driven Applications with Apache Kafka and Red Hat AMQ Streams

Go to community

Welcome to the Developing Event-Driven Applications with Apache Kafka and Red Hat AMQ Streams group!

cschunke

26 wrz 2023

We are excited to launch a space dedicated to the Red Hat Training course Developing Event-Driven Applications with Apache Kafka and Red Hat AMQ Streams! To gain the most value from this group - click the "Join Group" button in the upper right hand corner of the group home page.We encourage group members to collaborate in this group to discuss topics, ask questions, share best practices and tips, provide course feedback, and share their accomplishments as it relates to AD482.Read more about Developing Event-Driven Applications with Apache Kafka and Red Hat AMQ Streams here.

899

Revision: ad482-1.8-cc2ae1c

Click CREATE to build all of the virtual machines needed for the classroom lab environment. This may take several minutes to complete. Once created the environment can then be stopped and restarted to pause your experience.

If you DELETE your lab, you will remove all of the virtual machines in your classroom and lose all of your progress.