Developing Event-driven Applications with Apache Kafka and Red Hat AMQ Streams

After completing this section, you should be able to create a Kafka Connect connector.

Connectors integrate Kafka Connect with external technologies, such as databases, and third-party services. The main responsibility of connectors is to stream data from an external system to Kafka, or from Kafka to an external system.

Kafka Connect has two types of connectors.

Connectors are components that implement the Kafka Connect API. When you use connectors, you use configuration files to describe the data you want to copy, and the Kafka Connect components do all the work for you.

The connector implementation is responsible for splitting the copying job into tasks. Those tasks are later distributed to connect workers that perform the copy.

Converters are another important component of Kafka Connect. Their main responsibility is to handle the serialization and deserialization of data. For example, a converter can take input from JDBC and convert it to JSON.

For maximum reusability, converters are a separate component from connectors. This means that you can use the same converter with a source connector, and with a sink connector.

Apache Kafka provides the following converters:

To use connectors, you must create a connector instance in your Kafka Connect cluster. After creating the instance, you can configure, monitor and manage the running connector instance.

The connector's configuration depends on the external system that they integrate with. They might have a different purpose, but there are a few common options that you must configure in all your connectors:

Sink connectors must also set one of the following options to control their input:

The preceding configuration options are in the format of Java properties that Kafka Connect accepts natively in its API.

AMQ Streams on the Red Hat OpenShift Container Platform (RHOCP) provides two options to create and manage connectors, the KafkaConnector custom resource, and the Kafka Connect REST API.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: my-source-connector 
  labels:
    strimzi.io/cluster: my-connect-cluster 
spec:
  class: org.apache.kafka.connect.file.FileStreamSourceConnector 
  tasksMax: 2 
  config: 
    file: "/opt/kafka/LICENSE"
    topic: my-topic

{
  "name": "my-source-connector", 
  "config": { 
    "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector", 
    "tasks.max": "2", 
    "file": "/opt/kafka/LICENSE",
    "topic": "my-topic"
  }
}

A connector plug-in is a Java package containing an implementation that knows how to communicate with an external system. The AMQ Streams container images for Kafka Connect include two built-in file connectors.

To extend the connection capabilities of Kafka Connect, you must create a custom Kafka Connect image, and add the connector classes you need. There are different procedures to add your own connector classes to Kafka Connect images:

For the details of each procedure, and more information about each strategy consult the Extending Kafka Connect with connector plug-ins document in the references section.