Developing Event-driven Applications with Apache Kafka and Red Hat AMQ Streams

The topic where the record is sent.

The partition where the record is sent.

If not provided, then the key field is used to choose the partition.

The identifier of the record. The key is used to choose the partition where the record is sent. Records with the same key are sent to the same partition.

If both partition and key fields are not provided, then the partition is assigned in a round-robin fashion.

The value or payload. In event-driven architectures, this is usually the event.

Represents the creation time of the record. If not provided, then the producer uses the current time.

The timestamp used by Kafka depends on the topic configuration:

A list of key/value objects that might contain additional information relevant for your application.

By using this method, you do not wait for the broker to send back a response. The message is sent to the broker and your application continues processing the code after sending the message. In case of an error, your application is not notified.

ProducerRecord<String, String> record = new ProducerRecord<>(
        "topic",
        "value"
    );

try {
    producer.send(record);
} catch (Exception e) {
    e.printStackTrace();
}

In the previous code snippet, you create a record without a key. The send method returns a Future object, which is ignored to avoid waiting for a response, ensuring a "Fire-and-Forget" pattern. Although you can not capture broker errors with this approach, the try/catch block can capture other exceptions (for example, serialization or time out issues).

Not waiting for a response from the broker might increase the performance of your producer application. This is a useful approach if each individual message is not critical. For example, in an application that is sending messages to track web page traffic, expediency of the messages being sent is a higher priority than ensuring each message is delivered. Even with this approach, Kafka is highly available and it is not common that messages are dropped.

By using this method, the thread where you execute the action waits for the broker response. This leads to poor performance if the brokers are busy because the thread is blocked until the request is completed.

Using this approach, however, allows you to capture errors from the broker. Retriable errors are those caused by temporary issues (for example, a failed network connection). The producer can automatically retry these errors. Some errors are not caused by temporary issues (for example, if the message size is too large).

ProducerRecord<String, String> record = new ProducerRecord<>(
        "topic",
        "value"
    );

try {
    producer.send(record).get();
} catch (Exception e) {
    e.printStackTrace();
}

In the previous snippet, you execute get on the Future object returned by the send method. This blocks the current thread until the producer receives a response from the broker.

By using this method, the thread does not wait for a response from the broker. Instead, you define a callback to be executed when the response is ready. The producer thread continues sending messages and is interrupted to execute the callback whenever it is ready.

private class ProducerCallback implements Callback {
    @Override
    public void onCompletion(RecordMetadata recordMetadata, Exception e) {
        if (e != null) {
            e.printStackTrace();
        }
    }
}

ProducerRecord<String, String> record = new ProducerRecord<>(
        "topic",
        "value"
    );

producer.send(record, new ProducerCallback());

The previous snippet defines a callback that implements the org.apache.kafka.clients.producer.Callback interface. If an error has occurred, then the exception on the onCompletion method will not be null.

You should not include any heavy tasks inside the callback because it blocks the producer thread from sending messages or processing other callbacks. If long tasks are required, then you should use a separate thread.

This approach is a mix of the previous methods and it is commonly used in production environments.

The producer does not wait for a response from the broker. The message is marked as written as soon as it is sent. If an error occurs (for example, the leader is not available), then the producer does not receive a response. Not waiting for a response means that the producer is available for sending new messages faster.

The producer receives a response as soon as the leader receives the message. The producer does not wait for the message to be written in other replicas. If the leader crashes and another replica without the message is elected, then the message is lost.

The producer receives a response whenever the message is written to all available replicas. This ensures that the message is replicated and it is not lost if the leader crashes and a new leader election is needed.

If the leader is the only replica, however, then Kafka relies on the min.insync.replicas property. This property states how many replicas should be in-sync (with the same replicated data) before a producer request with ack=all is allowed. For example, if min.insync.replicas=2 and the leader is the only replica available, then the request is rejected. Otherwise, the behavior is the same as ack=1 and losing messages without replication is possible.