Apache Kafka Necessities Cheatsheet – Java Code Geeks

1. Introduction

Apache Kafka is a distributed occasion streaming platform designed to deal with large-scale real-time information streams. It was initially developed by LinkedIn and later open-sourced as an Apache challenge. Kafka is understood for its high-throughput, fault-tolerance, scalability, and low-latency traits, making it a superb selection for varied use instances, similar to real-time information pipelines, stream processing, log aggregation, and extra.

Kafka follows a publish-subscribe messaging mannequin, the place producers publish messages to matters, and shoppers subscribe to these matters to obtain and course of the messages.

2. Putting in and Configuring Kafka

To get began with Apache Kafka, you’ll want to obtain and arrange the Kafka distribution. Right here’s how you are able to do it:

2.1 Downloading Kafka

Go to the Apache Kafka web site (https://kafka.apache.org/downloads) and obtain the newest secure model.

2.2 Extracting the Archive

After downloading the Kafka archive, extract it to your required location utilizing the next instructions:

# Exchange kafka_version with the model you downloaded
tar -xzf kafka_version.tgz
cd kafka_version

2.3 Configuring Kafka

Navigate to the config listing and modify the next configuration information as wanted:

server.properties: Important Kafka dealer configuration.

zookeeper.properties: ZooKeeper configuration for Kafka.

3. Beginning Kafka and ZooKeeper

To run Kafka, you’ll want to begin ZooKeeper first, as Kafka relies on ZooKeeper for sustaining its cluster state. Right here’s methods to do it:

3.1 Beginning ZooKeeper

bin/zookeeper-server-start.sh config/zookeeper.properties

3.2 Beginning Kafka Dealer

To start out the Kafka dealer, use the next command:

bin/kafka-server-start.sh config/server.properties

4. Creating and Managing Subjects

Subjects in Kafka are logical channels the place messages are printed and consumed. Let’s discover ways to create and handle matters:

4.1 Making a Subject

To create a subject, use the next command:

bin/kafka-topics.sh --create --topic my_topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

On this instance, we create a subject named my_topic with three partitions and a replication issue of 1.

4.2 Itemizing Subjects

To checklist all of the matters within the Kafka cluster, use the next command:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

4.3 Describing a Subject

To get detailed details about a selected subject, use the next command:

bin/kafka-topics.sh --describe --topic my_topic --bootstrap-server localhost:9092

5. Producing and Consuming Messages

Now that we have now a subject, let’s discover methods to produce and eat messages in Kafka.

5.1 Producing Messages

To provide messages to a Kafka subject, use the next command:

bin/kafka-console-producer.sh --topic my_topic --bootstrap-server localhost:9092

After operating this command, you can begin typing your messages. Press Enter to ship every message.

5.2 Consuming Messages

To eat messages from a Kafka subject, use the next command:

bin/kafka-console-consumer.sh --topic my_topic --bootstrap-server localhost:9092

This may begin consuming messages from the desired subject within the console.

5.3 Shopper Teams

Shopper teams enable a number of shoppers to work collectively to learn from a subject. Every shopper in a bunch will get a subset of the messages. To make use of shopper teams, present a bunch id when consuming messages:

bin/kafka-console-consumer.sh --topic my_topic --bootstrap-server localhost:9092 --group my_consumer_group

6. Configuring Kafka Producers and Customers

Kafka gives varied configurations for producers and shoppers to optimize their conduct. Listed here are some important configurations:

6.1 Producer Configuration

To configure a Kafka producer, create a producer.properties file and set properties like bootstrap.servers, key.serializer, and worth.serializer.

# producer.properties

bootstrap.servers=localhost:9092
key.serializer=org.apache.kafka.frequent.serialization.StringSerializer
worth.serializer=org.apache.kafka.frequent.serialization.StringSerializer

Use the next command to run the producer with the desired configuration:

bin/kafka-console-producer.sh --topic my_topic --producer.config path/to/producer.properties

6.2 Shopper Configuration

For shopper configuration, create a shopper.properties file with properties like bootstrap.servers, key.deserializer, and worth.deserializer.

# shopper.properties

bootstrap.servers=localhost:9092
key.deserializer=org.apache.kafka.frequent.serialization.StringDeserializer
worth.deserializer=org.apache.kafka.frequent.serialization.StringDeserializer
group.id=my_consumer_group

Run the buyer utilizing the configuration file:

bin/kafka-console-consumer.sh --topic my_topic --consumer.config path/to/shopper.properties

7. Kafka Join

Kafka Join is a strong framework that means that you can simply combine Apache Kafka with exterior methods. It’s designed to offer scalable and fault-tolerant information motion between Kafka and different information storage methods or information processing platforms. Kafka Join is right for constructing information pipelines and transferring information to and from Kafka with out writing customized code for every integration.

Kafka Join consists of two essential elements: Supply Connectors and Sink Connectors.

7.1 Supply Connectors

Supply Connectors assist you to import information from varied exterior methods into Kafka. They act as producers, capturing information from the supply and writing it to Kafka matters. Some standard supply connectors embrace:

• JDBC Supply Connector: Captures information from relational databases utilizing JDBC.
• FileStream Supply Connector: Reads information from information in a specified listing and streams them to Kafka.
• Debezium Connectors: Supplies connectors for capturing modifications from varied databases like MySQL, PostgreSQL, MongoDB, and so on.

7.2 Sink Connectors

Sink Connectors assist you to export information from Kafka to exterior methods. They act as shoppers, studying information from Kafka matters and writing it to the goal methods. Some standard sink connectors embrace:

• JDBC Sink Connector: Writes information from Kafka matters to relational databases utilizing JDBC.
• HDFS Sink Connector: Shops information from Kafka matters in Hadoop Distributed File System (HDFS).
• Elasticsearch Sink Connector: Indexes information from Kafka matters into Elasticsearch for search and evaluation.

7.3 Configuration

To configure Kafka Join, you usually use a properties file for every connector. The properties file accommodates important data just like the connector title, Kafka brokers, subject configurations, and connector-specific properties. Every connector might have its personal set of required and non-compulsory properties.

Right here’s a pattern configuration for the FileStream Supply Connector:

title=my-file-source-connector
connector.class=org.apache.kafka.join.file.FileStreamSourceConnector
duties.max=1
file=/path/to/inputfile.txt
subject=my_topic

7.4 Working Kafka Join

To run Kafka Join, you should utilize the connect-standalone.sh or connect-distributed.sh scripts that include Kafka.

Standalone Mode

In standalone mode, Kafka Join runs on a single machine, and every connector is managed by a separate course of. Use the connect-standalone.sh script to run connectors in standalone mode:

bin/connect-standalone.sh config/connect-standalone.properties config/your-connector.properties

Distributed Mode

In distributed mode, Kafka Join runs as a cluster, offering higher scalability and fault tolerance. Use the connect-distributed.sh script to run connectors in distributed mode:

bin/connect-distributed.sh config/connect-distributed.properties

7.5 Monitoring Kafka Join

Kafka Join exposes a number of metrics that may be monitored for understanding the efficiency and well being of your connectors. You should utilize instruments like JConsole, JVisualVM, or combine Kafka Join with monitoring methods like Prometheus and Grafana to observe the cluster.

8. Kafka Streams

Kafka Streams is a shopper library in Apache Kafka that permits real-time stream processing of information. It means that you can construct purposes that eat information from Kafka matters, course of the information, and produce the outcomes again to Kafka or different exterior methods. Kafka Streams gives a easy and light-weight method to stream processing, making it a beautiful selection for constructing real-time information processing pipelines.

8.1 Key Ideas

Earlier than diving into the main points of Kafka Streams, let’s discover some key ideas:

• Stream: A steady circulate of information information in Kafka is represented as a stream. Every file within the stream consists of a key, a worth, and a timestamp.
• Processor: A processor is a basic constructing block in Kafka Streams that processes incoming information information and produces new output information.
• Topology: A topology defines the stream processing circulate by connecting processors collectively to type a processing pipeline.
• Windowing: Kafka Streams helps windowing operations, permitting you to group information inside specified time intervals for processing.
• Stateful Processing: Kafka Streams helps stateful processing, the place the processing logic considers historic information inside a specified window.

8.2 Kafka Streams Utility

To create a Kafka Streams software, you’ll want to arrange a Kafka Streams topology and outline the processing steps. Right here’s a high-level overview of the steps concerned:

Create a Properties Object

Begin by making a Properties object to configure your Kafka Streams software. This contains properties just like the Kafka dealer handle, software ID, default serializers, and deserializers.

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "my-streams-app");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

Outline the Topology

Subsequent, outline the topology of your Kafka Streams software. This entails creating processing steps and connecting them collectively.

StreamsBuilder builder = new StreamsBuilder();

// Create a stream from a Kafka subject
KStream<String, String> inputStream = builder.stream("input_topic");

// Carry out processing operations
KStream<String, String> processedStream = inputStream
    .filter((key, worth) -> worth.startsWith("important_"))
    .mapValues(worth -> worth.toUpperCase());

// Ship the processed information to a different Kafka subject
processedStream.to("output_topic");

// Construct the topology
Topology topology = builder.construct();

Create and Begin the Kafka Streams Utility

As soon as the topology is outlined, create a KafkaStreams object with the outlined properties and topology, and begin the appliance:

KafkaStreams streams = new KafkaStreams(topology, props);
streams.begin();

8.3 Stateful Processing with Kafka Streams

Kafka Streams gives state shops that assist you to preserve stateful processing throughout information information. You’ll be able to outline a state retailer and use it inside your processing logic to take care of state data.

8.4 Windowing Operations

Kafka Streams helps windowing operations, permitting you to group information information inside particular time home windows for aggregation or processing. Windowing is crucial for time-based operations and calculations.

8.5 Interactive Queries

Kafka Streams additionally allows interactive queries, permitting you to question the state shops utilized in your stream processing software.

8.6 Error Dealing with and Fault Tolerance

Kafka Streams purposes are designed to be fault-tolerant. They routinely deal with and get better from failures, making certain steady information processing.

8.7 Integration with Kafka Join and Kafka Producer/Shopper

Kafka Streams can simply combine with Kafka Join to maneuver information between Kafka matters and exterior methods. Moreover, you should utilize Kafka producers and shoppers inside Kafka Streams purposes to work together with exterior methods and providers.

9. Kafka Safety

Making certain the safety of your Apache Kafka cluster is crucial to defending delicate information and stopping unauthorized entry. Kafka gives varied safety features and configurations to safeguard your information streams. Let’s discover some important features of Kafka safety:

9.1 Authentication and Authorization

Kafka helps each authentication and authorization mechanisms to regulate entry to the cluster.

Authentication

Kafka presents a number of authentication choices, together with:

• SSL Authentication: Safe Sockets Layer (SSL) allows encrypted communication between shoppers and brokers, making certain safe authentication.
• SASL Authentication: Easy Authentication and Safety Layer (SASL) gives pluggable authentication mechanisms, similar to PLAIN, SCRAM, and GSSAPI (Kerberos).

Authorization

Kafka permits fine-grained management over entry to matters and operations utilizing Entry Management Lists (ACLs). With ACLs, you may outline which customers or teams are allowed to learn, write, or carry out different actions on particular matters.

9.2 Encryption

Kafka gives information encryption to guard information whereas it’s in transit between shoppers and brokers.

SSL Encryption

SSL encryption, when mixed with authentication, ensures safe communication between shoppers and brokers by encrypting the information transmitted over the community.

Encryption at Relaxation

To guard information at relaxation, you may allow disk-level encryption on the Kafka brokers.

Safe ZooKeeper

As Kafka depends on ZooKeeper for cluster coordination, securing ZooKeeper can be essential.

Chroot

Kafka means that you can isolate the ZooKeeper occasion utilized by Kafka by utilizing a chroot path. This helps forestall different purposes from accessing Kafka’s ZooKeeper occasion.

Safe ACLs

Be sure that the ZooKeeper occasion utilized by Kafka has safe ACLs set as much as prohibit entry to approved customers and processes.

9.3 Safe Replication

You probably have a number of Kafka brokers, securing replication between them is crucial.

Inter-Dealer Encryption

Allow SSL encryption for inter-broker communication to make sure safe information replication.

Managed Shutdown

Configure managed shutdown to make sure brokers shut down gracefully with out inflicting information loss or inconsistency throughout replication.

Safety Configuration

To allow safety features in Kafka, you’ll want to modify the Kafka dealer configuration and alter the shopper configurations accordingly.

Dealer Configuration

Within the server.properties file, you may configure the next security-related properties:

listeners=PLAINTEXT://:9092,SSL://:9093
safety.inter.dealer.protocol=SSL
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=keystore_password
ssl.key.password=key_password

Shopper Configuration

Within the shopper purposes, you’ll want to set the safety properties to match the dealer configuration:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9093");
props.put("safety.protocol", "SSL");
props.put("ssl.keystore.location", "/path/to/client_keystore.jks");
props.put("ssl.keystore.password", "client_keystore_password");
props.put("ssl.key.password", "client_key_password");

10. Replication Issue

Replication issue is a vital idea in Apache Kafka that ensures information availability and fault tolerance inside a Kafka cluster. It defines the variety of copies, or replicas, of every Kafka subject partition that ought to be maintained throughout the brokers within the cluster. By having a number of replicas of every partition, Kafka ensures that even when some brokers or machines fail, the information stays accessible and the cluster stays operational.

10.1 How Replication Issue Works

When a brand new subject is created or when an current subject is configured to have a selected replication issue, Kafka routinely replicates every partition throughout a number of brokers. The partition chief is the first reproduction chargeable for dealing with learn and write requests for that partition, whereas the opposite replicas are known as follower replicas.

10.2 Modifying Replication Issue

Altering the replication issue of an current subject entails reassigning partitions and including or eradicating replicas. This course of ought to be carried out rigorously, as it might influence the efficiency of the cluster throughout rebalancing.

To extend the replication issue, you’ll want to add new brokers after which reassign the partitions with the brand new replication issue utilizing the kafka-reassign-partitions.sh device.

To lower the replication issue, you’ll want to reassign the partitions and take away replicas earlier than eradicating the brokers from the cluster.

11. Partitions

Partitions are a basic idea in Apache Kafka that enables information to be distributed and parallelized throughout a number of brokers in a Kafka cluster. A subject in Kafka is split into a number of partitions, and every partition is a linearly ordered sequence of messages. Understanding partitions is essential for optimizing information distribution, load balancing, and managing information retention inside Kafka.

11.1 How Partitions Work

When a subject is created, it’s divided right into a configurable variety of partitions. Every partition is hosted on a selected dealer within the Kafka cluster. The variety of partitions in a subject will be set when creating the subject, and the partitions stay mounted after creation. Messages produced to a subject are written to considered one of its partitions primarily based on the message’s key or utilizing a round-robin mechanism if no key’s offered.

11.2 Advantages of Partitions

Partitioning gives a number of benefits:

11.3 Partition Key

When producing messages to a Kafka subject, you may specify a key for every message. The secret’s non-compulsory, and if not offered, messages are distributed to partitions utilizing a round-robin method. When a key’s offered, Kafka makes use of the important thing to find out the partition to which the message can be written.

11.4 Selecting the Variety of Partitions

The variety of partitions for a subject is a vital consideration and ought to be chosen rigorously primarily based in your use case and necessities.

11.5 Modifying Partitions

As soon as a subject is created with a selected variety of partitions, the variety of partitions can’t be modified instantly. Including or decreasing partitions requires cautious planning and entails the next steps:

Growing Partitions

To extend the variety of partitions, you may create a brand new subject with the specified partition depend and use Kafka instruments like kafka-topics.sh to reassign messages from the outdated subject to the brand new one.

Lowering Partitions

Lowering the variety of partitions is more difficult and would possibly contain reassigning messages manually to take care of information integrity.

12. Batch Dimension

Batch dimension in Apache Kafka refers back to the variety of messages which might be amassed and despatched collectively as a batch from producers to brokers. By sending messages in batches as an alternative of individually, Kafka can obtain higher efficiency and cut back community overhead. Configuring an acceptable batch dimension is crucial for optimizing Kafka producer efficiency and message throughput.

12.1 How Batch Dimension Works

When a Kafka producer sends messages to a dealer, it could select to batch a number of messages collectively earlier than sending them over the community. The producer collects messages till the batch dimension reaches a configured restrict or till a sure time interval elapses. As soon as the batch dimension or time restrict is reached, the producer sends your complete batch to the dealer in a single request.

12.2 Configuring Batch Dimension

In Kafka, you may configure the batch dimension for a producer utilizing the batch.dimension property. This property specifies the utmost variety of bytes {that a} batch can comprise. The default worth is 16384 bytes (16KB).

You’ll be able to alter the batch dimension primarily based in your use case, community situations, and message dimension. Setting a bigger batch dimension can enhance throughput, nevertheless it may additionally enhance the latency for particular person messages throughout the batch. Conversely, a smaller batch dimension might cut back latency however might end in a better variety of requests and elevated community overhead.

12.3 Monitoring Batch Dimension

Monitoring the batch dimension is essential for optimizing producer efficiency. You should utilize Kafka’s built-in metrics and monitoring instruments to trace batch size-related metrics, similar to common batch dimension, most batch dimension, and batch ship time.

13. Compression

Compression in Apache Kafka is a characteristic that enables information to be compressed earlier than it’s saved on brokers or transmitted between producers and shoppers. Kafka helps varied compression algorithms to cut back information dimension, enhance community utilization, and improve total system efficiency. Understanding compression choices in Kafka is crucial for optimizing storage and information switch effectivity.

13.1 How Compression Works

When a producer sends messages to Kafka, it could select to compress the messages earlier than transmitting them to the brokers. Equally, when messages are saved on the brokers, Kafka can apply compression to cut back the storage footprint. On the buyer facet, messages will be decompressed earlier than being delivered to shoppers.

13.2 Compression Algorithms in Kafka

Kafka helps the next compression algorithms:

13.3 Configuring Compression in Kafka

To allow compression in Kafka, you’ll want to configure the producer and dealer properties.

Producer Configuration

Within the producer configuration, you may set the compression.kind property to specify the compression algorithm to make use of. For instance:

compression.kind=gzip

Dealer Configuration

Within the dealer configuration, you may specify the compression kind for each producer and shopper requests utilizing the compression.kind property. For instance:

compression.kind=gzip

13.4 Compression in Kafka Streams

When utilizing Apache Kafka Streams, you may also configure compression for the state shops utilized in your stream processing software. This may also help cut back storage necessities for stateful information within the Kafka Streams software.

13.5 Issues for Compression

Whereas compression presents a number of advantages, it’s important to think about the next elements when deciding whether or not to make use of compression:

14. Retention Coverage

Retention coverage in Apache Kafka defines how lengthy information is retained on brokers inside a Kafka cluster. Kafka means that you can set totally different retention insurance policies at each the subject degree and the dealer degree. The retention coverage determines when Kafka will routinely delete outdated information from matters, serving to to handle storage utilization and stop unbounded information development.

14.1 How Retention Coverage Works

When a message is produced to a Kafka subject, it’s written to a partition on the dealer. The retention coverage defines how lengthy messages inside a partition are saved earlier than they’re eligible for deletion. Kafka makes use of a mix of time-based and size-based retention to find out which messages to retain and which to delete.

14.2 Configuring Retention Coverage

The retention coverage will be set at each the subject degree and the dealer degree.

Subject-level Retention Coverage

When making a Kafka subject, you may specify the retention coverage utilizing the retention.ms property. This property units the utmost time, in milliseconds, {that a} message will be retained within the subject.

For instance, to set a retention coverage of seven days for a subject:

bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic my_topic --partitions 3 --replication-factor 2 --config retention.ms=604800000

Dealer-level Retention Coverage

It’s also possible to set a default retention coverage on the dealer degree within the server.properties file. The log.retention.hours property specifies the default retention time for matters that don’t have a selected retention coverage set.

For instance, to set a default retention coverage of seven days on the dealer degree:

log.retention.hours=168

14.3 Dimension-based Retention

Along with time-based retention, Kafka additionally helps size-based retention. With size-based retention, you may set a most dimension for the partition log. As soon as the log dimension exceeds the desired worth, the oldest messages within the log are deleted to create space for brand new messages.

To allow size-based retention, you should utilize the log.retention.bytes property. For instance:

log.retention.bytes=1073741824

14.4 Log Compaction

Along with time and size-based retention, Kafka additionally gives a log compaction characteristic. Log compaction retains solely the newest message for every distinctive key in a subject, making certain that the latest worth for every key’s at all times obtainable. This characteristic is beneficial for sustaining the newest state of an entity or for storing changelog-like information.

To allow log compaction for a subject, you should utilize the cleanup.coverage property. For instance:

cleanup.coverage=compact

14.5 Issues for Retention Coverage

When configuring the retention coverage, contemplate the next elements:

15. Kafka Monitoring and Administration

Monitoring Kafka is crucial to make sure its clean operation. Listed here are some instruments and strategies for efficient Kafka monitoring:

16. Dealing with Information Serialization

Kafka means that you can use totally different information serializers on your messages. Right here’s how one can deal with information serialization in Apache Kafka:

17. Kafka Ecosystem: Extra Elements

Kafka’s ecosystem presents varied further elements that reach its capabilities. Listed here are some important ones:

18. Conclusion

This was the Apache Kafka Necessities Cheatsheet, offering you with a fast reference to the basic ideas and instructions for utilizing Apache Kafka. As you delve deeper into the world of Kafka, keep in mind to discover the official documentation and neighborhood assets to achieve a extra complete understanding of this highly effective occasion streaming platform.