My Kafka Learning
Though I am not an expert in Kafka but I am giving some of my learning points as a part of my Kafka integration exercise with Pega Infinity. To begin with Kafka, I was curious to know how it works and what are all the various components to understand well along with Kafka cluster architecture. We can get any Kafka related information from Google and I am trying to cover the fundamentals/basics & the importance of Kafka in my words. For any extra information related to Kafka, it is good to refer the books or we can do some research in net.
Kafka is a distributed
publish-subscribe messaging system that is designed to be fast, scalable and
durable. It was developed by LinkedIn and open sourced in the year 2011. It
makes an extremely desirable option for data integration with the increasing
complexity in real time data processing challenges. It is great solution for
applications that require large scale message processing.
The main component of Kafka are:
1.
Zookeeper –Zookeeper is used for
managing and coordinating Kafka broker
2.
Kafka Cluster – which contains one or more
servers called as brokers
3.
Producer – which publish messages to Kafka
4.
Consumer – which consumes messages from Kafka.
5. Topic –A
topic is a category or feed name to which records are published.
6. Partitions – Topics
are broken up into segments called partitions.
7. Messages – Messages
are simply byte arrays and the developers can use them to store any object in
any format – with String, JSON the most common.
8. Replicas – A replica of a partition is a "backup" of a partition. Replicas never read or write data. They are used to prevent data loss.
9. Consumer Group – A consumer group includes the set of consumer processes that are subscribing to a specific topic.
10. Offset – The offset is a unique identifier of a record within a partition. It denotes the position of the consumer in the partition.
11. Node – A node is a single computer in the Apache Kafka cluster.
Basic Kafka cluster architecture:
To
understand need of Kafka, first we need to understand what are data
pipelines?
Data
Pipelines: Data pipelines are the ones, which are establishing the
connection for communication between two systems or services.
Kafka decouples the data
pipelines between the systems and thus makes the communication between
systems simpler and manageable.
Some of the features of Kafka:
- Highly scalable with no downtime.
- Kafka with proper configuration ensures zero data loss.
- Kafka can be used along with real time streaming apps like Spark & Storm
- Kafka persists all the messages to the disk and can further used for batch consumption.
Some other common use-cases for Kafka:
- Log aggregation: Kafka can be used across an organization to collect logs from multiple services and make them available in standard format to multiple consumers, including Hadoop.
- Website activity tracking: Web application sends events such as page views and searches to Kafka, where they become available for real-time processing, dashboards and offline analytics in Hadoop.
- Stream processing: A framework such as Spark Streaming reads data from a topic, processes it and writes processed data to a new topic where it becomes available for users and applications.
Kafka
has great performance, and it is stable, provides reliable durability, has a
flexible publish-subscribe/queue that scales well with N-number of consumer
groups, has robust replication, provides Producers with tunable consistency
guarantees, and it provides preserved ordering.
Small Exercise: Pega Infinity integration to Kafka
As a part of small experiment from my local V8.1 Pega Environment (PE), I did install Kafka, Zookeper before integration work. The process of installation from Windows OS is available =>https://dzone.com/articles/running-apache-kafka-on-windows-os
After installation, run the below commands from each command prompt in Windows separately.
1) C:>zkserver => Run Zookeeper
2) C:\Kafka\kafka_2.11-2.1.0>.\bin\windows\kafka-server-start.bat .\config\server.properties => Run Kafka server
3) Create a Topic:
C:\Kafka\kafka_2.11-2.1.0>bin\windows>kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic SamplePegaTopic
Once the above is done, we need to create the Data Set rule from Pega and do the necessary configurations to connect Kafka server. Below is the high level snapshot of how I have designed in Pega Infinity to push the data and pull the data using a sample topic created :SamplePegaTopic in my local Kafka server. You can see how the stream of data is pushed into Kafka. In this simple scenario, Pega is used as a producer (post messages) and a consumer (pull/read messages) from the topic.
This is a small POC on how to connect Kafka and it can be extendable based on our needs while working on the actual requirements. There is another approach in Pega to connect Kafka and pull the messages using Data Flow rule. From the Data Flow, we can setup the Data Set and we can also tell the system to work on a specific node and we can focus on the recent messages from the queue. The usage of any approach depends on the requirement and there is no hard & fast rule to use the approach.
Note: As I mentioned, I am not expert in Kafka and still I need to focus on many things to become master JJ
********** HAPPY LEARNING & SHARING ************
References:
