Kafka usecases
A brief overview of Kafa platform:
Kafka (Topics and State Stores):
Apache Kafka is a streaming platform
which enables users to publish data and also subscribe to different streams of
records. Kafka stores the streams in a fault tolerant way. It is used to build
reliable real-time streaming data pipelines. Further, it runs as a cluster and
implements the concept of topics or feed name which records are published to.
Kafka has great performance, and it is
stable, provides reliable durability, has a flexible publish-subscribe/queue
that scales well with N-number of consumer groups, has robust replication,
provides Producers with tunable consistency guarantees, and it provides
preserved ordering.
Usecase#1: App Analytics Realtime Dashboard (AARD)
Develop a scalable system which can do real-time analytics for different health applications. The occurrence of different health conditions can be regarded as the complex events and thus this concept can be extended to other use cases easily. Large number of users should be able to send the data in real-time, and should be able to receive the feedback and result. Keeping the requirements in mind we can use Kafka and Spark to develop the system. In this setting, multiple users are running Kafka producer clients, which are sending data in real-time. Spark streaming is used to process data from Kafka of different window sizes to analyze the health conditions. We can develop and test the heart attack risk and stress prediction as one sample complex events detection use case. We can simulate & and test it with multiple health datasets.
The system architecture is shown in the figure 1. Different users act as data producers and consumers in the Kafka model. Users publishes their health data for further processing. In this system the assumption is that users are aware of the different topics available for different types of sensory data. For example ECG data might be published to a topic dedicated to it, and BP data to a different topic and so-on. Kafka takes care of the scalability of the system as the number of users may grow or shrink. It also serves as a data retention, storage and forwarding interface.
The simplified data pipeline for the heart risk failure is shown in figure 2. The user publishes the ECG and BP data to the Kafka. Heart Risk predictor is running as spark job, and uses spark streaming to continuously listen the Kafka topic for the user data. It is subscribed to two topics, one for ECG and one for BP. The Heart Risk Predictor publishes its output to the Kafka topic to which the user is listening. The sample data format of the ECG is shown below where all the fields are self-explainable.
Eg: { "UserID " : " 1 0 1 " , "DataType " : "ECG" , "ValueType " : "DOUBLE " , "Value " : 8 2 . 2 8 , "TimeStamp " : 1 4 9 8 0 0 4 5 0 2 }
Note: Spark: a big data processing framework that runs on clusters
and we can get more details in apache site
Usecase#2: Freight Monitoring System (FMS)
Consider a hypothetical fleet management company that needs a dashboard to get the insight of its day to day activities related to vehicles. Each vehicle in this fleet management company is fitted with a GPS based geolocation emitter, which emits location data containing the following information.
1) Vehicle Id: A unique id is given to each vehicle on registration with the company.
2) Latitude and Longitude: geolocation information of vehicle.
3) Availability: The value of this field signifies whether the vehicle is available to take a booking or not. Current Status (Online/Offline) denotes whether the vehicle is on duty or not.
There are multiple ways to meet these requirements. We will try to keep this simple and follow the following steps to fulfill this need:
- All GPS signals will be sent to a topic.
- Our stream processor will read the records from this topic and perform the required grouping, aggregation, and materialization to Kafka state-stores.
The REST interface will be exposed. This will read and serve the data from Kafka state-stores, which were created in the earlier step.
Kafka (Topics and State Stores):
gpslocation: A Kafka topic that receives all messages that are emitted from vehicles.
statusCount: A state store that will maintain the “Online” and “Offline” vehicle counts. This is a key-value store, where the key is the status of the vehicle; value will be a count of the vehicle.
totalCount: A state store that will maintain the count of vehicles. This is done by counting the unique vehicle Id’s from all messages.
References:
