{"id":10289,"date":"2020-03-09T11:28:56","date_gmt":"2020-03-09T10:28:56","guid":{"rendered":"https:\/\/blog.mi.hdm-stuttgart.de\/?p=10289"},"modified":"2023-08-06T21:44:19","modified_gmt":"2023-08-06T19:44:19","slug":"distributed-stream-processing-frameworks-what-they-are-and-how-they-perform","status":"publish","type":"post","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2020\/03\/09\/distributed-stream-processing-frameworks-what-they-are-and-how-they-perform\/","title":{"rendered":"Distributed stream processing frameworks – what they are and how they perform"},"content":{"rendered":"\n

This blog aims to provide an overview about the topic of stream processing and its capabilites in a large scale environment. The post starts with an introduction to stream processing. After that, it explains how stream processing works and shows different areas of application as well as some common stream processing frameworks. Finally, this article will provide a performance comparison of several common frameworks based on benchmarking data.<\/p>\n\n\n

<\/p>\n\n\n\n

Let’s begin with an introduction to stream processing.<\/p>\n\n\n\n

Distributed stream processing engines are gaining popularity over the last years. Stream processing is a technology that can query continous streams of data in real-time and perform operations on the received data. It also goes by the name event-processing, Complex Event Processing, real-time-analytics or stream analytics. It allows to process data in real time as soon as it arrives in the system and can be used to quickly detect conditions in the received data. As an example imagine a temperature sensor that continously sends the temperature level. Once a certain temperature level is reached the system can trigger an alert based on the received data. [1] <\/p>\n\n\n\n

At a more technical level the stream processing engines process data in a pipeline-like structure. Data gets processed in terms of a Directed Acyclic Graph (see below). The processing can chain various functions together but never go back to an earlier point in the graph. Depending on the processed data steps in the chain can be skipped.<\/p>\n\n\n\n

\"Directed
Directed Acyclic Graph. Source: [5]<\/figcaption><\/figure>\n\n\n\n

\nThe\navailable frameworks approach this differently. Some frameworks let\nthe developer define the graph explicitly, thus the coding is at a\nmuch lower level. In these frameworks like Apache Storm or Apache\nSamza the developer has full control over the code but it is possible\nto write inefficient code.<\/p>\n\n\n\n

In\nother frameworks like Apache Flink or Apache Spark the developer can\nsimply chain functions together and the framework constructs the\ngraph. Thus the code is shaped in a very functional style as shown in\nthe example code snippet below. The code snippet shows a simple\napplication that counts words from an incoming stream for five\nseconds. [5]<\/p>\n\n\n\n

\"Apache
Apache Flink word count example code. Source: [4]<\/figcaption><\/figure>\n\n\n\n

On the contrary there is the “classic” approach of batch processing. Processing happens on blocks of data that have been collected and stored over a period of time. Depending on the size of the application this can be a huge amount of data with possibly millions of records.[1]
<\/p>\n\n\n\n


So why use stream processing instead of just processing the data in batches? And when?<\/strong><\/h2>\n\n\n\n

Some data that has to be processed inside an application naturally comes as a never-ending stream of data. For example healthcare sensor data, traffic sensor data or almost all IOT devices produce events continously. These types of data are time-series data for which the time of arrival or the time at which the event occured is important. Stream Processing frameworks naturally fit this model of time-series data. Detecting patterns and anomalies within data streams becomes easy. Additionally, you can inspect multiple data streams at once. [1]<\/p>\n\n\n\n

Processing constantly arriving streaming data in batches would require to stop data collection at some point, store the data and then process it. Then you’ll have to worry about aggregation across multiple batches. So using a framework that fits this model makes perfect sense. [1]<\/p>\n\n\n\n

Now let’s look at some example usecases where streaming can be used beneficially:
<\/p>\n\n\n\n


Event-driven applications<\/strong><\/h2>\n\n\n\n

An event-driven application retrieves events from possibly multiple sources and performs operations on these events. Instead of writing data to a transactional database the data and state of the application are kept on the local system. By keeping the data on the local system the performance (latency and throughput) of the application is improved and possible network failure will be avoided. The application writes checkpoints periodically for fault-tolerance but this can happen asynchronously and does not impact performance. Examples for event-driven applications can be fraud detection, anomaly detection or rule-based alerting. [2]<\/p>\n\n\n\n

The following image, taken from Apache Flink, illustrates how the architecture of an event-driven application is structured. (Apache Flink will be described later in this blog post)<\/p>\n\n\n\n

\"Architecture
Architecture of an event-driven application. Source: [2]<\/figcaption><\/figure>\n\n\n\n


Data Analytics Applications<\/strong><\/h2>\n\n\n\n

A data analytics application extracts insights from raw data. While this can also be done with batch processing it is a good usecase for a streaming application. The advantage of using a stream processing engine is the capability of using real-time-data and continously producing results. The results can be persisted to an external storage and\/or shown in a live report in real time. As a result of using stream processing, the latency of processing events gets lowered. Examples for data analytics applications can be quality monitoring of telecommunication networks or analysis of product updates. [2]<\/p>\n\n\n\n

The following image shows two architectures of data analytics applications using Apache Flink.<\/p>\n\n\n\n

\"\"
Batch and streaming analytics architectures in comparison. Source: [2]<\/figcaption><\/figure>\n\n\n\n

Now that we’ve seen an introduction to stream processing as well as some usecases. let’s look at what stream processing frameworks exist.<\/p>\n\n\n\n


An overview about available frameworks<\/strong><\/h2>\n\n\n\n

The history of stream processing began with Apache Hadoop’s batch processing engine and later shifted towards stream processing. By now, the following popular frameworks have implementations for stream processing:<\/p>\n\n\n\n