What is Streaming Data Analytics?

The days where the historical data analysis and batch reports conquered are through. Today’s word demands much smarter and powerful data analytics systems. Here comes the Streaming Data Analytics to the rescue. The Streams lets you process data as it flows into your application, powering real-time dashboards and on-the-fly analytics and delivering data seamlessly to Hadoop clusters and NoSQL databases.

In making that observation I want to say: What a difference a year makes. Less than a year ago, I characterized streaming analytics as “the outlier in discussions about big data architectures.” Now, with the rapid rise of the streaming economy and the recent avid adoption of Spark Streaming, the topic has gone mainstream (pun intended).

In fact, one might argue that streaming data architectures are as fundamental to today’s “live” cloud-oriented data services as relational data architectures were to the prior era of on-premises database computing.

That explains why, for example, there’s growing interest in such approaches as “Lambda architecture,” which refers to the need to integrate both batch and streaming data processing within a common architecture under a common development, runtime, and administration paradigm.

Learning the data analysis and the rise of streaming analytics also explains why we’re seeing a surge in attempts to categorize “live” data integration patterns under which various stream processing architectures can operate with one another, as well as with batch, at-rest, and other “less than live” database architectures. For example, check out this recent blog post by Ashish Singh, who outlines a “canonical stream-processing architecture” built from two Apache codebases, Kafka, and Spark Streaming.

    The Apache Hadoop ecosystem has become a preferred platform for enterprises seeking to process and understand large-scale data in real time. Technologies like Apache Kafka, Apache Flume, Apache Spark, Apache Storm, and Apache Samza are increasingly pushing the envelope on what is possible. It is often tempting to bucket large-scale streaming use cases together but in reality they tend to break down into a few different architectural patterns, with different components of the ecosystem better suited for different problems.

In other words, the focus is on Hadoop’s streaming-focused “ecosystem,” not on its storage-centric platforms. That becomes clear as Malaska lays out the four principal stream-computing patterns:

• Stream ingestion that persists events at low latency to various stores (HDFS, HBase, Solr, and more)

• Near real-time processing with external context persisted and/or accessed from various stores

• Event-partitioned processing that persists relevant external content at ultra-low-latency to various in-memory platforms

• Complex multipotency (real-time and mini-batch) topologies that enable statefully, in-stream interception, sessionization, aggregation, windowed computation, machine learning processing, and other functions with high transactionality and accuracy