New database solution supported by Apache Spark!

Yes, that’s right! Now Apache Spark is powering live SQL analytics in a newly unveiled database solution software called SnappyData. This purposeful tool is created by the same team which created GemFire and the main reason for it to be built upon Apache Spark is that to enable larger database solution. This is so far the recent such event where a large database solution is created using Apache Spark. The program can use an exclusive database to analyze OLAP and OLTP workloads parallel.

Empowered by in-memory data analytics engine

SnappyData proves extremely capable of performing live SQL analytics on static and streaming data sets effectively. It uses the Apache Spark’s in-memory data analytics engine in order achieve excellent performance. Queries in SnappyData are written using conventional SQL or exclusive Spark abstractions and so the existing works which are already done in either paradigm can be reused effectively.

To capture and store the data, SnappyData software possesses a distributed data store called Snappy-Store. It acts as an own data store and as well as a sort of asynchronous writeback cache for various other data sources (example: Hadoop/HDFS) This signifies that the existing datasets are able to be accessed with the help of SnappyData irrespective of the need for formal migration.

SnappyData is also equipped to provide solutions to various problems that have the possibility to arise during data streaming. For example, in the case of too much data flow demanding a response in real-time, SnappyData can use AQP (Approximate Query Processing) or any other similar method to sample the streaming data and retrieve the desired answer.

In most occasions, the end outputs are not that exact when compared to operating on the whole data set for the reason that Approximate Query Processing is not available for all types of queries. On that particular note, we can assume that the AQP queries have primary intentions to run faster with less CPU and memory requirements.

Yet again a data analysis solution

In fact, this is not the first time where the world witnesses the usage of Spark as the heart of a data analysis solution software that includes processing of OLTP and OLAP workloads. For instance, Splice Machine is an in-memory database system which was built on top of several Hadoop components to run both OLTP and OLAP works under the similar hood. One special thing about SnappyData is that it diverges several qualities from Splice Machine in terms of utilizing the Apache Spark effectively. SnappyData also claims that it would extend Spark Streaming in several manners enabling the streams to be manipulated and queried even if they were just tables, during join operations.