What are the Hadoop and Big Data components that needs an upgrade in 2017?

We are living in this data-driven world with big data analytic adventure scattering across the businesses and organizations. Acquiring Hadoop and Big Data skills had become a necessity and these platforms reached almost every big organization to help them stay ahead in spite of competition. However, not every component of Big Data platforms remains shiny and new anymore. In fact, some crucial components and technologies may be holding you back. Remember, this is the fastest-moving area of enterprise tech -- so much so that some software acts as a placeholder until better bits arrive.

Slow MapReduce – It is a fact that most Big Data professionals would agree. MapReduce processes are ridiculously slow. It's rarely the optimistic way to go about a problem. Posing as effective alternatives there are other algorithms to choose from -- the most common is DAG, of which MapReduce can be considered a subset. If you’ve done a bunch of custom MapReduce jobs, the performance difference compared to Spark is worth the cost and trouble of switching.

The dominance of Spark over Storm

With the advent of technologies such as Apex and Flink, there are better, lower-latency alternatives to Spark than Storm. In addition to that, while evaluating the latency tolerance and whether the bugs you have in your lower-level, more complicated code are worth a few extra milliseconds. Storm doesn’t have the support that it could, with Hortonworks as the only real backer -- and with Hortonworks facing increasing market pressure, Storm is unlikely to get more attention.

Spark already does what Pig does

While considering the efficiency and performance one must opt for Apache Spark to add in their Big Data platform cluster. Spark is capable of doing all the functionalities of Apache Pig. Thus there is no need to install Pig to the Big Data environment.

Java Syntax aren’t friendly enough for Big Data

Though the Java Virtual Machine (JVM) is awesome compiler and interpreter that any object-oriented programming language could offer, the Java language and its syntax are a bit clunky for big data processes. Plus, newer constructs like Lambda have been bolted onto the side in a somewhat awkward manner. The big data world has largely moved to Scala and Python (the latter when you can afford the performance hit and need Python libraries or are infested with Python developers). Of course, you can use R for stats, until you rewrite it in Python because R doesn’t have all the fun scale features.

Hortonworks Tez doesn’t do what Spark can’t do

Hortonworks Tez is a DAG implementation and it is described by one of its developers as like writing in “assembly language.” At the moment, with a Hortonworks distribution, you’ll end up using Tez behind Hive and other tools -- but you can already use Spark as the engine in other distributions. Tez has always been kind of buggy anyhow. Again, this is one vendor’s project and doesn’t have the industry or community support of other technologies. It doesn’t have any runaway advantages over other solutions. This is an engine I’d look to consolidate out.