Expert's advice for Hadoop Developers

Here are the tips for Developing Effective Big Data Applications using Hadoop framework. Witnesses so many use cases of Hadoop and big data and in an enterprise, there are several challenges Hadoop developers need to overcome when identifying the use case and measuring its success rate. The largest barrier to enterprise adoption of Hadoop is not having a clearly defined big data use case. So when and where do you start? Career counselors at Sulekha have collated some of the best tips from Hadoop experts for beginners to get started with Hadoop deployments across the enterprise. These tips will help you learn how to use the most popular open source big data framework Hadoop.

Best suiting MapReduce Language

There are many languages and frameworks that sit on top of MapReduce, so it’s worth thinking up-front which one to use for a particular problem. There is no one-size-fits-all language; each has different strengths and weaknesses.

Java: Good for: speed; control, binary data, working with existing Java or MapReduce libraries.

Pipes: Good for: working with existing C++ libraries.

Streaming: Good for: writing MapReduce programs in scripting languages.

Dumbo (Python), Happy (Jython), Wukong (Ruby), mrtoolkit (Ruby): Good for: Python/Ruby programmers who want quick results, and are comfortable with the MapReduce abstraction.

Pig, Hive, Cascading: Good for: higher-level abstractions; joins; nested data.

While there are no hard and fast rules, in general, we recommend using pure Java for large, recurring jobs, Hive for SQL style analysis and data warehousing, and Pig or Streaming for ad-hoc analysis.

Input data “chunk” size matters

Are you generating large, unbounded files, like log files? Or lots of small files, like image files? How frequently do you need to run jobs?

Answers to these questions determine how your store and process data using HDFS. For large unbounded files, one approach (until HDFS appends are working) is to write files in batches and merge them periodically. For lots of small files, see The Small Files Problem. HBase is a good abstraction for some of these problems too, so may be worth considering.

Benefit of using SequenceFile and MapFile containers

SequenceFiles are a very useful tool. They are:

Splittable. So they work well with MapReduce: each map gets an independent split to work on.

Compressible. By using block compression you get the benefits of compression (use less disk space, faster to read and write), while keeping the file splittable still.

Compact. SequenceFiles are usually used with Hadoop Writable objects, which have a pretty compact format.

A MapFile is an indexed SequenceFile, useful for if you want to do look-ups by key.

However, both are Java-centric, so you can’t read them with non-Java tools. The Thrift and Avro projects are the places to look for language-neutral container file formats. (For example, see Avro’s DataFileWriter although there is no MapReduce integration yet.)