4 best practices of a Hadoop Administrator!

Being one of the best open-source frameworks to manage big data in a distributed environment, Apache Hadoop keeps spreading its usage scale every other day. Thus, many database techies aspire to learn and administer Hadoop platform to survive and upgrade their career.

#4 – Stay connected to Hadoop Community

This is one of the essentials. Though the framework is built with potential features, still there are lots of bugs that need to be solved in Hadoop. It is more likely for any Hadoop administrator to encounter issues and troubles that need to be resolved. Since Hadoop is an open-source resource hence you cannot avail any instant tech support or service while using Hadoop unlike any other expensive platform. In such cases, it is wise of you to seek the help of someone in the Hadoop community. Thus, staying connected with the communities and forums pertaining to Hadoop administration and development would prove helpful in terms of connecting with the problem solvers. (Tip - Always keep yourself updated with the details of 10 Hadoop communities and forums at least)

#3 – Turn on the Trash function

Hadoop admin’s tasks are often hectic. Mistakes are certain to happen. One such mistake is accidental data deletion. While the giant Hadoop manages huge clusters of big data, often sized in terabytes, it’s too late to regret after deleting any data accidentally. There is no recycle bin or trash bin to restore the accidentally deleted data. Here comes the Trash function to the rescue. The Trash function belongs to the time machine in Hadoop which acts similar to the  recycle bin in the Windows operating system. By default, this function is in off state. All you got to do is just turn on the function and enjoy the feature.

To open and activate the Trash bin, here are the simple steps.

Open Core-site.xml VI and update the below configuration settings in the value units for minutes.

<property><name>fs.trash.interval</name><value>1440</value></property>

You can open the garbage bin using the “Delete” command and use the following “skiTrash” parameter to delete the contents.

        FS RM - skipTrash - /xxxx Hadoop

#2 – Using Hive or Pig

While Native Hadoop MapReduce can be developed using Java programming language, the administrator can also use Python, Shell and Perl programming languages to develop them. But instead of all, installing and using Apache Hive or Apache Pig is a wise idea that saves  a lot of time and effort.

#1 - NN and DN memory

After installing the Hadoop cluster, it is very important to reconfigure the file memory settings. Depending on the number of files and its memory, the NN and DN memory might vary. The administrator needs to modify the DN and NN memory settings as per the requirement. Open the following file to edit memory settings.

bin/hadoop-evn.sh

The following memory configuration is recommended by the experts.

15-25 GB NN:JT:2-4GBGB DN:1-4GB Child, VM 1-2 GB TT:1-2

(Note: A typical cluster with huge number of small files needs at least 20GB NN memory and 2GB DN memory.)

Learning Hadoop is interesting and career promising. If you’re interested to become a Hadoop administrator, get free career-counseling here…