How to make use of DistCp performance improvements in Hadoop?

Every Hadoop developer would be aware of the DistCp tool which is used within Hadoop clusters. With the help of this tool, the user can take effective data backups within and across the Apache Hadoop clusters. Every data backup process conducted by running DistCp is defined as a ‘backup cycle’ and this process is slower when compared to many other processes in Apache Hadoop. Despite its slow performance, the tool gained enormous significance and popularity. In this blog, let us examine its importance and effective strategies to make use of these DistCp performance improvements.

Working of DistCp

The DistCp tool uses the MapReduce jobs for copying files from clusters and the following two steps will explain how this data transfer is done.

• Firstly, DistCp creates a copy list, a list that contains the files to be copied.

• Then, it runs a MapReduce job to identify and copy the files specified in the copy list.

Note: The MapReduce job copies the files only with mappers, subsets of files specified in the copy list.

Benefits of using DistCp

Using DistCP helps the users to avoid any data inconsistency because of the content changes while copying files. This is achieved by using read-only snapshots of the directory instead of using the actual directory.

The approach followed by DistCp is proving to be very useful, particularly while copying big directories and renaming those big directories in primary clusters. DistCp will not rename the copied directory without HDFS-7535 and this will control any large volumes of real-data copied to the cluster that is already available.

Latest Improvements in DistCp

• The time consumed by DistCp to create copy list is minimized.

• The tasks that every DistCp mapper should do are reduced and optimized.

• The number of files that can work on every backup cycle is reduced.

• The memory overflow risks while processing large directories are minimized.

• The data consistency during data backup is further improved.

How to use this feature?

Following are the typical steps to be followed to use this feature,

1. Firstly locate the source directory and create a snapshot with name s0.

1. Use the distcp command on the command line with destination directory using the following syntax,

1. distcp –update /. Snapshot/s0

1. Again create a snapshot with a name s0 in your desired destination directory.

1. Now make some changes in the source directory

1. Create another snapshot and name it as s1 by using the following command-line syntax,

1. distcp-update-diff s0 s1

1. Now create another snapshot with the same name s1 in your desired destination directory.

1. Repeat the steps 4 to 6 with a new snapshot name.

After following the above steps, you can clearly notice the new improvements incorporated in the DistCp.

Inspired by the world of Big Data and Hadoop? Looking for an opportunity to build a Hadoop career? Your opportunities are here…