A review on hadoop - HDFS infrastructure extensions

• Published in: 2013 IEEE Conference on Information and Communication Technologies pp. 132 - 137
• Main Authors: Kala Karun, A., Chitharanjan, K.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.04.2013
• Subjects: Data Layout; Distributed Computing; Fault tolerance; Fault tolerant systems; File systems; Hadoop; HDFS; Indexes; Layout; Trojan horses
• ISBN: 9781467357593; 1467357596
• DOI: 10.1109/CICT.2013.6558077

Apache's Hadoop 1 as of now is pretty good but there are scopes of extensions and enhancements. A large number of improvements are proposed to Hadoop which is an open source implementation of Google's Map/Reduce framework. It enables distributed, data intensive and parallel applications by decomposing a massive job into smaller tasks and a massive data set into smaller partitions such that each task processes a different partition in parallel. Hadoop uses Hadoop distributed File System (HDFS) which is an open source implementation of the Google File System (GFS) for storing data. Map/Reduce application mainly uses HDFS for storing data. HDFS is a very large distributed file system that uses commodity hardware and provides high throughput as well as fault tolerance. Many big enterprises believe that within a few years more than half of the world's data will be stored in Hadoop. HDFS stores files as a series of blocks and are replicated for fault tolerance. Strategic data partitioning, processing, layouts, replication and placement of data blocks will increase the performance of Hadoop and a lot of research is going on in this area. This paper reviews some of the major enhancements suggested to Hadoop especially in data storage, processing and placement.