A comparison of parallel large-scale knowledge acquisition using rough set theory on different MapReduce runtime systems

• Published in: International journal of approximate reasoning Vol. 55; no. 3; pp. 896 - 907
• Main Authors: Zhang, Junbo, Wong, Jian-Syuan, Li, Tianrui, Pan, Yi
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier Inc 01.03.2014; Elsevier
• Subjects: Applied sciences; Artificial intelligence; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Data processing. List processing. Character string processing; Exact sciences and technology; Information systems. Data bases; Knowledge acquisition; Large-scale; MapReduce; Memory organisation. Data processing; Rough sets; Software; Knowledge acquisition; Large-scale; Rough sets; MapReduce; High performance; Data analysis; Scalability; Information rate; Data transmission; Very large databases; Knowledge base; Data mining; Distributed computing; Minimum time; Rough set theory; Experimental result; Scientific research; Massive parallelism; Knowledge discovery; Large scale; Transmission speed
• ISSN: 0888-613X; 1873-4731
• DOI: 10.1016/j.ijar.2013.08.003

Nowadays, with the volume of data growing at an unprecedented rate, large-scale data mining and knowledge discovery have become a new challenge. Rough set theory for knowledge acquisition has been successfully applied in data mining. The recently introduced MapReduce technique has received much attention from both scientific community and industry for its applicability in big data analysis. To mine knowledge from big data, we present parallel large-scale rough set based methods for knowledge acquisition using MapReduce in this paper. We implemented them on several representative MapReduce runtime systems: Hadoop, Phoenix and Twister. Performance comparisons on these runtime systems are reported in this paper. The experimental results show that (1) The computational time is mostly minimum on Twister while employing the same cores; (2) Hadoop has the best speedup for larger data sets; (3) Phoenix has the best speedup for smaller data sets. The excellent speedups also demonstrate that the proposed parallel methods can effectively process very large data on different runtime systems. Pitfalls and advantages of these runtime systems are also illustrated through our experiments, which are helpful for users to decide which runtime system should be used in their applications.