COMPARING THE PERFORMANCE OF APACHE SPARK AND APACHE HADOOP MAPREDUCE ON BIG DATA PROCESSING

Abstract

Imagine the massive volume of data in the world, and the rapid growth of it every moment and every second, these data that carry many useful values, which help companies to succeed and increase a competitive advantage, is called 'Big Data', due to its sheer Volume, Variety, Velocity and Veracity. Most of this data is unstructured, structured or semi structured. The large amounts of data created a need for new frameworks for processing. The “Apache Hadoop MapReduce" model is a framework for processing large-scale datasets with parallel and distributed algorithms. The “Apache Hadoop MapReduce“allows for the distributed processing of large data sets across clusters of computers using simple programming models. Recently a framework called Apache Spark has emerged, focused on micro-batch data processing. In addition the main feature of Spark is the in-memory computation. In this research, we perform a comparative study on the performance of these two frameworks. Additionally we use bigdatabench (tool) to load dataset up to 420 million records. Experimental results show that Spark has better performance and overall lower runtimes than Apache Hadoop MapReduce.