Academia.eduAcademia.edu

Outline

Title
Abstract
Data Integration
The Resilient Distributed Dataset in Spark
Care of Structured Data
Support for Intricate Analysis
References
All Topics
Computer Science
Data Science

A Close-Up View About Spark in Big Data Jurisdiction

Profile image of Nikhat AkhtarNikhat Akhtar

2018, International Journal of Engineering Research and Application (IJERA), ISSN : 2248-9622

https://doi.org/10.9790/9622-0801022641
visibility

…

description

16 pages

link

1 file

Abstract

The Big data is the name used ubiquitously now a day in distributed paradigm on the web. As the name point out it is the collection of sets of very large amounts of data in pet bytes, Exabyte etc. related systems as well as the algorithms used to analyze this enormous data. Hadoop technology as a big data processing technology has proven to be the go to solution for processing enormous data sets. MapReduce is a conspicuous solution for computations, which requirement one-pass to complete, but not exact efficient for use cases that need multi-pass for computations and algorithms. The Job output data between every stage has to be stored in the file system before the next stage can begin. Consequently, this method is slow, disk Input/output operations and due to replication. Additionally, Hadoop ecosystem doesn't have every component to ending a big data use case. Suppose we want to do an iterative job, you would have to stitch together a sequence of MapReduce jobs and execute them in sequence. Every this job has high-latency, and each depends upon the completion of the previous stage. Apache Spark is one of the most widely used open source processing engines for big data, with wealthy language-integrated APIs and an extensive range of libraries. Apache Spark is a usual framework for distributed computing that offers high performance for both batch and interactive processing. In this paper, we aimed to demonstrate a close-up view about Apache Spark and its features and working with Spark using Hadoop. We are in a nutshell discussing about the Resilient Distributed Datasets (RDD), RDD operations, features, and limitation. Spark can be used along with MapReduce in the same Hadoop cluster or can be used lonely as a processing framework. In the last comparative analysis between Spark and Hadoop and MapReduce in this paper.

Related papers

Big data analytics on Apache Spark
Joshua Zhexue Huang

International Journal of Data Science and Analytics, 2016

Apache Spark has emerged as the de facto framework for big data analytics with its advanced in-memory programming model and upper-level libraries for scalable machine learning, graph analysis, streaming and structured data processing. It is a general-purpose cluster computing framework with language-integrated APIs in Scala, Java, Python and R. As a rapidly evolving open source project, with an increasing number of contributors from both academia and industry, it is difficult for researchers to comprehend the full body of development and research behind Apache Spark, especially those who are beginners in this area. In this paper, we present a technical review on big data analytics using Apache Spark. This review focuses on the key components, abstractions and features of Apache Spark. More specifically, it shows what Apache Spark has for designing and implementing big data algorithms and pipelines for machine learning, graph analysis and stream processing. In addition, we highlight some research and development directions on Apache Spark for big data analytics.

View PDFchevron_right
Survey on Frameworks for Distributed Computing: Hadoop, Spark and Storm
Telmo Morais

The storage and management of information has always been a challenge for software engineering, new programing approaches had to be found, parallel processing and then distributed computing programing models were developed, and new programing frameworks were developed to assist software developers. This is where Hadoop framework, an open source implementation of MapReduce programing model, that also takes advantage of a distributed file system, takes its lead, but in the meantime, since its presentation, there were evolutions to the MapReduce and new programing models that were introduced by Spark and Storm frameworks, that show promising results.

View PDFchevron_right
Consideration of parallel data processing over an apache spark cluster
Masato Oguchi

2017 IEEE International Conference on Big Data (Big Data), 2017

View PDFchevron_right
discusses an efficient way of analyzing Big Data stored in Hadoop Distributed File System HDFS using Apache Spark framework, and its advantages over Hadoop MapReduce framework
milind betkar

With the size of data increasing each day, the traditional methods of data processing have become inefficient and time consuming. Today, Facebook, Google, Twitter are generating Petabytes of data each day. This large amount of data is given the term 'Big Data'. To overcome this inefficiency, the processing of Data can be performed using Apache spark. Apache

View PDFchevron_right
Techniques Management Big data Apache Hadoop and Apache Spark and which is better In structuring and processing data
ahmed altriki

University of Benghazi, 2020

Abstract: Newly considered both Big data Apache Hadoop and Apache Spark It is an important techniques in managing huge databases and thus this article provides an overview of Hadoop MapReduce. Apache Spark is used in big data increase the number of users of social networking sites of various types or industries. Big Data can be classified in the form of structured, unstructured and semi-structured form. Traditional data processing techniques are unable to store and process this huge amount of data, and due to this, they face many challenges in processing Big Data and demands of the end user. This paper also shows a comparison of certain criteria to determine which is better accordingly. Keywords: Big Data; Hadoop; Map Reduce; Spark.

View PDFchevron_right
Clash of the Titans: MapReduce vs. Spark for Large Scale Data Analytics
臻轩 潘

MapReduce and Spark are two very popular open source cluster computing frameworks for large scale data analytics. These frameworks hide the complexity of task parallelism and fault-tolerance, by exposing a simple programming API to users. In this paper, we evaluate the major architectural components in MapReduce and Spark frameworks including: shuffle, execution model, and caching, by using a set of important analytic workloads. To conduct a detailed analysis, we developed two profiling tools: (1) We correlate the task execution plan with the resource utilization for both MapReduce and Spark, and visually present this correlation; (2) We provide a breakdown of the task execution time for in-depth analysis. Through detailed experiments, we quantify the performance differences between MapReduce and Spark. Furthermore, we attribute these performance differences to different components which are architected differently in the two frameworks. We further expose the source of these performance differences by using a set of micro-benchmark experiments. Overall, our experiments show that Spark is about 2.5x, 5x, and 5x faster than MapReduce, for Word Count, k-means, and PageRank, respectively. The main causes of these speedups are the efficiency of the hash-based aggre-gation component for combine, as well as reduced CPU and disk overheads due to RDD caching in Spark. An exception to this is the Sort workload, for which MapReduce is 2x faster than Spark. We show that MapReduce's execution model is more efficient for shuffling data than Spark, thus making Sort run faster on MapRe-duce.

View PDFchevron_right
Big Data Processing with Hadoop-MapReduce in Cloud Systems
BIEN LE ANH HUNG

Today, we " re surrounded by data like oxygen. The exponential growth of data first presented challenges to cutting-edge businesses such as Google, Yahoo, Amazon, Microsoft, Facebook, Twitter etc. Data volumes to be processed by cloud applications are growing much faster than computing power. This growth demands new strategies for processing and analyzing information. Hadoop-MapReduce has become a powerful Computation Model addresses to these problems. Hadoop HDFS became more popular amongst all the Big Data tools as it is open source with flexible scalability, less total cost of ownership & allows data stores of any form without the need to have data types or schemas defined. Hadoop MapReduce is a programming model and software framework for writing applications that rapidly process vast amounts of data in parallel on large clusters of compute nodes. In this paper I have provided an overview, architecture and components of Hadoop, HCFS (Hadoop Cluster File System) and MapReduce programming model, its various applications and implementations in Cloud Environments.

View PDFchevron_right
Big Data Analysis Using Apache Spark MLlib and Hadoop HDFS with Scala and Java
Alaa Jumaa

Kurdistan Journal of Applied Research, 2019

Nowadays with the technology revolution the term of big data is a phenomenon of the decade moreover, it has a significant impact on our applied science trends. Exploring well big data tool is a necessary demand presently. Hadoop is a good big data analyzing technology, but it is slow because the Job result among each phase must be stored before the following phase is started as well as to the replication delays. Apache Spark is another tool that developed and established to be the real model for analyzing big data with its innovative processing framework inside the memory and high-level programming libraries for machine learning, efficient data treating and etc. In this paper, some comparisons are presented about the time performance evaluation among Scala and Java in apache spark MLlib. Many tests have been done in supervised and unsupervised machine learning methods with utilizing big datasets. However, loading the datasets from Hadoop HDFS as well as to the local disk to identify...

View PDFchevron_right
Big Data Analysis: Apache Spark Perspective
Dr. Tariq Rahim Soomro

Big Data have gained enormous attention in recent years. Analyzing big data is very common requirement today and such requirements become nightmare when analyzing of bulk data source such as twitter twits are done, it is really a big challenge to analyze the bulk amount of twits to get relevance and different patterns of information on timely manner. This paper will explore the concept of Big Data Analysis and recognize some meaningful information from some sample big data source, such as Twitter twits, using one of industries emerging tool, known as Spark by Apache

View PDFchevron_right
Distributed big data analysis using Spark parallel data processing
beei iaes

Bulletin of Electrical Engineering and Informatics, 2022

Nowadays, the big data marketplace is rising rapidly. The big challenge is finding a system that can store and handle a huge size of data and then processing that huge data for mining the hidden knowledge. This paper proposed a comprehensive system that is used for improving big data analysis performance. It contains a fast big data processing engine using Apache Spark and a big data storage environment using Apache Hadoop. The system tests about 11 Gigabytes of text data which are collected from multiple sources for sentiment analysis. Three different machine learning (ML) algorithms are used in this system which is already supported by the Spark ML package. The system programs were written in Java and Scala programming languages and the constructed model consists of the classification algorithms as well as the pre-processing steps in a figure of ML pipeline. The proposed system was implemented in both central and distributed data processing. Moreover, some datasets manipulation manners have been applied in the system tests to check which manner provides the best accuracy and time performance. The results showed that the system works efficiently for treating big data, it gains excellent accuracy with fast execution time especially in the distributed data nodes.

View PDFchevron_right

References (41)

  1. SamanSarraf, Mehdi Ostadhashem, "Big data application in functional magnetic resonance imaging using apache spark", 2016 Future Technologies Conference (FTC), San Francisco, CA, USA, Pages: 281 -284, Year: 2016, DOI: 10.1109/FTC.2016.7821623
  2. Dr. Yusuf Perwej, "An Experiential Study of the Big Data," for published in the International Transaction of Electrical and Computer Engineers System (ITECES), USA, ISSN (Print): 2373-1273 ISSN (Online): 2373- 1281, Vol. 4, No. 1, page 14-25, March 2017, DOI:10.12691/iteces-4-1-3.
  3. J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," in Proceedings of the 6th Conference on Symposium on Opearting Systems Design & Implementation -Volume 6, 2004, p. 10.
  4. Apache Spark, "Apache Spark-lightning-fast cluster computing," 2016, accessed 19- February-2016. [Online]. Available: http://spark.apache.org
  5. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, "Spark: cluster computing with working sets," in Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing (HotCloud'10), USENIX Association, Berkeley, CA, 2010, p. 10-10.
  6. H. Karau, A. Konwinski, P. Wendell, and M. Zaharia, Learning Spark. Sebastopol, CA: O'Reilly Media, 2015.
  7. NikhatAkhtar, FirojParwej, Dr. Yusuf Perwej, "A Perusal Of Big Data Classification And Hadoop Technology," for published in the International Transaction of Electrical and Computer Engineers System (ITECES), USA, ISSN (Print): 2373-1273 ISSN (Online): 2373- 1281, Vol. 4, No. 1, page 26-38, May 2017, DOI: 10.12691/iteces-4-1-4.
  8. N. Islam, S. Sharmin, M. Wasi-ur-Rahman, X. Lu, D. Shankar, D. K. Panda, "Performance characterization and acceleration of in- memory file systems for Hadoop and Spark applications on HPC clusters," in 2015 IEEE International Conference on Big Data (Big Data), October 29, 2015-November 1, 2015, pp. 243-252.
  9. X. Lin, P. Wang, and B. Wu, "Log analysis in cloud computing environment with Hadoop and Spark," in 2013 5th IEEE International Conference on Broadband Network & Multimedia Technology (IC-BNMT), November 1
  10. L. Gu and H. Li, "Memory or time: performance evaluation for iterative operation on Hadoop and Spark," in 2013 IEEE 10th International Conference on High Performance Comput. and Comm.& 2013 IEEE International Conference on Embedded and Ubiquitous Computing (HPCC_EUC), November 13-15, 2013, pp. 721-727.
  11. K. Wang and M. M. H. Khan, "Performance prediction for Apache Spark platform," in 2015 IEEE 12th International Conference on Embedded Software and Systems (ICESS), 2015 IEEE 17th International Conference on High Performance Computing and Communications (HPCC), 2015 IEEE 7th International Symposium on Cyberspace Safety and Security (CSS), August 24-26, 2015, pp. 166-173.
  12. Tim Kraska, AmeetTalwalkar, John Duchi, ReanGri_th, Michael Franklin, and Michael Jordan.MLbase: A Distributed Machine- learning System. In Conference on Innovative Data Systems Research, 2013.
  13. XiangruiMeng, Joseph Bradley, Evan Sparks, and ShivaramVenkataraman. Ml pipelines: A new high-level api for MLlib. https://databricks.com/?p=2473, 2015.
  14. Armbrust, R. S. Xin, C. Lian, Y. Huai, D. Liu, J. K. Bradley, X. Meng, T. Kaftan, M. J. Franklin, A. Ghodsi et al., "Spark sql: Relational data processing in spark", Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data., ACM, pp. 1383-1394, 2015.
  15. N. Chaimov, A. Malony, S. Canon, C. Iancu, K. Z. Ibrahim, J. Srinivasan, "Scaling Spark on HPC Systems", Proceedings of the 25th ACM International Symposium on High- Performance Parallel and Distributed Computing, 2016.
  16. New directions for Apache Spark in 2015," http://www.slideshare.net/databricks/new- directions-for-apache-spark-in-2015.
  17. "Apache Spark-Lightning-Fast Cluster Computing", 2016, [online] Available: http://spark.apache.org.
  18. J. Liu, Y. Liang, C. Fang, and N. Ansari, "Spark-based Large-scale Matrix Inversion for Big Data Processing," IEEE INFOCOM Workshop of Big Data Sciences, Technologies, and Applications (BDSTA) ,accepted, 2016.
  19. Omar Backhoff, EiriniNtoutsi,"Scalable Online-Offline Stream Clustering in Apache Spark", Data Mining Workshops (ICDMW), 2016 IEEE 16th International Conference on, Barcelona, Spain ,12-15 Dec. 2016.
  20. DOI: 10.1109/ICDMW.2016.0014
  21. David Siegal ,JiaGuo ,G. Agrawal," Smart- MLlib: A High-Performance Machine- Learning Library",Cluster Computing (CLUSTER), 2016 IEEE International Conference on, Taipei, Taiwan ,12-16 Sept. 2016DOI: 10.1109/CLUSTER.2016.49
  22. J. E. Gonzalez, R. S. Xin, A. Dave, D. Crankshaw, M. J. Franklin, I. Stoica, "Graphx: Graph processing in a distributed dataflow framework", Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation ser. OSDI'14, pp. 599-613, 2014.
  23. S. Das, Y. Sismanis, K. S. Beyer, R. Gemulla, P. J. Haas, and J. McPherson. Ricardo: integrating R and Hadoop. In
  24. SIGMOD 2010 , pages 987-998. ACM, 2010.
  25. L. Yejas, D. Oscar, W. Zhuang, and A. Pannu. Big R:Large-Scale Analytics on Hadoop Using R. InIEEE BigData 2014, pages 570-577.
  26. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, I. Stoica, "Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In- Memory Cluster Computing", Proceedings of the USENIX Conference on Networked Systems Design and Implementation '12), pp. 15-28, Apr. 2012.
  27. Teng-Sheng Moh,"DBSCAN on Resilient Distributed Datasets", High Performance Computing & Simulation (HPCS), 2015
  28. International Conference on, Amsterdam, Netherlands, 20-24 July 2015.
  29. Zixia Liu ,Hong Zhang, Liqiang Wang," Hierarchical Spark: A Multi-Cluster Big Data Computing Framework",Cloud Computing (CLOUD), 2017 IEEE 10th International Conference on, Honolulu, CA, USA, Electronic ISBN: 978-1-5386-1993-3 , 25-30 June 2017.
  30. Hamid Mushtaq, Zaid Al-Ars,"Cluster-based Apache Spark implementation of the GATK DNA analysis pipeline",Bioinformatics and Biomedicine (BIBM), 2015 IEEE International Conference on, Washington, DC, USA, 9-12
  31. Nov. 2015., DOI:10.1109/BIBM.2015.7359893
  32. Benjamin Hindman, Andy Konwinski, MateiZaharia, Ali Ghodsi, Anthony D. Joseph, Randy Katz, Scott Shenker, Ion Stoica, "Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center", University of California, Berkley, September 2010.
  33. Yusuf Perwej, BedineKerim, MohmedSirelkhtemAdrees, Osama E. Sheta, " An Empirical Exploration of the Yarn in Big Data" for published in the International Journal of Applied Information Systems (IJAIS), ISSN : 2249-0868 , Foundation of Computer Science FCS, New York, USA Volume 12 , No.9, page 19-29 , December 2017 DOI : 10.5120/ijais2017451730
  34. Nhan Nguyen, Mohammad MaifiHasan Khan, Yusuf Albayram, Kewen Wang, "Understanding the Influence of Configuration Settings: An Execution Model-Driven Framework for Apache Spark Platform", Cloud Computing (CLOUD) 2017 IEEE 10th International Conference on, pp. 802-807, 2017, ISSN 2159-6190.
  35. Kewen Wang, M. M.HasanKhan,"Performance Prediction for Apache Spark Platform",2015 IEEE 12th International Conferen on Embedded Software and Systems (ICESS), 2015 IEEE 17th International Conference on, New York, NY, USA, 24-26 Aug. 2015. DOI: 10.1109/HPCC-CSS-ICESS.2015.246
  36. Kai Hildebrandt, Fabian Panse, NiklasWilcke,"Large-Scale Data Pollution with Apache Spark",IEEE Transactions on Big Data, PP 1 -1, Issue: 99,Electronic ISSN: 2332-7790 , 09 January 2017DOI: 10.1109/TBDATA.2016.2637378
  37. YassirSamadi ,MostaphaZbakh ,Claude Tadonki ,"Comparative study between Hadoop and Spark based on Hibench benchmarks",Cloud Computing Technologies and Applications (CloudTech), 2016 2nd International Conference on, Marrakech, Morocco, 24-26 May 2016.DOI: 10.1109/CloudTech.2016.7847709
  38. IstvanSzegedi, "Apache Spark: a fast big data analytics engine", [online] Available: https://dzone.com/articles/apache-spark-fast- big-data.
  39. Juwei Shi , YunjieQiu, Umar FarooqMinhas , Limei Jiao , Chen Wang , Berthold Reinwald , and Fatma O ̈ zcan , "Clash of Titans: MapReduce vs. Spark for Large Scale Data Analytics", Proceedings of the VLDB Endowment, Vol. 8, No. 13 Copyright 2015 VLDB Endowment 2150 8097/15/09
  40. PolatoIvanilton, R é Reginaldo, Goldman Alfredo, Kon Fabio, "A comprehensive view of Hadoop research-A systematic literature review", Journal of Network and Computer Applications, vol. 46, pp. 1-25, November 2014.
  41. *. "A Close-Up View About Spark in Big Data Jurisdiction." International Journal of Engineering Research and Applications (IJERA), vol. 08, no. 01, 2018, pp. 26-41. International Journal of Engineering Research and Applications (IJERA) is UGC approved Journal with Sl. No. 4525, Journal no. 47088. Indexed in Cross Ref, Index Copernicus (ICV 80.82), NASA, Ads, Researcher Id Thomson Reuters, DOAJ.

Related papers

A Comparative Study on Hadoop MapReduce and Apache Spark Framework for Big Data Analytics
vandana vijay

International Journal of Research Publication and Reviews, 2024

In today internet world, due to the current advent of new technologies, mobile devices, and communication media like social networking sites, the amount of data generated every year is growing at a very high rate. The growth of this generated data is beyond our imagination. It is impossible to store these huge data sets in RDMBSs like MySQL, as there are no specific formats of the data and that can be in either text or image formats. It requires the need of technologies which can easily manage and process huge volumes of structured and unstructured data in real-time and can protect data privacy and security. Big data technologies like MapReduce, Apache Flume, and Apache Spark can capture, store, and analyze this huge amount of data in very efficient and less costly manner. Spark and MapReduce programming frameworks provide an effective open-source solution for managing and analyzing the Big Data. MapReduce is a high-performance distributed Big Data programming framework. It processes the data in batch processing environment. On the other hand, Apache Spark is a scalable distributed inmemory data processing engine. It processes the data in both batch and real time environment. It uses Resilient Distributed Datasets (RDD) and Directed Acyclic Graph (DAG) for data processing. In this paper, a review on Hadoop MapReduce and Apache Spark have been made by comparing them on various parameters like performance, streaming, fault tolerance, storage, language support, and reliability.

View PDFchevron_right
A REVIEW: MAPREDUCE AND SPARK FOR BIG DATA ANALYTICS
vaishali chauhan

In this paper we discuss the various challenges of Big Data and problem arises due to continuous explosion of data resulting from the likes of social media and other online sources to gain access to deeper analysis of their data. This paper discusses two of the comparison of Hadoop Map Reduce and the recently introduced Apache Spark – both of which provide a processing model for analyzing big data. Although both of these options are based on the concept of Big Data, their performance varies significantly based on the use case under implementation. Data growing at very high speed and is having very large volume. Presently, to assemble the large volume of dataset at lesser cost, storage technology and data collection has made it possible for any organization.

View PDFchevron_right
Performance Analysis of Distributed Computing Frameworks for Big Data Analytics: Hadoop Vs Spark
Sonali Agarwal

Computación y Sistemas, 2020

In the last one decade, the tremendous growth in data emphasizes big data storage and management issues with the highest priorities. For providing better support to software developers for dealing with big data problems, new programming platforms are continuously developing and Hadoop MapReduce is a big gamechanger followed by Spark, which sets the world of big data on fire with its processing speed and comfortable APIs. Hadoop framework emerged as a leading tool based on the MapReduce programming model with a distributed file system. Spark is on the other hand, recently developed big data analysis and management framework used to explore unlimited underlying features of Big Data. In this research work, a comparative analysis of Hadoop MapReduce and Spark has been presented based on working principle, performance, cost, ease of use, compatibility, data processing, failure tolerance, and security. Experimental analysis has been performed to observe the performance of Hadoop MapReduce and Spark for establishing their suitability under different constraints of the distributed computing environment.

View PDFchevron_right
HADOOP, MAP REDUCE AND SPARK
Evelyn Anyebe

In the early 2000s, there was an explosion in data generated, from the Internet to social networks, web servers, sensors and smart devices. The amount and kind of data being stored and transferred was changing but technologies available for data storage and analysis could not keep up with this explosion. This is big data and it was a problem. Hadoop, Map Reduce and Spark represent efforts by toward solving this problem.

View PDFchevron_right
Big Data Processing Using Spark in Cloud
Valentina Emilia Balas

Studies in Big Data, 2019

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

View PDFchevron_right

Related topics

  • MapReduce and Hadoop
  • Big Data Technologies
  • Application of Big Data
  • Spark Ecosystem
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved