Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Simulation Environment and Experimental Process
Related Work
Conclusions and Future Work
References
All Topics
Engineering

Failure detector abstractions for MapReduce-based systems

Profile image of Maria Alejandra Sarmiento PerezMaria Alejandra Sarmiento Perez

2017, Information Sciences

https://doi.org/10.1016/J.INS.2016.08.013
visibility

…

description

16 pages

link

1 file

Abstract

Omission failures represent an important source of problems in data-intensive computing systems. In these frameworks, omission failures are caused by slow tasks, known as stragglers , which can strongly jeopardize the workload performance. In the case of MapReduce-based systems, many state-of-the-art approaches have preferred to explore and extend speculative execution mechanisms. Other alternatives have based their contributions in doubling the computing resources for their tasks. Nevertheless, none of these approaches has addressed a fundamental aspect related to the detection and further solving of the omission failures, that is, the timeout service adjustment. In this paper, we have studied the omission failures in MapReduce systems, formalizing their failure detector abstraction by means of three different algorithms for defining the timeout. The first abstraction, called High Relax Failure Detector (HR-FD), acts as a static alternative to the default timeout, which is able to estimate the completion time for the user workload. The second abstraction, called Medium Relax Failure Detector (MR-FD), dynamically modifies the timeout, according to the progress score of each workload. Finally, taking into account that some of the user requests are strictly deadline-bounded, we have introduced the third abstraction, called Low Relax Failure Detector (LR-FD), which is able to merge the MapReduce dynamic timeout with an external monitoring system, in order to enforce more accurate failure detections. Whereas HR-FD shows performance improvements for most of the user request (in particular , small workloads), MR-FD and LR-FD enhance significantly the current timeout selection , for any kind of scenario, regardless of the workload type and failure injection time.

Related papers

Experimental Analysis in Hadoop MapReduce: A Closer Look at Fault Detection and Recovery Techniques
Hazrina Sofian

Sensors

Hadoop MapReduce reactively detects and recovers faults after they occur based on the static heartbeat detection and the re-execution from scratch techniques. However, these techniques lead to excessive response time penalties and inefficient resource consumption during detection and recovery. Existing fault-tolerance solutions intend to mitigate the limitations without considering critical conditions such as fail-slow faults, the impact of faults at various infrastructure levels and the relationship between the detection and recovery stages. This paper analyses the response time under two main conditions: fail-stop and fail-slow, when they manifest with node, service, and the task at runtime. In addition, we focus on the relationship between the time for detecting and recovering faults. The experimental analysis is conducted on a real Hadoop cluster comprising MapReduce, YARN and HDFS frameworks. Our analysis shows that the recovery of a single fault leads to an average of 67.6% re...

View PDFchevron_right
Fault Tolerance in MapReduce: A Survey
MARIA VALENTINA FLORES PEREZ

Computer Communications and Networks, 2016

Data-intensive computing has become one of the most popular forms of parallel computing. This is due to the explosion of digital data we are living. This data expansion has mainly come from three sources: (i) scientific experiments from fields such as astronomy, particle physics, or genomics; (ii) data from sensors; and (iii) citizens publications in channels such as social networks. Data-intensive computing systems, such as Hadoop MapReduce, have as main goal the processing of an enormous amount of data in a short time, by transmitting the computation where the data resides. In failure-free scenarios, these frameworks usually achieve good results. Given that failures are common at large scale, these frameworks exhibit some fault tolerance and dependability techniques as built-in features. In particular, MapReduce frameworks tolerate machine failures (crash failures) by re-executing all the tasks of the failed machine by the virtue of data replication. Furthermore, in order to mask temporary failures caused by network or machine overload (timing failure) where some tasks are performing relatively slower than other tasks, Hadoop relaunches other copies of these tasks on other machines.

View PDFchevron_right
Benchmarking Dependability of MapReduce Systems
Damián Serrano

2012 IEEE 31st Symposium on Reliable Distributed Systems, 2012

MapReduce is a popular programming model for distributed data processing. Extensive research has been conducted on the reliability of MapReduce, ranging from adaptive and on-demand fault-tolerance to new fault-tolerance models. However, realistic benchmarks are still missing to analyze and compare the effectiveness of these proposals. To date, most MapReduce fault-tolerance solutions have been evaluated using microbenchmarks in an ad-hoc and overly simplified setting, which may not be representative of real-world applications. This paper presents MRBS, a comprehensive benchmark suite for evaluating the dependability of MapReduce systems. MRBS includes five benchmarks covering several application domains and a wide range of execution scenarios such as data-intensive vs. compute-intensive applications, or batch applications vs. online interactive applications. MRBS allows to inject various types of faults at different rates. It also considers different application workloads and dataloads, and produces extensive reliability, availability and performance statistics. We illustrate the use of MRBS with Hadoop clusters running on Amazon EC2, and on a private cloud.

View PDFchevron_right
Medusa: An Efficient Cloud Fault-Tolerant MapReduce
Miguel Correia

2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), 2016

Applications such as web search and social networking have been moving from centralized to decentralized cloud architectures to improve their scalability. MapReduce, a programming framework for processing large amounts of data using thousands of machines in a single cloud, also needs to be scaled out to multiple clouds to adapt to this evolution. The challenge of building a multi-cloud distributed architecture is substantial. Notwithstanding, the ability to deal with the new types of faults introduced by such setting, such as the outage of a whole datacenter or an arbitrary fault caused by a malicious cloud insider, increases the endeavor considerably. In this paper we propose Medusa, a platform that allows MapReduce computations to scale out to multiple clouds and tolerate several types of faults. Our solution fulfills four objectives. First, it is transparent to the user, who writes her typical MapReduce application without modification. Second, it does not require any modification to the widely used Hadoop framework. Third, the proposed system goes well beyond the fault-tolerance offered by MapReduce to tolerate arbitrary faults, cloud outages, and even malicious faults caused by corrupt cloud insiders. Fourth, it achieves this increased level of fault tolerance at reasonable cost. We performed an extensive experimental evaluation in the ExoGENI testbed, demonstrating that our solution significantly reduces execution time when compared to traditional methods that achieve the same level of resilience.

View PDFchevron_right

References (38)

  1. Apache Hadoop NextGen MapReduce (YARN), (2015). http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html .
  2. The Apache Hadoop Project, (2015). http://hadoop.apache.org/ .
  3. G. Ananthanarayanan, S. Agarwal, S. Kandula, A. Greenberg, I. Stoica, D. Harlan, E. Harris, Scarlett: coping with skewed content popularity in mapreduce clusters, in: Proceedings of the Sixth Conference on Computer Systems, EuroSys '11, ACM, New York, NY, USA, 2011, pp. 287-300, doi: 10.1145/1966445. 196 6 472 .
  4. G. Ananthanarayanan, A. Ghodsi, S. Shenker, I. Stoica, Effective straggler mitigation: attack of the clones, in: Proceedings of the 10th USENIX Conference on Networked Systems Design and Implementation, nsdi'13, USENIX Association, Berkeley, CA, USA, 2013, pp. 185-198 . http://dl.acm.org/citation.cfm? id=2482626.2482645 .
  5. G. Ananthanarayanan, A. Ghodsi, A. Wang, D. Borthakur, S. Kandula, S. Shenker, I. Stoica, PACMan: coordinated memory caching for parallel jobs, in: Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, NSDI'12, USENIX Association, Berkeley, CA, USA, 2012 . 20-20. http://dl.acm.org/citation.cfm?id=2228298.2228326 .
  6. G. Ananthanarayanan, M.C.-C. Hung, X. Ren, I. Stoica, A. Wierman, M. Yu, GRASS: trimming stragglers in approximation analytics, in: Proceedings of the 11 t h USENIX Conference on Networked Systems Design and Implementation, NSDI'14, USENIX Association, Berkeley, CA , USA , 2014, pp. 289-302 . http://dl.acm.org/citation.cfm?id=2616448.2616475 .
  7. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, E. Harris, Reining in the outliers in map-reduce clusters using mantri, in: Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation, OSDI'10, USENIX Association, Berkeley, CA , USA , 2010, pp. 1-16 . http://dl.acm.org/citation.cfm?id=1924943.1924962 .
  8. R. Appuswamy, C. Gkantsidis, D. Narayanan, O. Hodson, A. Rowstron, Scale-up vs scale-out for Hadoop: time to rethink? in: Proceedings of the 4th Annual Symposium on Cloud Computing, SOCC '13, ACM, New York, NY, USA, 2013, pp. 20:1-20:13, doi: 10.1145/2523616.2523629 .
  9. C. Cachin , R. Guerraoui , L. Rodrigues , Introduction to Reliable and Secure Distributed Programming (2. ed.), Springer, 2011 .
  10. T.D. Chandra, S. Toueg, Unreliable failure detectors for reliable distributed systems, J. ACM 43 (1996) 225-267, doi: 10.1145/226643.226647 .
  11. Q. Chen , C. Liu , Z. Xiao , Improving mapreduce performance using smart speculative execution strategy, Comput. IEEE Trans. 63 (4) (2014) 954-967 .
  12. A. Clement, M. Kapritsos, S. Lee, Y. Wang, L. Alvisi, M. Dahlin, T. Riche, Upright cluster services, in: Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, SOSP '09, ACM, New York , NY, USA, 2009, pp. 277-290, doi: 10.1145/1629575.1629602 .
  13. T. Condie, N. Conway, P. Alvaro, J.M. Hellerstein, K. Elmeleegy, R. Sears, Mapreduce online, in: Proceedings of the 7th USENIX Conference on Networked Systems Design and Implementation, NSDI'10, USENIX Association, Berkeley, CA, USA, 2010 . 21-21. http://dl.acm.org/citation.cfm?id=1855711.1855732 .
  14. P. Costa, M. Pasin, A. Bessani, M. Correia, Byzantine fault-tolerant mapreduce: Faults are not just crashes, in: Proceedings of the 3rd IEEE Second International Conference on Cloud Computing Technology and Science, CLOUDCOM '11, IEEE Computer Society, Washington, DC, USA, 2010, pp. 17-24, doi: 10.1109/CloudCom.2010.25 .
  15. J. Dean , S. Ghemawat , Mapreduce: simplified data processing on large clusters, Commun. ACM 51 (1) (2008) 107-113 .
  16. J. Dean , S. Ghemawat , G. Inc , Mapreduce: simplified data processing on large clusters, in: Proceedings of the 6th Conference on Symposium on Operating Systems Design & Implementation, OSDI'04, USENIX Association, 2004 .
  17. F. Dinu , T.E. Ng , Understanding the effects and implications of compute node related failures in Hadoop, in: HPDC '12: Proceedings of the 21st Inter- national Symposium on High-Performance Parallel and Distributed Computing, ACM, New York, NY, USA, 2012, pp. 187-198 .
  18. F. Dinu , T.S.E. Ng , Hadoop's overload tolerant design exacerbates failure detection and recovery, in: Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation, NetDB'11, ACM, New York , NY, USA, 2011, pp. 1-7 .
  19. K. Elmeleegy, Piranha: optimizing short jobs in Hadoop, Proc. VLDB Endow. 6 (11) (2013) 985-996 . http://dl.acm.org/citation.cfm?id=2536222.2536225 .
  20. F.C. Freiling, R. Guerraoui, P. Kuznetsov, The failure detector abstraction, ACM Comput. Surv. 43 (2011) 9:1-9:40, doi: 10.1145/1883612.1883616 .
  21. M. Isard, M. Budiu, Y. Yu, A. Birrell, D. Fetterly, Dryad: distributed data-parallel programs from sequential building blocks, in: Proceedings of the 2nd ACM SIGOPS/EuroSys 2007, EuroSys '07, ACM, New York, NY, USA, 2007, pp. 59-72, doi: 10.1145/1272996.1273005 .
  22. J. Kephart , D. Chess , The vision of autonomic computing, Computer 36 (1) (2003) 41-50 .
  23. S.Y. Ko, I. Hoque, B. Cho, I. Gupta, Making cloud intermediate data fault-tolerant, in: Proceedings of the 1st ACM symposium on Cloud computing, SoCC '10, ACM, New York, NY, USA, 2010, pp. 181-192, doi: 10.1145/1807128.1807160 .
  24. B. Memishi , S. Ibrahim , M.S. Pérez , G. Antoniu , On the dynamic shifting of the mapreduce timeout, in: R. Kannan, R.U. Rasool, H. Jin, S. Balasundaram (Eds.), Handbook of Research on Managing and Processing Big Data in Cloud Computing, IGI Global, Hershey, Pennsylvania (USA), 2016 .
  25. B. Memishi, M.S. Pérez, G. Antoniu, Diarchy: an optimized management approach for mapreduce masters, Procedia Comput. Sci. 51 (2015) 9-18 . International Conference On Computational Science, {ICCS} 2015 Computational Science at the Gates of Nature. http://www.sciencedirect.com/science/ article/pii/S1877050915009874 .
  26. G. Mone, Beyond hadoop, Commun. ACM 56 (1) (2013) 22-24, doi: 10.1145/2398356.2398364 .
  27. J. Montes, A. Sánchez, B. Memishi, M.S. Pérez, G. Antoniu, GMone: a complete approach to cloud monitoring, Future Gener. Comput. Syst. 29 (8) (2013) 2026-2040, doi: 10.1016/j.future.2013.02.011 .
  28. K. Morton, M. Balazinska, D. Grossman, ParaTimer: A Progress Indicator for MapReduce DAGs, in: Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data, SIGMOD '10, ACM, New York, NY, USA, 2010, pp. 507-518, doi: 10.1145/1807167.1807223 .
  29. K. Morton , A. Friesen , M. Balazinska , D. Grossman , Estimating the progress of mapreduce pipelines, in: Data Engineering (ICDE), 2010 IEEE 26th International Conference on, 2010, pp. 6 81-6 84 .
  30. M. Nami , K. Bertels , A survey of autonomic computing systems, in: Autonomic and Autonomous Systems, 2007. ICAS07. Third International Conference on, 2007 . 26-26.
  31. J.S. Plank, M. Allen, R. Wolski, The effect of timeout prediction and selection on wide area collective operations, in: Proceedings of the IEEE Inter- national Symposium on Network Computing and Applications (NCA'01), NCA '01, IEEE Computer Society, Washington, DC, USA, 2001, pp. 320-329 . http://dl.acm.org/citation.cfm?id=580585.883098 .
  32. A. Sánchez, J. Montes, M.S. Pérez, T. Cortes, An autonomic framework for enhancing the quality of data grid services, Future Generation Comp. Syst. 28 (7) (2012) 1005-1016, doi: 10.1016/j.future.2011.08.016 .
  33. T. White , Hadoop -The Definitive Guide: Storage and Analysis at Internet Scale (3. ed., revised and updated), O'Reilly, 2012 .
  34. H. Xu , W.C. Lau , Speculative execution for a single job in a mapreduce-like system, in: Cloud Computing (CLOUD), 2014 IEEE 7th International Confer- ence on, 2014, pp. 586-593 .
  35. H. Xu, W.C. Lau, Optimization for speculative execution in a mapreduce-like cluster, in: 2015 IEEE Conference on Computer Communications, INFOCOM 2015, Kowloon, Hong Kong, April 26 -May 1, 2015, 2015, pp. 1071-1079, doi: 10.1109/INFOCOM.2015.7218480 .
  36. M. Zaharia, D. Borthakur, J.S. Sarma, K. Elmeleegy, S. Shenker, I. Stoica, Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling, in: Proceedings of the 5th European Conference on Computer Systems, EuroSys '10, ACM, New York, NY, USA, 2010, pp. 265-278, doi: 10.1145/1755913.1755940 .
  37. M. Zaharia, A. Konwinski, A.D. Joseph, R. Katz, I. Stoica, Improving mapreduce performance in heterogeneous environments, in: Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI'08, USENIX Association, Berkeley, CA , USA , 2008, pp. 29-42 . http: //dl.acm.org/citation.cfm?id=1855741.1855744 .
  38. H. Zhu , C. Haopeng , Adaptive failure detection via heartbeat under Hadoop, in: Proceedings of the 2011 IEEE Asia-Pacific Services Computing Confer- ence, ApSCC'11, IEEE, New York, NY, USA, 2011, pp. 231-238 .

Related papers

RAFT at work: speeding-up mapreduce applications under task and node failures
Jorge-Arnulfo Quiané-Ruiz

2011

The MapReduce framework is typically deployed on very large computing clusters where task and node failures are no longer an exception but the rule. Thus, fault-tolerance is an important aspect for the efficient operation of MapReduce jobs. However, currently MapReduce implementations fully recompute failed tasks (subparts of a job) from the beginning. This can significantly decrease the runtime performance of

View PDFchevron_right
Chronos: Failure-aware scheduling in shared Hadoop clusters
tran anh phuong

2015 IEEE International Conference on Big Data (Big Data), 2015

Hadoop emerged as the de facto state-of-the-art system for MapReduce-based data analytics. The reliability of Hadoop systems depends in part on how well they handle failures. Currently, Hadoop handles machine failures by re-executing all the tasks of the failed machines (i.e., executing recovery tasks). Unfortunately, this elegant solution is entirely entrusted to the core of Hadoop and hidden from Hadoop schedulers. The unawareness of failures therefore may prevent Hadoop schedulers from operating correctly towards meeting their objectives (e.g., fairness, job priority) and can significantly impact the performance of MapReduce applications. This paper presents Chronos, a failure-aware scheduling strategy that enables an early yet smart action for fast failure recovery while still operating within a specific scheduler objective. Upon failure detection, rather than waiting an uncertain amount of time to get resources for recovery tasks, Chronos leverages a lightweight preemption technique to carefully allocate these resources. In addition, Chronos considers data locality when scheduling recovery tasks to further improve the performance. We demonstrate the utility of Chronos by combining it with Fifo and Fair schedulers. The experimental results show that Chronos recovers to a correct scheduling behavior within a couple of seconds only and reduces the job completion times by up to 55% compared to state-of-the-art schedulers.

View PDFchevron_right
Deadline-based workload management for mapreduce environments: Pieces of the performance puzzle
Abhishek Verma

2012

Abstract Hadoop and the associated MapReduce paradigm, has become the de facto platform for cost-effective analytics over “Big Data”. There is an increasing number of MapReduce applications associated with live business intelligence that require completion time guarantees. In this work, we introduce and analyze a set of complementary mechanisms that enhance workload management decisions for processing MapReduce jobs with deadlines.

View PDFchevron_right
MRBS: Towards Dependability Benchmarking for Hadoop MapReduce
Damián Serrano

Lecture Notes in Computer Science, 2013

MapReduce is a popular programming model for distributed data processing. Extensive research has been conducted on the reliability of MapReduce, ranging from adaptive and on-demand fault-tolerance to new fault-tolerance models. However, realistic benchmarks are still missing to analyze and compare the effectiveness of these proposals. To date, most MapReduce fault-tolerance solutions have been evaluated using microbenchmarks in an ad-hoc and overly simplified setting, which may not be representative of real-world applications. This paper presents MRBS, a comprehensive benchmark suite for evaluating the dependability of MapReduce systems. MRBS includes five benchmarks covering several application domains and a wide range of execution scenarios such as data-intensive vs. compute-intensive applications, or batch applications vs. online interactive applications. MRBS allows to inject various types of faults at different rates and produces extensive reliability, availability and performance statistics. The paper illustrates the use of MRBS with Hadoop clusters.

View PDFchevron_right
Experience with benchmarking dependability and performance of MapReduce systems
Damián Serrano

Performance Evaluation, 2016

MapReduce provides a convenient means for distributed data processing and automatic parallel execution on clusters of machines. It has various applications and is used by several services featuring fault tolerance and scalability. Many studies investigated the dependability and performance of MapReduce, ranging from job scheduling to data placement and replication, adaptive and ondemand fault tolerance to new fault tolerance models. However, the ad-hoc and overly simplified setting used to evaluate most MapReduce fault tolerance and performance improvement solutions poses significant challenges to the analysis and comparison of the effectiveness of these solutions. The paper precisely addresses this issue and presents MRBS, a comprehensive benchmark suite for evaluating the dependability and performance of MapReduce systems. MRBS includes five benchmarks covering several application domains and a wide range of execution scenarios such as data-intensive vs. compute-intensive applications, or batch applications vs. online interactive applications. MRBS allows to inject various workloads, dataloads and faultloads, and produces extensive reliability, availability and performance statistics. We implemented the MRBS benchmark suite for Hadoop MapReduce, and we illustrate its use with various case studies running on Amazon EC2 and on a private cloud.

View PDFchevron_right

Related topics

  • Engineering
  • Computer Science

    • Cited by

    Experimental Analysis in Hadoop MapReduce: A Closer Look at Fault Detection and Recovery Techniques
    Hazrina Sofian

    Sensors

    Hadoop MapReduce reactively detects and recovers faults after they occur based on the static heartbeat detection and the re-execution from scratch techniques. However, these techniques lead to excessive response time penalties and inefficient resource consumption during detection and recovery. Existing fault-tolerance solutions intend to mitigate the limitations without considering critical conditions such as fail-slow faults, the impact of faults at various infrastructure levels and the relationship between the detection and recovery stages. This paper analyses the response time under two main conditions: fail-stop and fail-slow, when they manifest with node, service, and the task at runtime. In addition, we focus on the relationship between the time for detecting and recovering faults. The experimental analysis is conducted on a real Hadoop cluster comprising MapReduce, YARN and HDFS frameworks. Our analysis shows that the recovery of a single fault leads to an average of 67.6% re...

    View PDFchevron_right
    580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved