Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Scope, Types of State, and Related Work
Scope
Related Work
Correlated Checkpointing
Characteristics System Main Mechanism Objective
Implementations of State and Limitations
Open Discussions
Integrating State Management Into Data Frameworks
Evaluating State Management Methods
Conclusion
References
All Topics
Computer Science
Information Systems

A Survey of State Management in Big Data Processing Systems

Profile image of Volker MarklVolker Markl

2017, arXiv (Cornell University)

https://doi.org/10.48550/ARXIV.1702.01596
visibility

…

description

25 pages

link

1 file

Abstract

The concept of state and its applications vary widely across big data processing systems. This is evident in both the research literature and existing systems, such as Apache Flink, Apache Heron, Apache Samza, Apache Spark, and Apache Storm. Given the pivotal role that state management plays, particularly, for iterative batch and stream processing, in this survey, we present examples of state as an enabler, discuss the alternative approaches used to handle and implement state, capture the many facets of state management, and highlight new research directions. Our aim is to provide insight into disparate state management techniques, motivate others to pursue research in this area, and draw attention to open problems.

Related papers

Towards Elastic Stream Processing: Patterns and Infrastructure
Kai-Uwe Sattler

Distributed, highly-parallel processing frameworks as Hadoop are deemed to be state-of-the-art for handling big data today. But they burden application developers with the task to manually implement program logic using lowlevel batch processing APIs. Thus, a movement can be observed that high-level languages are developed which allow to declaratively model dataflows that are automatically optimized and mapped to the batch-processing backends. However, most of these systems are based on programming models as MapReduce that provide elasticity and fault-tolerance in a natural manner since intermediate results are materialized and, therefore, processes can simply be restarted and scaled with partitioning input datasets. For continuous query processing on data streams, these concepts cannot be applied directly since it must be guaranteed that no data is lost when nodes fail. Usually, these long running queries contain operators that maintain state information which depends on the data that has already been processed and hence they cannot be restarted without information loss. This also is an issue when streaming tasks should be scaled. Therefore, integrating elasticity and fault-tolerance in this context is a challenging task which is subject of this paper. We show how common patterns from parallel and distributed algorithms can be applied to tackle these problems and how they are mapped to the Mesos cluster management system.

View PDFchevron_right
Discretized Streams: A Fault-Tolerant Model for Scalable Stream Processing
Scott Shenker

2012

Many "big data" applications need to act on data arriving in real time. However, current programming models for distributed stream processing are relatively low-level, often leaving the user to worry about consistency of state across the system and fault recovery. Furthermore, the models that provide fault recovery do so in an expensive manner, requiring either hot replication or long recovery times. We propose a new programming model, discretized streams (D-Streams), that offers a high-level functional API, strong consistency, and efficient fault recovery. D-Streams support a new recovery mechanism that improves efficiency over the traditional replication and upstream backup schemes in streaming databasesparallel recovery of lost state-and unlike previous systems, also mitigate stragglers. We implement D-Streams as an extension to the Spark cluster computing engine that lets users seamlessly intermix streaming, batch and interactive queries. Our system can process over 60 million records/second at sub-second latency on 100 nodes.

View PDFchevron_right
Exploring Shared State in Key-Value Store for Window-Based Multi-pattern Streaming Analytics
MARIA VALENTINA FLORES PEREZ

2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)

We are now witnessing an unprecedented growth of data that needs to be processed at always increasing rates in order to extract valuable insights. Big Data streaming analytics tools have been developed to cope with the online dimension of data processing: they enable real-time handling of live data sources by means of stateful aggregations (operators). Current state-of-art frameworks (e.g. Apache Flink [1]) enable each operator to work in isolation by creating data copies, at the expense of increased memory utilization. In this paper, we explore the feasibility of deduplication techniques to address the challenge of reducing memory footprint for window-based stream processing without significant impact on performance. We design a deduplication method specifically for windowbased operators that rely on key-value stores to hold a shared state. We experiment with a synthetically generated workload while considering several deduplication scenarios and based on the results, we identify several potential areas of improvement. Our key finding is that more fine-grained interactions between streaming engines and (key-value) stores need to be designed in order to better respond to scenarios that have to overcome memory scarcity.

View PDFchevron_right
Apache Flink™: Stream and Batch Processing in a Single Engine
seif haridi

IEEE Data Eng. Bull., 2015

Modern enterprise applications are currently undergoing a complete paradigm shift away from traditional transactional processing to combined analytical and transactional processing. This challenge of combining two opposing query types in a single database management system results in additional requirements for transaction management as well. In this paper, we discuss our approach to achieve high throughput for transactional query processing while allowing concurrent analytical queries. We present our approach to distributed snapshot isolation and optimized two-phase commit protocols.

View PDFchevron_right
Towards Concurrent Stateful Stream Processing on Multicore Processors
Shuhao Zhang

2020 IEEE 36th International Conference on Data Engineering (ICDE), 2020

Recent data stream processing systems (DSPSs) can achieve excellent performance when processing large volumes of data under tight latency constraints. However, they sacrifice support for concurrent state access that eases the burden of developing stateful stream applications. Recently, some have proposed managing concurrent state access during stream processing by modeling state accesses as transactions. However, these are realized with locks involving serious contention overhead. Their coarse-grained processing paradigm further magnifies contention issues and tends to poorly utilize modern multicore architectures. This paper introduces TStream, a novel DSPS supporting efficient concurrent state access on multicore processors. Transactional semantics is employed like previous work, but scalability is greatly improved due to two novel designs: 1) dual-mode scheduling, which exposes more parallelism opportunities, 2) dynamic restructuring execution, which aggressively exploits the parallelism opportunities from dual-mode scheduling without centralized lock contentions. To validate our proposal, we evaluate TStream with a benchmark of four applications on a modern multicore machine. The experimental results show that 1) TStream achieves up to 4.8 times higher throughput with similar processing latency compared to the state-of-the-art and 2) unlike prior solutions, TStream is highly tolerant of varying application workloads such as key skewness and multi-partition state accesses.

View PDFchevron_right
Scalable State Management for Scientific Applications in the Cloud
Tonglin Li

The data generated by scientific simulations and experimental facilities is beginning to revolutionize the infrastructure support needed by these applications. The on-demand aspect and flexibility of cloud computing environments makes it an attractive platform for data-intensive scientific applications. However, cloud computing poses unique challenges for these applications. For example, cloud computing environments are heterogeneous, dynamic and non-persistent which can make reproducibility a challenge. The volume, velocity, variety, veracity and value of data combined with the characteristics of cloud environment make it important to track the state of execution data and application’s entire lifetime information to understand and ensure reproducibility. This paper proposes and implements a state management system (FRIEDA-State) for high-throughput and data-intensive scientific applications running in cloud environments. Our design addresses the challenges of state management in cloud environments and offers various configurations. Our implementation is built on top of FRIEDA (Flexible Robust Intelligent Elastic Data Management), a data management and execution framework for cloud environments. Our experiment results on two cloud test beds (FutureGrid and Amazon) show that the proposed solution has a minimal overhead (1.2ms/operation at a scale of 64 virtual machines) and is suitable for state management in cloud environments.

View PDFchevron_right
M3: Stream Processing on Main-Memory MapReduce
Ahmed Saad aly

2012 IEEE 28th International Conference on Data Engineering, 2012

The continuous growth of social web applications along with the development of sensor capabilities in electronic devices is creating countless opportunities to analyze the enormous amounts of data that is continuously steaming from these applications and devices. To process large scale data on large scale computing clusters, MapReduce has been introduced as a framework for parallel computing. However, most of the current implementations of the MapReduce framework support only the execution of fixed-input jobs. Such restriction makes these implementations inapplicable for most streaming applications, in which queries are continuous in nature, and input data streams are continuously received at high arrival rates. In this demonstration, we showcase M 3 , a prototype implementation of the MapReduce framework in which continuous queries over streams of data can be efficiently answered. M 3 extends Hadoop, the open source implementation of MapReduce, bypassing the Hadoop Distributed File System (HDFS) to support main-memory-only processing. Moreover, M 3 supports continuous execution of the Map and Reduce phases where individual Mappers and Reducers never terminate.

View PDFchevron_right
A Survey of Distributed Data Stream Processing Frameworks
Haruna Manu Isah

IEEE Access, 2019

Big data processing systems are evolving to be more stream oriented where each data record is processed as it arrives by distributed and low-latency computational frameworks on a continuous basis. As the stream processing technology matures and more organizations invest in digital transformations, new applications of stream analytics will be identified and implemented across a wide spectrum of industries. One of the challenges in developing a streaming analytics infrastructure is the difficulty in selecting the right stream processing framework for the different use cases. With a view to addressing this issue, in this paper we present a taxonomy, a comparative study of distributed data stream processing and analytics frameworks, and a critical review of representative open source (Storm, Spark Streaming, Flink, Kafka Streams) and commercial (IBM Streams) distributed data stream processing frameworks. The study also reports our ongoing study on a multilevel streaming analytics architecture that can serve as a guide for organizations and individuals planning to implement a real-time data stream processing and analytics framework. INDEX TERMS Dataflow architectures, data stream architectures, distributed processing systems comparison, survey, taxonomy.

View PDFchevron_right
A Comprehend The Apache Flink In Big Data Environments
Dr. Yusuf Perwej

IOSR Journal of Computer Engineering (IOSR-JCE), e-ISSN: 2278-0661, P-ISSN: 2278-8727, USA, Volume 20, Issue 1, Ver. IV, Page 48-58, 2018

Nowadays the amount of data on the web is persistently growing. Due to the enormous amount of data available on the World Wide Web, the processing cannot be done manually. As data have been primary resource, how to manage and make use of big data superior has attracted much consideration. In particular, with the development of the internet of things (IOT), the processing of the enormous amount of real-time data has become a significant challenge in research and applications. In this context, Apache Flink is an open-source system for processing streaming and batch data. The Flink is a big data processing framework that permits programmers to process the enormous amount of data in a very scalable and dexterous manner. TheFlink is a data processing tool that cancapableofhandlingbothstreaming data and batch data, providing suppleness and suddenness for users.The Flink provides a streaming data flow engine that conveys greatly high throughput surprisingly with low latency. The Flink is based upon a fault tolerant, which manages the delivery of data as well as communications within the cluster. In this paper, we are in a nutshell discussing about the Apache Flink along with some current research activities on top of the Apache Flink ecosystem,substitute to Hadoop MapReduce, Flink architectureanddata processing in Flink.In the last comparative analysis between Flink andSparkand Hadoop in this paper.

View PDFchevron_right
Cloud-based data stream processing
Leonardo Aniello

Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems - DEBS '14, 2014

In this tutorial we present the results of recent research about the cloud enablement of data streaming systems. We illustrate, based on both industrial as well as academic prototypes, new emerging uses cases and research trends. Specifically, we focus on novel approaches for (1) scalability and fault tolerance in large scale distributed streaming systems. In general, new fault tolerance mechanisms strive to be more robust and at the same time introduce less overhead. Novel load balancing approaches focus on elastic scaling over hundreds of instances based on the data and query workload. Finally, we present open challenges for the next generation of cloud-based data stream processing engines.

View PDFchevron_right

References (139)

  1. C. Aggarwal, J. Han, J. Wang, P. Yu. A Framework for Clustering Evolving Data Streams. In VLDB, pages 81-92, 2003.
  2. C. Aggarwal, P. Yu. A survey of synopsis construction in data streams. In Data Streams, Advances in Database Systems, vol. 31. Springer, New York, 2007.
  3. D. Agrawal et al. Road to Freedom in Big Data Analytics. In EDBT, pages 479-484, 2016.
  4. D. Agrawal et al. Rheem: Enabling Multi-Platform Task Execution. In SIGMOD, pages 2069-2072, 2016.
  5. Y. Ahmad, O. Kennedy, C. Koch, M. Nikolic. DBToaster: Higher-order Delta Processing for Dynamic, Frequently Fresh Views. In PVLDB, 5(10):968-979, 2012.
  6. A. Alexandrov et al. The Stratosphere platform for big data analytics. VLDB Journal, 23(6):939-964, 2014.
  7. A. Alexandrov et al. Implicit Parallelism through Deep Language Embedding. In SIGMOD, pages 47-61, 2015.
  8. R. Ananthanarayanan, et al. Photon: fault-tolerant and scalable joining of continuous data streams. In SIGMOD, pages 577- 588, 2013.
  9. A. Arasu, S. Babu, and J. Widom. The CQL Continuous Query Language: Semantic Foundations and Query Execution. VLDB Journal, 15(2):121-142, 2006.
  10. B. Bahmani, A. Chowdhury, and A. Goel. Fast incremental and personalized PageRank. In PVLDB, 4(3):173-184, 2010.
  11. B. Bahmani, B. Moseley, A. Vattani, R. Kumar, and S. Vassilvitskii. Scalable k-means++. In PVLDB, 5(7):622-633, 2012.
  12. O. Benjelloun, A. D. Sarma, A. Halevy, and J. Widom. ULDBs: Databases with Uncertainty and Lineage. In VLDB, pages 953-964, 2006.
  13. M. S. Bouguerra, D. Trystram, F. Wagner. Complexity Analysis of Checkpoint Scheduling with Variable Costs. IEEE Transactions on Computers, 62(6):1269-1275, 2013.
  14. O. Boykin, S. Ritchie, I. O'Connell, J. Lin. Summingbird: A Framework for Integrating Batch and Online MapReduce Computations. In PVLDB, 7(13):1441-1451, 2014.
  15. A. Brito, C. Fetzer, H. Sturzrehm, and P. Felber. Speculative Out-of-Order Event Processing with Software Transaction Memory. In DEBS, pages 265-275, 2008.
  16. A. Z. Broder, R. Lempel, F. Maghoul, and J. Pedersen. Efficient PageRank approximation via graph aggregation. Information Retrieval, 9(2):123-138, 2006.
  17. Y. Cai, P. G. Giarrusso, T. Rendel, and K. Ostermann. A theory of changes for higher-order languages: incrementalizing λ-calculi by static differentiation. In Programming Language Design and Implementation (PLDI), pages 145-155, 2014.
  18. P. Carbone, G. Fóra, S. Ewen, S. Haridi, K. Tzoumas. Lightweight Asynchronous Snapshots for Distributed Dataflows. The Computing Research Repository (CoRR), abs/1506.08603, 2015.
  19. P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, K. Tzoumas. Apache Flink™: Stream and Batch Processing in a Single Engine. IEEE Data Engineering Bulletin, 38(4):28- 38, 2015.
  20. Z. Chen, J. Dongarra. Highly Scalable Self-Healing Algorithms for High Performance Scientific Computing. IEEE Transactions on Computers, 58(11):1512-1524, 2009.
  21. R. Chitta, R. Jin, T. C. Havens, and A. K. Jain. Approximate kernel k-means: solution to large scale kernel clustering. In Knowledge discovery and data mining (KDD), pages 895-903, 2011.
  22. M. B. Cohen, S. Elder, C. Musco, and M. Persu. Dimensionality Reduction for k-means Clustering and Low Rank Approximation. In Symposium on Theory of Computing (STOC), pages 163-172, 2015.
  23. T. Condie, N. Conway, P. Alvaro, and J. M. Hellerstein. MapReduce online. In NSDI, 2010.
  24. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. Communication of ACM, 51(1):107-113, 2008.
  25. L. Ding, N. Mehta, E. A. Rundensteiner, G. T. Heineman. Joining Punctuated Streams. In EDBT, pages 587-604, 2004.
  26. J. Ding et al. Efficient Operator State Migration for Cloud- Based Data Stream Management Systems. The Computing Research Repository (CoRR), abs/1501.03619, 2016.
  27. S. Dudoladov, C. Xu, S. Schelter, A. Katsifodimos, S. Ewen, K. Tzoumas, V. Markl. Optimistic Recovery for Iterative Dataflows in Action. In SIGMOD, pages 1439-1443, 2015.
  28. C. Doulkeridis and K. Nørvåg. A Survey of Large-Scale Analytical Query Processing in MapReduce. VLDB Journal, 23(3):355-380, 2014.
  29. J. Ekanayake and G. Fox. High performance parallel computing with clouds and cloud technologies. In CloudComp, 2009.
  30. A. Elmore et al. A Demonstration of the BigDAWG Polystore System. In PVLDB, 8(12), 2015.
  31. S. Ewen, K. Tzoumas, M. Kaufmann, and V. Markl. Spinning fast iterative data flows. In PVLDB, 5(11):1268-1279, 2012.
  32. S. Ewen, S. Schelter, K. Tzoumas, D. Warneke, V. Markl. Iterative parallel data processing with stratosphere: an inside look. In SIGMOD, pages 1053-1056, 2013.
  33. L. Fegaras. Incremental Query Processing on Big Data Streams. TKDE, 2016.
  34. L. Fegaras. An Algebra for Distributed Big Data Analytics. Technical Report, 2016.
  35. Y.-H. Feng, et al. Efficient and Adaptive Stateful Replication for Stream Processing Engines in High-Availability Cluster. TPDS, 22(11):1788-1796, 2011.
  36. R. C. Fernandez, M. Migliavacca, E. Kalyvianaki, and P. Pietzuch. Integrating Scale Out and Fault Tolerance in Stream Processing using Operator State Management. In SIGMOD, 2013.
  37. R. C. Fernandez, M. Migliavacca, E. Kalyvianaki, and P. Pietzuch. Making State Explicit for Imperative Big Data Processing. In USENIX ATC, 2014.
  38. R. C. Fernandez, P. Garefalakis, P. Pietzuch. Java2SDG: Stateful Big Data Processing for the Masses. In ICDE, pages 1390-1393, 2016.
  39. Y. Fujiwara, M. Nakatsuji, M. Onizuka, and M. Kitsuregawa. Fast and exact top-k search for random walk with restart. In PVLDB, 5(5):442-453, 2012.
  40. Y. Fujiwara, M. Nakatsuji, T. Yamamuro, H. Shiokawa, and M. Onizuka. Efficient personalized pagerank with accuracy assurance. In Knowledge discovery and data mining (KDD), pages 15-23, 2012.
  41. Y. Fujiwara, M. Nakatsuji, H. Shiokawa, T. Mishima, and M. Onizuka. Fast and exact top-k algorithm for pagerank. In Conference on Artificial Intelligence (AAAI), pages 1106- 1112, 2013.
  42. M. Garofalakis, J. Gehrke, R. Rastogi. Querying and mining data streams: you only get one look (a tutorial). In SIGMOD, 2002.
  43. B. Gedik. Partitioning functions for stateful data parallelism in stream processing. VLDB Journal, 23(4):517-539, 2014.
  44. M. I. Gordon, W. Thies, and S. Amarasinghe. Exploiting Coarse-Grained Task, Data, and Pipeline Parallelism in Stream Programs. In Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 151-162, 2006.
  45. V. Gulisano, R. J.-Peris, M. P.-Martínez, C. Soriente, P. Valduriez. StreamCloud: An Elastic and Scalable Data Stream System. TPDS, 23(12):2351-2365, 2012.
  46. D. Hakkarinen and Z. Chen. Multilevel Diskless Checkpointing. IEEE Transactions on Computers, 62(4):772- 783, 2013.
  47. M. A. Hammer, J. Dunfield, K. Headley, N. Labich, J. S. Foster, M. Hicks, and D. V. Horn. 2015. Incremental computation with names. SIGPLAN, 50(10):748-766, 2015.
  48. M. Hirzel, R. Soulé, S. Schneider, B. Gedik, and R. Grimm. A catalog of stream processing optimizations. ACM Computing Surveys (CSUR), 46(4), 2014.
  49. J. H. Hwang, M. Balazinska, A. Rasin, U. Cetintemel, M. Stonebraker, S. Zdonik. High-availability algorithms for distributed stream processing. In ICDE, pages 779-790, 2005.
  50. J. H. Hwang, Y. Xing, U. Cetintemel, S. Zdonik. A Cooperative, Self-Configuring High-Availability Solution for Stream Processing. In ICDE, 2007.
  51. I. Jangjaimon and N.-F. Tzeng. Adaptive incremental checkpointing via delta compression for networked multicore systems. In IEEE IPDPS, pages 7-18, 2013.
  52. T. Johnson, S. Muthukrishnan, and I. Rozenbaum. Sampling algorithms in a stream operator. In SIGMOD, pages 1-12, 2005.
  53. T. Kanungo et al. A local search approximation algorithm for k-means clustering. In symposium on Computational geometry (SCG), pages 10-18, 2002.
  54. C. Koch. Incremental query evaluation in a ring of databases. In PODS, pages 87-98, 2010.
  55. C. Koch, Y. Ahmad, O. Kennedy, M. Nikolic, A. Nötzli, D. Lupei, A. Shaikhha. DBToaster: higher-order delta processing for dynamic, frequently fresh views. VLDB Journal, 23(2):253-278, 2014.
  56. C. Koch, D. Lupei, V. Tannen. Incremental View Maintenance for Collection Programming. In PODS, pages 75-90, 2016.
  57. Y. Kwon, M. Balazinska, and A. Greenberg. Fault-tolerant stream processing using a distributed, replicated file system. In PVLDB, 1(1):574-585, 2008.
  58. B. Koldehofe, R. Mayer, U. Ramachandran, K. Rothermel, M. Völz. Rollback-recovery without checkpoints in distributed event processing systems. In DEBS, pages 27-38. 2013.
  59. R. Kuntschke, B. Stegmaier, A. Kemper. Data Stream Sharing. Technical Report, TU Munich, 2005.
  60. H. G. Li, S. Chen, J. Tatemura, D. Agrawal, K. S. Candan, W. P. Hsiung. Safety Guarantee of Continuous Join Queries over Punctuated Data Streams. In VLDB, pages 19-30, 2006.
  61. J. Li, K. Tufte, V. Shkapenyuk, V. Papadimos, T. Johnson, and D. Maier. Out-of-order processing: a new architecture for high-performance stream systems. In PVLDB, 1(1):274-288, 2008.
  62. X. Lin, H. Lu, J. Xu, J.X. Yu. Continuously maintaining quantile summaries of the most recent N elements over a data stream. In ICDE, pages 362-373, 2004.
  63. B. Liu, Y. Zhu, E. A. Rundensteiner. Run-Time Operator State Spilling for Memory Intensive Long-Running Queries. In SIGMOD, pages 347-358, 2006.
  64. M. Liu, Z. G. Ives, and B. T. Loo. Enabling Incremental Query Re-Optimization. In SIGMOD, pages 1705-1720, 2016.
  65. W. Liu, G. Li, and J. Cheng. Fast PageRank approximation by adaptive sampling. Knowledge of Information System, 42(1):127-146, 2015.
  66. D. Logothetis, K. Yocum. Data Indexing for Stateful, Large- scale Data Processing. In NETDB, 2009.
  67. D. Logothetis, C. Olston, B. Reed, K.C. Webb, K. Yocum. Stateful Bulk Processing for Incremental Analytics. In ACM Symposium on Cloud Computing (SoCC), pages 51-62, 2010.
  68. G. Losa et al. CAPSULE: language and system support for efficient state sharing in distributed stream processing systems. In DEBS, pages 268-277, 2012.
  69. Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, J. Hellerstein. Distributed GraphLab: A Framework for Machine Learning in the Cloud. In PVLDB, 5(8):716-727, 2012.
  70. V. Markl: Breaking the Chains: On Declarative Data Analysis and Data Independence in the Big Data Era. In PVLDB, 7(13):1730-1733, 2014.
  71. N. Marz and J. Warren. Big Data: Principles and best practices of scalable realtime data systems. ISBN 9781617290343, 328 pages, 2015.
  72. T. D. Matteis, G. Mencagli. Parallel Patterns for Window- Based Stateful Operators on Data Streams: An Algorithmic Skeleton Approach. Journal of Parallel Programming, pages 1-20, 2016.
  73. F. McSherry, R. Isaacs, M. Isard, D. G. Murray. Composable Incremental and Iterative Data-Parallel Computation with Naiad. Technical Report number MSR-TR-2012-105. Microsoft Research Silicon Valley, 2012.
  74. F. McSherry, D. G. Murray, R. Isaacs, M. Isard. Differential Dataflow. In CIDR, 2013.
  75. J. Meehan et al. S-Store: streaming meets transaction processing. In PVLDB, 8(13):2134-2145, 2015.
  76. J. Meehan, S. Zdonik, S. Tian, Y. Tian, N. Tatbul, A. Dziedzic, A. Elmore. Integrating Real-Time and Batch Processing in a Polystore. In High-Performance Extreme Computing Conference (HPEC), 2016.
  77. I. Mitliagkas, M. Borokhovich, A. G. Dimakis, and C. Caramanis. FrogWild!: fast PageRank approximations on graph engines. In PVLDB, 8(8):874-885, 2015.
  78. M. Mokbel, M. Lu, and W. Aref. Hash-merge join: A non- blocking join algorithm for producing fast and early join results. In ICDE, pages 251-262, 2004.
  79. D. G. Murray, F. McSherry, R. Isaacs, M. Isard, P. Barham, and M. Abadi. Naiad: a timely dataflow system. In ACM Symposium on Operating Systems Principles (SOSP), pages 439-455, 2013.
  80. K. G. S. Madsen, Y. Zhou. Dynamic Resource Management in a Massively Parallel Stream Processing Engine. In CIKM, pages 13-22, 2015.
  81. K. G. S. Madsen, Y. Zhou, J. Cao. Integrative Dynamic Reconfiguration in a Parallel Stream Processing Engine. The Computing Research Repository (CoRR), abs/1602.03770, 2016.
  82. N. Naksinehaboon et al. Reliability-Aware Approach: An Incremental Checkpoint/Restart Model in HPC Environments. In CCGRID, pages 783-788, 2008.
  83. B. Nicolae and F. Cappello. AI-Ckpt: leveraging memory access patterns for adaptive asynchronous incremental checkpointing. In High-performance parallel and distributed computing (HPDC), pages 155-166, 2013.
  84. M. Nikolic, M. Elseidy, C. Koch. LINVIEW: incremental view maintenance for complex analytical queries. In SIGMOD, pages 253-264, 2014.
  85. M. Nikolic, M. Dashti, C. Koch. How to Win a Hot Dog Eating Contest: Distributed Incremental View Maintenance with Batch Updates. In SIGMOD, pages 511-526, 2016.
  86. B. Ottenwalder, B. Koldehofe, K. Rothermel, and U. Ramachandran. MigCEP: operator migration for mobility driven distributed complex event processing. In DEBS, pages 183-194, 2013.
  87. S. Padmanabhan, T. Malkemus, A. Jhingran, and R. Agarwal. Block Oriented Processing of Relational Database Operations in Modern Computer Architectures. In ICDE, pages 567-574, 2001.
  88. J. X. Parreira, D. Donato, S. Michel, and G. Weikum. Efficient and decentralized PageRank approximation in a peer-to-peer web search network. In PVLDB, 415-426, 2006.
  89. J. X. Parreira et al. The JXP Method for Robust PageRank Approximation in a Peer-to-Peer Web Search Network. The VLDB Journal, 17(2):291-313, 2008.
  90. M. Paun et al. Incremental Checkpoint Schemes for Weibull Failure Distribution. Journal on Foundation of Computer Science, 21(3):329-344, 2010.
  91. P. Pietzuch, J. Ledlie, J. Shneidman, M. Roussopoulos, M. Welsh, M. Seltzer. Network-aware operator placement for stream-processing systems. In ICDE, 2006.
  92. K. Ren, T. Diamond, D. J. Abadi, and A. Thomson. Low- Overhead Asynchronous Checkpointing in Main-Memory Database Systems. In SIGMOD, pages 1539-1551, 2016.
  93. Y. Robert, F. Vivien, D. Zaidouni. On the complexity of scheduling checkpoints for computational workflows. In DSN, pages 1-6, 2012.
  94. N. E. Sayed and B. Schroeder. Checkpoint/Restart in Practice: When Simple is Better. In IEEE International Conference on Cluster Computing (CLUSTER), pages 84-92, 2014.
  95. S. Schelter, S. Ewen, K. Tzoumas, V. Markl. "All Roads Lead to Rome:" Optimistic Recovery for Distributed Iterative Data Processing. In CIKM, pages 1919-1928, 2013.
  96. Z. Sebepou, and K. Magoutis. CEC: Continuous Eventual Checkpointing for data stream processing operators. In DSN, pages 145-156, 2011.
  97. J. Sermulins, W. Thies, R. Rabbah, and S. Amarasinghe. Cache Aware Optimization of Stream Programs. In Languages, Compiler, and Tool Support for Embedded Systems (LCTES), pages 115-126, 2005.
  98. M. A. Shah, J. M. Hellerstein, S. Chandrasekaran and M. J. Franklin. Flux: An Adaptive Partitioning Operator for Continuous Query Systems. In ICDE, 2003.
  99. G. J. Silva, B. Gedik, H. Andrade, K.-L. Wu. Language Level Checkpointing Support for Stream Processing Applications. In DSN, 2009.
  100. L. Su and Y. Zhou. Tolerating correlated failures in Massively Parallel Stream Processing Engines. In ICDE, pages 517-528, 2016.
  101. N. Tatbul et al. Handling Shared, Mutable State in Stream Processing with Correctness Guarantees. IEEE Data Engineering Bulletin, 38(4):94-104, 2015.
  102. A. Toshniwal et al. Storm@twitter. In SIGMOD, pages 147-156, 2014.
  103. Y. -C. Tu, S. Liu, S. Prabhakar, and B. Yao. Load shedding in stream databases: a control-based approach. In VLDB, pages 787-798, 2006.
  104. P. A. Tucker, D. Maier, T. Sheard, and L. Fegaras. Exploiting punctuation semantics in continuous data streams. TKDE, 15(3):555-568, 2003.
  105. P. Upadhyaya et al. A Latency and Fault-Tolerance Optimizer for Online Parallel Query Plans. In SIGMOD, pages 241-252, 2011.
  106. T. Urhan and M. J. Franklin. Xjoin: A reactively- scheduled pipelined join operator. IEEE Data Engineering Bulletin, 23(2):27-33, 2000.
  107. S. Viglas, J. F. Naughton, and J. Burger. Maximizing the output rate of multi-way join queries over streaming information sources. In VLDB, pages 285-296, 2003.
  108. H. Wang, L.-S. Peh, E. Koukoumidis, S. Tao, M. C. Chan. Meteor Shower: A Reliable Stream Processing System for Commodity Data Centers. In IEEE IPDPS, pages 1180- 1191, 2012.
  109. M. Weimer, T. Condie, and R. Ramakrishnan. Machine learning in ScalOps, a higher order cloud computing language. In NIPS BigLearn, Vol. 9, pages 389-396, 2011.
  110. X. Wu, et al. Top 10 algorithms in data mining. Knowledge Information System, 14(1):1-37, 2007.
  111. Y. Wu, K. Tan. elastic stateful stream computation in the cloud. In ICDE, pages 723-734, 2015.
  112. C. Xu, M. Holzemer, M. Kaul, V. Markl. Efficient fault- tolerance for iterative graph processing on distributed dataflow systems. In ICDE, pages 613-624, 2016.
  113. Z. B. Yossef and L. Mashiach. Local approximation of PageRank and reverse PageRank. In Research and development in information retrieval (SIGIR), pages 865-866, 2008.
  114. J. W. Young. A first order approximation to the optimum checkpoint interval. Communication of ACM, 17(9), 1974.
  115. W. Yu, X. Lin, W. Zhang. Fast incremental SimRank on link-evolving graphs. In ICDE, pages 304-315, 2014.
  116. M. Zaharia et al. Resilient Distributed Datasets: A Fault- Tolerant Abstraction for In-Memory Cluster Computing. In NSDI, 2012.
  117. G. Zeng. Fast approximate k-means via cluster closures.
  118. In Computer Vision and Pattern Recognition (CVPR), pages 3037-3044, 2012.
  119. H. Zhang, G. Chen, B. C. Ooi, K. L. Tan, M. Zhang. In- memory big data management and processing: A survey. TKDE, 27 (7):1920-1948, 2015.
  120. F. Zhu, Y. Fang, K. C.-C. Chang, and J. Ying. Incremental and accuracy-aware personalized pagerank through scheduled approximation. In PVLDB, 6(6):481-492, 2013.
  121. F. Zhu, Y. Fang, K. C.-C. Chang, and J. Ying. Scheduled approximation for Personalized PageRank with Utility-based Hub Selection. VLDB Journal, 24(5):655-679, 2015.
  122. Y. Zhu, E. Rundensteiner, G. T. Heineman. Dynamic Plan Migration for Continuous Queries over Data Streams. In SIGMOD, 2004.
  123. M. Zinkevich, M. Weimer, A. J. Smola, and L. Li. Parallelized stochastic gradient descent. In Neural Information Processing Systems (NIPS), pages 2595-2603, 2010.
  124. P. Van Roy, S. Haridi. Concepts, Techniques, and Models of Computer Programming. MIT Press, Cambridge, 2004.
  125. S. Sakr, A. Liu, A. Fayoumi. The Family of MapReduce and Large Scale Data Processing Systems. Journal of ACM Computing Surveys (ACM CSUR), 46(1), 2013.
  126. P. Carbone, S. Ewen, G. Fóra, S. Haridi, S. Richter, K. Tzoumas. State Management in Apache Flink: Consistent Stateful Distributed Stream Processing. PVLDB 10(12): 1718-1729, 2017.
  127. N. R Katsipoulakis, A. Labrinidis, P.K. Chrysanthis. A Holistic View of Stream Partitioning Costs. PVLDB, 10(11): 1286-1297, 2017.
  128. MAU Nasir, GDF Morales, D García-Soriano, N Kourtellis, M Serafini. The power of both choices: Practical load balancing for distributed stream processing engines. In ICDE, pages 137-148, 2015.
  129. L. Wang, T. Z. J. Fu, R. T. B. Ma, M. Winslett, Z. Zhang. Elasticutor: Rapid Elasticity for Realtime Stateful Stream Processing. The Computing Research Repository (CoRR), abs/1711.01046, 2017.
  130. C. Hochreiner, M. Vögler, S. Schulte, S. Dustdar. Elastic Stream Processing for the Internet of Things. In CLOUD, pages 100-107, 2016.
  131. S. Kulkarni et al. Twitter Heron: Stream Processing at Scale. In SIGMOD, pages 239-250, 2015.
  132. D. J. Abadi et al. The design of the Borealis stream processing engine. In CIDR, pages 277-289, 2005.
  133. M. Balazinska, H. Balakrishnan, S. Madden, M. Stonebraker. Fault-tolerance in the Borealis distributed stream processing system. In SIGMOD, pages 13-24, 2005.
  134. M. Balazinska, H. Balakrishnan, S. Madden, M. Stonebraker. Fault-tolerance in the Borealis distributed stream processing system. TODS, 33(1):1-44, 2008.
  135. Apache Flink, http://flink.apache.org/
  136. Apache Samza, http://samza.apache.org/
  137. Apache Spark, http://spark.apache.org/
  138. Kappa Architecture, http://kappa-architecture.com
  139. MAU Nasir, GDF Morales, N. Kourtellis, M. Serafini. When two choices are not enough: Balancing at scale in Distributed Stream Processing. In ICDE, pages 589-600, 2016

Related papers

E-Storm: Replication-Based State Management in Distributed Stream Processing Systems
Benjamin Rubinstein

2017 46th International Conference on Parallel Processing (ICPP)

Apache Storm is a fault-tolerant, distributed inmemory computation system for processing large volumes of high-velocity data in real-time. As an integral part of the faulttolerance mechanism, Storm's state management is achieved by a checkpointing framework, which commits states regularly and recovers lost states from the latest checkpoint. However, this method involves a remote data store for state preservation and access, resulting in significant overheads to the performance of error-free execution. In this paper, we propose E-Storm, a replication-based state management system that actively maintains multiple state backups on different worker nodes. We build a prototype on top of Storm by extending it with monitoring and recovery modules to support inter-task state transfer whenever needed. The experiments carried out on synthetic and real-world streaming applications confirm that E-Storm outperforms the existing checkpointing method in terms of the resulting application performance, obtaining as much as 9.44 times throughput improvement while reducing the application latency down to 9.8%.

View PDFchevron_right
Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters
Scott Shenker

2012

Many important "big data" applications need to process data arriving in real time. However, current programming models for distributed stream processing are relatively low-level, often leaving the user to worry about consistency of state across the system and fault recovery. Furthermore, the models that provide fault recovery do so in an expensive manner, requiring either hot replication or long recovery times. We propose a new programming model, discretized streams (D-Streams), that offers a high-level functional programming API, strong consistency, and efficient fault recovery. D-Streams support a new recovery mechanism that improves efficiency over the traditional replication and upstream backup solutions in streaming databases: parallel recovery of lost state across the cluster. We have prototyped D-Streams in an extension to the Spark cluster computing framework called Spark Streaming, which lets users seamlessly intermix streaming, batch and interactive queries.

View PDFchevron_right
State management in Apache Flink®
Gyula Fóra

Proceedings of the VLDB Endowment

Stream processors are emerging in industry as an apparatus that drives analytical but also mission critical services handling the core of persistent application logic. Thus, apart from scalability and low-latency, a rising system need is first-class support for application state together with strong consistency guarantees, and adaptivity to cluster reconfigurations, software patches and partial failures. Although prior systems research has addressed some of these specific problems, the practical challenge lies on how such guarantees can be materialized in a transparent, non-intrusive manner that relieves the user from unnecessary constraints. Such needs served as the main design principles of state management in Apache Flink, an open source, scalable stream processor. We present Flink's core pipelined, in-flight mechanism which guarantees the creation of lightweight, consistent, distributed snapshots of application state, progressively, without impacting continuous execution. Co...

View PDFchevron_right
Stateful metadata for big data
Nagaraju pothineni

2019

Large volumes of data, characterized by large variety and high update velocities, pose challenges in terms of storage, application of concurrently occurring frequent updates, and serving processes that require the most accurate version of the data simultaneously. In most current schemes, it is not possible to guarantee all of these characteristics and a relaxing one or more requirements is necessary. The present disclosure describes a scalable, easy-to-maintain metadata mechanism that is fast and efficient to update, and can provide all the above guarantees on data. The metadata maintains lightweight validity markers, and simple algebra is performed thereof to surface the most up to date and accurate data while enabling constant updates to the data in a non-blocking fashion. KEYWORDS • Big data • Stateful metadata • Concurrent update • Non-blocking update • Validity marker BACKGROUND Most big data (BD) systems exhibit extremely large volumes of incoming data with immense variation, usually collected at enormous velocities. BD systems often have associated serving pipelines -such as e-commerce, advertising, and analytics or machine learning modeling -that utilize this data as input. Since the incoming data is received asynchronously from multiple uncorrelated sources, the data as received is not in a structured format that is readily suitable for 2

View PDFchevron_right
Real-time stream processing for Big Data
Wolfram Wingerath

it - Information Technology, 2016

With the rise of the web 2.0 and the Internet of things, it has become feasible to track all kinds of information over time, in particular fine-grained user activities and sensor data on their environment and even their biometrics. However, while efficiency remains mandatory for any application trying to cope with huge amounts of data, only part of the potential of today's Big Data repositories can be exploited using traditional batch-oriented approaches as the value of data often decays quickly and high latency becomes unacceptable in some applications. In the last couple of years, several distributed data processing systems have emerged that deviate from the batch-oriented approach and tackle data items as they arrive, thus acknowledging the growing importance of timeliness and velocity in Big Data analytics. In this article, we give an overview over the state of the art of stream processors for low-latency Big Data analytics and conduct a qualitative comparison of the most po...

View PDFchevron_right

Related topics

  • Information Systems
  • Computer Science
  • Distributed Computing
  • Data Science
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved