Apache Spark Streaming

Apache Spark has emerged as the de facto framework for big data analytics with its advanced in-memory programming model and upper-level libraries for scalable machine learning, graph analysis, streaming and structured data processing. It is a general-purpose cluster computing framework with language-integrated APIs in Scala, Java, Python and R. As a rapidly evolving open source project, with an increasing number of contributors from both academia and industry, it is difficult for researchers to comprehend the full body of development and research behind Apache Spark, especially those who are beginners in this area. In this paper, we present a technical review on big data analytics using Apache Spark. This review focuses on the key components, abstractions and features of Apache Spark. More specifically, it shows what Apache Spark has for designing and implementing big data algorithms and pipelines for machine learning, graph analysis and stream processing. In addition, we highlight some research and development directions on Apache Spark for big data analytics.

Aspect mining constitutes an essential part of delivering concise and, perhaps more importantly, accurately tailored cultural content. With the advent of social media, there is a data abundance so that analytics can be reliably designed for ultimately providing valuable information towards a given product or service. Naturally representing and efficiently processing a large number of opinions can be implemented with the use of streaming technologies. Big data analytics are especially important in the case of cultural content management where reviews and opinions may be analyzed in order to extract meaningful representations. In this paper, we present a NoSQL database approach for aspect mining of a cultural heritage scenario by taking advantage of Apache Spark streaming architecture.

In the era of big data, organizations are increasingly relying on robust data pipelines to manage, process, and analyze vast amounts of data in real-time. Apache Kafka and Apache Spark are two of the most prominent tools used in the construction of such pipelines. Kafka is renowned for its ability to handle real-time data streams with high throughput and low latency, making it ideal for collecting and distributing large volumes of data across systems. Spark, on the other hand, is a powerful analytics engine designed for fast data processing and complex computational tasks, supporting both batch and stream processing. This paper explores the key features, strengths, and limitations of Kafka and Spark in the context of data pipeline development. Kafka excels in scenarios requiring high availability and real-time data ingestion, serving as a durable, fault-tolerant messaging system that can integrate with various data sources and consumers. It supports a publish-subscribe model and maintains message order, ensuring reliability in data transmission. Meanwhile, Spark provides a robust framework for performing in-memory data processing, facilitating machine learning, graph computation, and SQL-based queries on large datasets. We analyze several case studies and real-world applications to illustrate how these technologies are implemented to meet diverse organizational needs. Through a comparative analysis, we identify scenarios where Kafka and Spark are used in tandem to leverage their complementary strengths, highlighting the benefits of integrating these tools within a unified data pipeline architecture. The discussion emphasizes the importance of choosing the right technology stack based on factors such as data velocity, volume, variety, and the complexity of processing requirements.

Emergency evacuation management in urban areas is paramount due to rapid population growth and urbanization. This research introduces an innovative methodology leveraging the Internet of Things (IoT) and Big Data Technologies to fortify emergency management systems. This research proposes a system that integrates IoT sensors, an MQTT broker, a Kafka broker, HDFS, and Spark streaming, all housed in the cloud infrastructure, facilitating real-time monitoring and calculation of varying risk scores through each of the exit routes during an emergency and providing the evacuees with a user interface that alerts them with the most probable safety exits. Unlike conventional evacuation systems that focus solely on identifying the shortest exit route, or have a static approach to processing data, our solution minimizes potential risks during an emergency by monitoring factors such as pressure, temperature, humidity and flame intensity in realtime and predicts risk scores for danger-prone exits considering the number of exits and accessibility of each exit, thus notifying the user of the same through a web interface. To illustrate the practical applicability of our system, we present a case study centred around a sports stadium, offering a tangible example of the proposed solution's effectiveness.

The topic of the 2020 DEBS Grand Challenge is to develop a solution for Non Intrusive Load Monitoring (NILM). Sensors continuously send voltage and current data into a stream processing application that would detect the pattern of power data based on the data characteristics. NILM is important in signal processing especially in those advancing areas such as 5G and IoT products, which generate massive amounts of data from the edge of the network. Our solution focuses on how to divide and parallelize jobs as small as possible while keeping some reasonable Service Level Agreement (SLA) including job sizes and latency so that it would be practical for edge or fog deployment. This paper describes our solution based on Apache Flink, a stream processing framework, and the DBSCAN density based clustering algorithm for anomaly detection through the context of data provided by DEBS Grand Challenge. CCS CONCEPTS • Networks ! Cloud computing; • Computer systems organization ! Real-time system architecture.

• RDDs provide a fault tolerant implementation of distributed immutable multisets. • Computations are defined as transformations on RDDs. • The set of predefined RDD transformations includes typical higher-order functions from functional programming like map, filter, etc. • It also includes aggregations by key and joins for RDDs of key-value pairs. • We can also use Spark actions, which allow us to collect results into the program driver, or store them into an external data store. A. Riesco (UCM) Temporal Random Testing for Spark Streaming ScalaMAD 2 / 42 A. Riesco (UCM) Temporal Random Testing for Spark Streaming ScalaMAD 3 / 42 Spark Spark Streaming Spark Streaming • These notions of transformations and actions are extended in Spark Streaming from RDDs to DStreams (Discretized Streams). • DStreams are series of RDDs corresponding to micro-batches. • These batches are generated at a fixed rate according to the configured batch interval. A. Riesco (UCM) Temporal Random Testing for Spark Streaming ScalaMAD 4 / 42

The paper addresses a highly relevant and contemporary topic in the field of data processing. Big data is a crucial aspect of modern computing, and the choice of processing framework can significantly impact performance and efficiency. The technical revolution of big data has changed how organizations handle and value large databases. As data quantities expand quickly, effective and scalable data processing systems are essential. MapReduce and Apache Spark are two of the most popular large data processing techniques. This study compares these two frameworks to determine their merits, shortcomings, and applicability for big data applications. Nearly quintillion bytes of data are created daily. Approximately 90% of data was produced in the previous two years. At this stage, data comes from temperature sensors, social media, movies, photographs, transaction records (like banking records), mobile phone conversations, GPS signals, etc. In this article, all key big data technologies are introduced. This document compares all big data technologies and discusses their merits and downsides. Run trials using multiple data sets of varying sizes to validate and explain the study. Graphical depiction shows how one tool outperforms others for given data. Big Data is data generated by the rapid usage of the internet, sensors, and heavy machinery, with great volume, velocity, variety, and veracity. Numbers, photos, videos, and text are omnipresent in every sector. Due to the pace and amount of data generation, the computing system struggles to manage large data. Data is stored in a distributed architectural file system due to its size and complexity. Big distributed file systems, which must be fault-tolerant, adaptable, and scalable, make complicated data analysis dangerous and time-consuming. Big data collection is called 'datafication'. Big data is 'datafied' for productivity. Organisation alone does not make Big Data valuable; we must choose what we can do with it.

Real world social networks are highly dynamic environments consisting of numerous users and communities, rendering the tracking of their evolution a challenging problem. In this work, we propose a parallel algorithm for tracking dynamic communities between consecutive timeframes of the social network, where communities are represented as undirected graphs. Our method compares the communities based on the widely adopted Jaccard similarity measure and is implemented on top of Apache Flink, a novel framework for parallel and distributed data processing. We evaluate the benefits, in terms of execution time, that parallel processing brings to community tracking on datasets carrying different quantitative characteristics, derived from two popular social media platforms; Twitter and Mathematics Stack Exchange Q&A. Experiments show that our parallel method has the ability to calculate the similarity of communities within seconds, even for large social networks, consisting of more than 600 communities per timeframe. CCS CONCEPTS • Human-centered computing → Social network analysis; • Computing methodologies → Parallel algorithms;

As data permeates all disciplines, the role of big data becomes increasingly important. Sensors, IoT devices, social networks, and online transactions are all generating data that can be monitored constantly to enable a business to identify opportunity to enhance customer service and increase revenue. This need for real-time processing of big data has led to the development of frameworks for distributed stream processing in clusters. It is important for such frameworks to be resilient against variable operating conditions such as server load variation, changes in data ingestion rates, and workload characteristics. In this thesis, we explore the effects of the batch size on the performance of streaming workloads by developing an adaptive batching framework and building load-balancing algorithms on top of this framework. We explore the idea of using a combination of adaptive batching of tuples and dynamic tuple dispatching to improve the throughput and load-distribution of the workload. We show through experiments that the system is able to be resilient and robust under varying operating conditions.

This paper addresses the challenge of predicting the level of parallelism in distributed stream processing (DSP) systems, which are essential to deal with different high workload requirements of various industries such as e-commerce, online gaming, etc., where DSP systems are extensively used. Existing DSP systems rely on either manual tuning of parallelism degree or workload-driven learned models for tuning parallelism, which is either not efficient or can lead to costly operator migrations and downtime when there are workload drifts. Thus, we argue for a learned model that can autonomously decide on the right parallelism degree while generalizing across workloads and meeting the current demands of DSP applications. We propose a novel approach that leverages zeroshot cost models to predict parallelism degree while generalizing across unseen streaming workloads out-of-the-box. To reduce training effort, we propose a rule-based strategy that selects parallelism degree and meaningful transferable features related to query workload and hardware that influences the parallelism decisions. We demonstrate the effectiveness of our strategy by evaluating it with different amount of training queries and show that it achieves lower costs for parallel continuous query processing. CCS CONCEPTS • Computer systems organization → Real-time systems.

This paper presents a benchmark of stream processing throughput comparing Apache Spark Streaming (under file-, TCP socket-and Kafka-based stream integration), with a prototype P2P stream processing framework, HarmonicIO. Maximum throughput for a spectrum of stream processing loads are measured, specifically, those with large message sizes (up to 10MB), and heavy CPU loads-more typical of scientific computing use cases (such as microscopy), than enterprise contexts. A detailed exploration of the performance characteristics with these streaming sources, under varying loads, reveals an interplay of performance trade-offs, uncovering the boundaries of good performance for each framework and streaming source integration. We compare with theoretic bounds in each case. Based on these results, we suggest which frameworks and streaming sources are likely to offer good performance for a given load. Broadly, the advantages of Spark's rich feature set comes at a cost of sensitivity to message size in particular-common stream source integrations can perform poorly in the 1MB-10MB range. The simplicity of HarmonicIO offers more robust performance in this region, especially for raw CPU utilization.

This paper presents a benchmark of stream processing throughput comparing Apache Spark Streaming (under file-, TCP socket-and Kafka-based stream integration), with a prototype P2P stream processing framework, HarmonicIO. Maximum throughput for a spectrum of stream processing loads are measured, specifically, those with large message sizes (up to 10MB), and heavy CPU loads-more typical of scientific computing use cases (such as microscopy), than enterprise contexts. A detailed exploration of the performance characteristics with these streaming sources, under varying loads, reveals an interplay of performance trade-offs, uncovering the boundaries of good performance for each framework and streaming source integration. We compare with theoretic bounds in each case. Based on these results, we suggest which frameworks and streaming sources are likely to offer good performance for a given load. Broadly, the advantages of Spark's rich feature set comes at a cost of sensitivity to message size in particular-common stream source integrations can perform poorly in the 1MB-10MB range. The simplicity of HarmonicIO offers more robust performance in this region, especially for raw CPU utilization.

In this paper, we present a scalable and real-time intelligent transportation system based on a big data framework. The proposed system allows for the use of existing data from road sensors to better understand traffic flow, traveler behavior, and increase road network performance. Our transportation system is designed to process large-scale stream data to analyze traffic events such as incidents, crashes and congestion. The experiments performed on the public transportation modes of the city of Casablanca in Morocco reveal that the proposed system achieves a significant gain of time, gathers large-scale data from many road sensors and is not expensive in terms of hardware resource consumption.

Business processes represent a cornerstone to the operation of any enterprise. They are the operational means for such organizations to fulfill their goals. Nowadays, enterprises are able to gather massive amounts of event data. These are generated as business processes are executed and stored in transaction logs, databases, e-mail correspondences, free form text on (enterprise) social media, and so on. Taping into these data, enterprises would like to weave data analytic techniques into their decision making capabilities. In recent years, the IT industry has witnessed significant advancements in the domain of Big Data analytics. Unfortunately, the business process management (BPM) community has not kept up to speed with such developments and often rely merely on traditional modeling-based approaches. New ways of effectively exploiting such data are not sufficiently used. In this paper, we advocate that a good understanding of the business process and Big Data worlds can play an effective role in improving the efficiency and the quality of various data-intensive business operations using a wide spectrum of emerging Big Data systems. Moreover, we coin the term process footprint as a wider notion of process data than that is currently perceived in the BPM community. A roadmap towards taking business process data intensive operations to the next level is shaped in this paper. INDEX TERMS Business process analytics, Big Data systems, process data-intensive operations • The (re)design phase focuses on BP models that capture an enterprise's know how when processing users' requests. All activities in the model are identified along

Stream Processing Engines (SPEs) have to support high data ingestion to ensure the quality and efficiency for the end-user or a system administrator. The data flow processed by SPE fluctuates over time, and requires real-time or near realtime resource pool adjustments (network, memory, CPU and other). This scenario leads to the problem known as skewed data production caused by the non-uniform incoming flow at specific points on the environment, resulting in slow down of applications caused by network bottlenecks and inefficient load balance. This work proposes Aten as a solution to overcome unbalanced data flows processed by Big Data Stream applications in heterogeneous systems. Aten manages data aggregation and data streams within message queues, assuming different algorithms as strategies to partition data flow over all the available computational resources. The paper presents preliminary results indicating that is possible to maximize the throughput and also provide low latency levels for SPEs.

The rapid growth of stream applications in financial markets, health care, education, social media, and sensor networks represents a remarkable milestone for data processing and analytic in recent years, leading to new challenges to handle Big Data in real-time. Traditionally, a single cloud infrastructure often holds the deployment of Stream Processing applications because it has extensive and adaptative virtual computing resources. Hence, data sources send data from distant and different locations of the cloud infrastructure, increasing the application latency. The cloud infrastructure may be geographically distributed and it requires to run a set of frameworks to handle communication. These frameworks often comprise a Message Queue System and a Stream Processing Framework. The frameworks explore Multi-Cloud deploying each service in a different cloud and communication via high latency network links. This creates challenges to meet real-time application requirements because the data streams have different and unpredictable latencies forcing cloud providers' communication systems to adjust to the environment changes continually. Previous works explore static micro-batch demonstrating its potential to overcome communication issues. This paper introduces BurstFlow, a tool for enhancing communication across data sources located at the edges of the Internet and Big Data Stream Processing applications located in cloud infrastructures. BurstFlow introduces a strategy for adjusting the micro-batch sizes dynamically according to the time required for communication and computation. BurstFlow also presents an adaptive data partition policy for distributing incoming streams across available machines by considering memory and CPU capacities. The experiments use a real-world multi-cloud deployment showing that BurstFlow can reduce the execution time up to 77% when compared to the state-of-the-art solutions, improving CPU efficiency by up to 49%. INDEX TERMS Big data, stream processing applications, multi cloud, micro-batches, data partition.

Large scale applications nowadays continuously generate massive amounts of data at high speed. Stream processing engines (SPEs) such as Apache Storm and Flink are becoming increasingly popular because they provide reliable platforms to process such fast data streams in real time. Despite previous research in the field of auto-scaling of resources, current SPEs, whether open source such as Apache Storm, or commercial such as streaming components in IBM Infosphere and Microsoft Azure, lack the ability to automatically grow and shrink to meet the needs of streaming data applications. Moreover, previous research on auto-scaling focuses on techniques for scaling resources reactively, which can delay the scaling decision unacceptably for time sensitive stream applications. To the best of our knowledge, there has been no or limited research using machine learning techniques to proactively predict future bottlenecks based on the data flow characteristics of the data stream workload. In this...

Most high-performance data processing (a.k.a. big data) systems allow users to express their computation using abstractions (like MapReduce), which simplify the extraction of parallelism from applications. Most frameworks, however, do not allow users to specify how communication must take place: That element is deeply embedded into the run-time system abstractions, making changes hard to implement. In this work, we describe Wathershed-ng, our re-engineering of the Watershed system, a framework based on the filter-stream paradigm and originally focused on continuous stream processing. Like other big-data environments, Watershed provided object-oriented abstractions to express computation (filters), but the implementation of streams was a run-time system element. By isolating stream functionality into appropriate classes, combination of communication patterns and reuse of common message handling functions (like compression and blocking) become possible. The new architecture even allows the design of new communication patterns, for example, allowing users to choose MPI, TCP, or shared memory implementations of communication channels as their problem demands. Applications designed for the new interface showed reductions in code size on the order of 50% and above in some cases. The performance results also showed significant improvements, because some implementation bottlenecks were removed in the re-engineering process.

Asking questions is the driving force for scientific progress. But as important as it is to ask questions, so important is to be able to understand the obtained answers. In this way, we are able to verify the sanity of the question itself and if necessary refine it. In the same spirit, in this PhD we focus on SQL queries over re-lational databases with the aim to (1) pinpoint the reasons on the SQL query that led to missing-answers (tuples that were expected but not obtained in the query result), (2) fix the SQL query so that it best fits the user's expectations, and (3) do both efficiently so as to be of practical use.

In recent years, users have come to expect reactivity from their applications, i.e. they assume that changes made by other users are immediately reflected in the interfaces they are using. Examples are shared worksheets and websites targeting social interaction. These applications require the underlying data storage to publish new and updated information as soon as it is created: Data access is push-based. In contrast, traditional database management [HSH07] has been tailored towards pull-based data access where information is only made available as a direct response to a client request. While triggers and other push-oriented mechanisms have been added to their initial design, they are outperformed by several orders of magnitude when held against natively push-based systems [SC05]. In consequence, the inadequacy of traditional database technology for handling rapidly changing data has been widely accepted as one of the fundamental challenges in database design [SCZ05]. To warrant low-latency updates in quickly evolving domains, systems for data stream management and processing [GZ10, Win+16] break with the idea of maintaining a persistent data repository. Instead of random access queries on static collections, they perform sequential, long-running queries or processing tasks over data streams. Data stream management and processing systems thus generate new output whenever new data becomes available and are therefore natively push-based. However, data is only available for processing in one single pass, because data streams are conceptually unbounded sequences of data items, infeasible to retain indefinitely. Database and data stream management, respectively, follow fundamentally different semantics regarding the way that data is processed and accessed as Table 1.1 summarizes. The concept of persistent collections conforms to applications that require a (consistent) view of their domain, for instance to keep track of warehouse stock or do financial accounting. The stream data model, on the other hand, comes natural for domains that facilitate a notion of event sequences or demand reasoning about the relationship between events, for example to analyze stock prices or identify malicious user behavior. However, the access paradigm-pull-based or

Big data processing systems are evolving to be more stream oriented where each data record is processed as it arrives by distributed and low-latency computational frameworks on a continuous basis. As the stream processing technology matures and more organizations invest in digital transformations, new applications of stream analytics will be identified and implemented across a wide spectrum of industries. One of the challenges in developing a streaming analytics infrastructure is the difficulty in selecting the right stream processing framework for the different use cases. With a view to addressing this issue, in this paper we present a taxonomy, a comparative study of distributed data stream processing and analytics frameworks, and a critical review of representative open source (Storm, Spark Streaming, Flink, Kafka Streams) and commercial (IBM Streams) distributed data stream processing frameworks. The study also reports our ongoing study on a multilevel streaming analytics architecture that can serve as a guide for organizations and individuals planning to implement a real-time data stream processing and analytics framework. INDEX TERMS Dataflow architectures, data stream architectures, distributed processing systems comparison, survey, taxonomy.

Aspect mining constitutes an essential part of delivering concise and, perhaps more importantly, accurately tailored cultural content. With the advent of social media, there is a data abundance so that analytics can be reliably designed for ultimately providing valuable information towards a given product or service. Naturally representing and efficiently processing a large number of opinions can be implemented with the use of streaming technologies. Big data analytics are especially important in the case of cultural content management where reviews and opinions may be analyzed in order to extract meaningful representations. In this paper, we present a NoSQL database approach for aspect mining of a cultural heritage scenario by taking advantage of Apache Spark streaming architecture.

Big data analytics platforms have played a critical role in the unprecedented success of data-driven applications. However, real-time and streaming data applications, and recent legislation, e.g., GDPR in Europe, have posed constraints on exchanging and analyzing data, especially personal data, across geographic regions. To address such constraints data has to be processed and analyzed in-situ and aggregated results have to be exchanged among the different sites for further processing. This introduces additional network delays due to the geographic distribution of the sites and potentially affecting the performance of analytics platforms that are designed to operate in datacenters with low network delays. In this paper, we show that the three most popular big data analytics systems (Apache Storm, Apache Spark, and Apache Flink) fail to tolerate round-trip times more than 30 milliseconds even when the input data rate is low. The execution time of distributed big data analytics tasks degrades substantially after this threshold, and some of the systems are more sensitive than others. A closer examination and understanding of the design of these systems show that there is no winner in all wide-area settings. However, we show that it is possible to improve the performance of all these popular big data analytics systems significantly amid even transcontinental delays (where inter-node delay is more than 30 milliseconds) and achieve performance comparable to this within a datacenter for the same load. Index Terms-Wide-area analytics, big data analytics, geodistributed systems, networked systems. I. INTRODUCTION B IG DATA analytics platforms [1], [2], [3], [4], [5], [6] have played a critical role in the unprecedented success of data-driven applications. Such platforms are typically deployed

Apache Spark has emerged as the de facto framework for big data analytics with its advanced in-memory programming model and upper-level libraries for scalable machine learning, graph analysis, streaming and structured data processing. It is a general-purpose cluster computing framework with language-integrated APIs in Scala, Java, Python and R. As a rapidly evolving open source project, with an increasing number of contributors from both academia and industry, it is difficult for researchers to comprehend the full body of development and research behind Apache Spark, especially those who are beginners in this area. In this paper, we present a technical review on big data analytics using Apache Spark. This review focuses on the key components, abstractions and features of Apache Spark. More specifically, it shows what Apache Spark has for designing and implementing big data algorithms and pipelines for machine learning, graph analysis and stream processing. In addition, we highlight some research and development directions on Apache Spark for big data analytics.

Big data processing systems are evolving to be more stream oriented where each data record is processed as it arrives by distributed and low-latency computational frameworks on a continuous basis. As the stream processing technology matures and more organizations invest in digital transformations, new applications of stream analytics will be identified and implemented across a wide spectrum of industries. One of the challenges in developing a streaming analytics infrastructure is the difficulty in selecting the right stream processing framework for the different use cases. With a view to addressing this issue, in this paper we present a taxonomy, a comparative study of distributed data stream processing and analytics frameworks, and a critical review of representative open source (Storm, Spark Streaming, Flink, Kafka Streams) and commercial (IBM Streams) distributed data stream processing frameworks. The study also reports our ongoing study on a multilevel streaming analytics architecture that can serve as a guide for organizations and individuals planning to implement a real-time data stream processing and analytics framework. INDEX TERMS Dataflow architectures, data stream architectures, distributed processing systems comparison, survey, taxonomy.

This paper presents an analysis of four online stream processing systems (MillWheel, S4, Spark Streaming and Storm) regarding the strategies they use for fault tolerance. We use this sort of system for processing of data streams that can come from different sources such as web sites, sensors, mobile phones or any set of devices that provide real-time high-speed data. Typically, these systems are concerned more with the throughput in data processing than on fault tolerance. However, depending on the type of application, we should consider fault tolerance as an important a feature. The work describes some of the main strategies for fault tolerance-replication components, upstream backup, checkpoint and recovery-and shows how each of the four systems uses these strategies. In the end, the paper discusses the advantages and disadvantages of the combination of the strategies for fault tolerance in these systems.

Index joins range unit pivotal for proficiency and quality once technique questions over colossal data. HIVE may be a cluster balanced immense data administration motor that is good for data examination applications and for OLAP for phenomenally "specific" inquiries whose yield sizes region unit little division from the contributing data, there the beast compel experiences poor execution because of repetitive circle I/O operations or end in starts of additional guide operations. Here all through this paper a shot is made and propose file joins procedure to rush up the inquiry strategy and incorporate it in Hive by mapping our vogue to the unique change stream to assess the execution, we've a slant to give and measure check inquiries on datasets created abuse TPC-H benchmark. Our outcomes show vital execution increase over moderately tremendous data sets and/or uncommonly specific questions having a two-way are a piece of and one be a piece of condition.

In this tutorial we present the results of recent research about the cloud enablement of data streaming systems. We illustrate, based on both industrial as well as academic prototypes, new emerging uses cases and research trends. Specifically, we focus on novel approaches for (1) scalability and (2) fault tolerance in large scale distributed streaming systems. In general, new fault tolerance mechanisms strive to be more robust and at the same time introduce less overhead. Novel load balancing approaches focus on elastic scaling over hundreds of instances based on the data and query workload. Finally, we present open challenges for the next generation of cloud-based data stream processing engines.

Resource management in Distributed Stream Processing Systems (DSPS) defines the way queries are deployed on in-network resources to deliver query results while fulfilling the Quality of Service (QoS) requirements of the end-users. Various resource management mechanisms have been proposed in DSPS; however, they become inefficient in challenging conditions imposed by the dynamic environment and heterogeneous resources. This is because they focus on pre-configuration of both single and static QoS requirements. In addition, they lack cooperation between heterogeneous resources which amplify the problem of coordination between resources. This could lead to severe performance degradation such as inconsistent and incorrect query results in comparison to homogeneous resources. To solve the above challenges, in this research work, we will propose mechanisms: (i) to forecast the performance of network and heterogeneous resources, (ii) to select an efficient resource management approach, and (iii) for cooperation between resources in a dynamic environment. CCS CONCEPTS • Information systems → Stream management.

Distributed Stream Processing (DSP) systems highly rely on parallelism mechanisms to deliver high performance in terms of latency and throughput. Yet the development of such parallel systems altogether comes with numerous challenges. In this paper, we focus on how to select appropriate resources for parallel stream processing under the presence of highly dynamic and unseen workloads. We present PANDA that provides a novel learned approach for highly efficient and parallel DSP systems. The main idea is to provide accurate resource estimates and hence optimal parallelism degree using zero-shot cost models to ensure the performance demands. CCS CONCEPTS • Computer systems organization → Real-time systems.

The advent of various processing frameworks which happens under big data technologies is due to tremendous dataset size and its complexity. The speed of execution was much higher with High Performance computing frameworks rather than big data processing frameworks. As majority of the jobs under big data are mostly data intensive rather than computation intensive, the High Performance Computing paradigms were not been used in big data processing. This paper reviews two distributed and parallel computing frameworks: Apache Spark and MPI. Sentiment analysis on twitter data is chosen as a test case application for benchmarking and implemented on Scala programming for spark processing and in C++ for MPI. Experiments were conducted on Google cloud virtual machines for three data set sizes, 100 GB, 500 GB and 1 TB to compare the execution times. Results shown that MPI outperforms Apache Spark in parallel and distributed cluster computing environments and hence the higher performance of MPI can be exploited in big data applications for improving speedups.

Event processing (EP) is a data processing technology that conducts online processing of event information. In this survey, we summarize the latest cutting-edge work done on EP from both industrial and academic research community viewpoints. We divide the entire field of EP into three subareas: EP system architectures, EP use cases, and EP open research topics. Then we deep dive into the details of each subsection. We investigate the system architecture characteristics of novel EP platforms, such as Apache Storm, Apache Spark, and Apache Flink. We found significant advancements made on novel application areas, such as the Internet of Things; streaming machine learning (ML); and processing of complex data types such as text, video data streams, and graphs. Furthermore, there has been significant body of contributions made on event ordering, system scalability, development of EP languages and exploration of use of heterogeneous devices for EP, which we investigate in the latter half o...

This paper presents a benchmark of stream processing throughput comparing Apache Spark Streaming (under file-, TCP socket-and Kafka-based stream integration), with a prototype P2P stream processing framework, HarmonicIO. Maximum throughput for a spectrum of stream processing loads are measured, specifically, those with large message sizes (up to 10MB), and heavy CPU loads-more typical of scientific computing use cases (such as microscopy), than enterprise contexts. A detailed exploration of the performance characteristics with these streaming sources, under varying loads, reveals an interplay of performance trade-offs, uncovering the boundaries of good performance for each framework and streaming source integration. We compare with theoretic bounds in each case. Based on these results, we suggest which frameworks and streaming sources are likely to offer good performance for a given load. Broadly, the advantages of Spark's rich feature set comes at a cost of sensitivity to message size in particular-common stream source integrations can perform poorly in the 1MB-10MB range. The simplicity of HarmonicIO offers more robust performance in this region, especially for raw CPU utilization.

Apache Flink is an open-source system for the scalable processing of batch and streaming data. Flink does not natively support efficient processing of spatial data streams, which is the requirement of many applications dealing with spatial data. Besides Flink, other scalable spatial data processing platforms including GeoSpark, Spatial Hadoop, GeoMesa and Parallel Secondo do not support streaming workloads and can only handle static/batch workloads. Hence this work presents GeoFlink, which extends Apache Flink to support spatial data types, index and continuous queries. To enable efficient processing of continuous spatial queries and for the effective data distribution among the Flink cluster nodes, a grid-based index is introduced. The grid index enables the pruning of spatial objects which cannot be part of a spatial query result and thus can guarantee efficient query processing, similarly it helps in preserving spatial data proximity, hence resulting in effective data distributio...

Natural hazards result in devastating losses in human life, environmental assets and personal, and regional and national economies. The availability of different big data such as satellite imageries, Global Positioning System (GPS) traces, mobile Call Detail Records (CDRs), social media posts, etc., in conjunction with advances in data analytic techniques (e.g., data mining and big data processing, machine learning and deep learning) can facilitate the extraction of geospatial information that is critical for rapid and effective disaster response. However, disaster response systems development usually requires the integration of data from different sources (streaming data sources and data sources at rest) with different characteristics and types, which consequently have different processing needs. Deciding which processing framework to use for a specific big data to perform a given task is usually a challenge for researchers from the disaster management field. Therefore, this paper ...

Distributed real-time computing has been the domain of practical system engineering for many decades. The development of a discipline of real-time programming would allow the construction of programs with analysable and variable timing properties. Such a discipline will need to be built on a well-integrated framework in which different methods are used where appropriate to obtain timing properties to which a high-level assurance can be attached. Time is everything in this current system, everything fast will gain the most valuable achievements especially in the business field. Whoever tries to get the first opportunity to fulfil the market needs will gain most profit. This paper looks into the various query engine models and frameworks that tries to improve on both the design and performance.

More and more use cases require fast, accurate, and reliable processing of large volumes of data. To do this, a distributed stream processing framework is needed which can distribute the load over several machines. In this work, we study and benchmark the scalability of stream processing jobs in four popular frameworks: Flink, Kafka Streams, Spark Streaming, and Structured Streaming. Besides that, we determine the factors that influence the performance and efficiency of scaling processing jobs with distinct characteristics. We evaluate horizontal, as well as vertical scalability. Our results show how the scaling efficiency is impacted by many factors including the initial cluster layout and direction of scaling, the pipeline design, the framework design, resource allocation, and data characteristics. Finally, we give some recommendations on how practitioners should undertake to scale their clusters.

The past decade saw an exponential rise in the amount of information available on the World Wide Web. Almost every business organization today uses web based technology to wield its huge client base. Consequently, managing the large data and mining pertinent content has become the need of the hour. This is where the field of big data analytics sows its seeds. The linchpin for this is the process of knowledge discovery. Analyzing server logs and other data footprints, aggregated from clients, can facilitate the building of a concrete knowledge base. Querying the knowledge base can help supplement business and other managerial decisions. The approach herein proposes a real time, generalized alternative to log file management and analysis. It incorporates the development of a sustainable platform which would enable the analysts to understand the essence of the data available.

Most high-performance data processing (a.k.a. big data) systems allow users to express their computation using abstractions (like MapReduce), which simplify the extraction of parallelism from applications. Most frameworks, however, do not allow users to specify how communication must take place: That element is deeply embedded into the run-time system abstractions, making changes hard to implement. In this work, we describe Wathershed-ng, our re-engineering of the Watershed system, a framework based on the filter-stream paradigm and originally focused on continuous stream processing. Like other big-data environments, Watershed provided object-oriented abstractions to express computation (filters), but the implementation of streams was a run-time system element. By isolating stream functionality into appropriate classes, combination of communication patterns and reuse of common message handling functions (like compression and blocking) become possible. The new architecture even allows the design of new communication patterns, for example, allowing users to choose MPI, TCP, or shared memory implementations of communication channels as their problem demands. Applications designed for the new interface showed reductions in code size on the order of 50% and above in some cases. The performance results also showed significant improvements, because some implementation bottlenecks were removed in the re-engineering process.

An anomaly (deviant objects, exceptions, peculiar objects) is an important concept of the analysis. The volume and velocity of the data within many systems makes it difficult to detect and process anomalies for Big Data in real-time. Many anomaly detective systems count on the historical data for detecting behaviors’. Considering it as a problem to financial institutions in Ghana, the researcher proposed robust anomaly detection framework. The proposed frame work defines Spark stream, as part of Spark ecosystem, which stream data in real-time. Also, the proposed framework data model was build using SVM, Linear regression and Logistic regression as a package found in Spark MLlib. Additionally, the proposed framework was explained clearly to be implemented in real systems for financial institutions.

Internet of Things (IoT) is a technology paradigm where millions of sensors monitor, and help inform or manage, physical, envi- ronmental and human systems in real-time. The inherent closed-loop re- sponsiveness and decision making of IoT applications makes them ideal candidates for using low latency and scalable stream processing plat- forms. Distributed Stream Processing Systems (DSPS) are becoming es- sential components of any IoT stack, but the efficacy and performance of contemporary DSPS have not been rigorously studied for IoT data streams and applications. Here, we develop a benchmark suite and per- formance metrics to evaluate DSPS for streaming IoT applications. The benchmark includes 13 common IoT tasks classified across various func- tional categories and forming micro-benchmarks, and two IoT applica- tions for statistical summarization and predictive analytics that leverage various dataflow compositional features of DSPS. These are coupled with stream workloads sourced ...

Internet of Things (IoT) is a technology paradigm where millions of sensors monitor, and help inform or manage, physical, environmental and human systems in real-time. The inherent closed-loop responsiveness and decision making of IoT applications makes them ideal candidates for using low latency and scalable stream processing platforms. Distributed Stream Processing Systems (DSPS) are becoming essential components of any IoT stack, but the efficacy and performance of contemporary DSPS have not been rigorously studied for IoT data streams and applications. Here, we develop a benchmark suite and performance metrics to evaluate DSPS for streaming IoT applications. The benchmark includes 13 common IoT tasks classified across various functional categories and forming micro-benchmarks, and two IoT applications for statistical summarization and predictive analytics that leverage various dataflow compositional features of DSPS. These are coupled with stream workloads sourced from real IoT observations from smart cities. We validate the IoT benchmark for the popular Apache Storm DSPS, and present empirical results.

In the Big Data era, stream processing has become a common requirement for many data-intensive applications. This has lead to many advances in the development and adaption of large scale streaming systems. Spark and Flink have become a popular choice for many developers as they combine both batch and streaming capabilities in a single system. However, introducing the Spark Structured Streaming in version 2.0 opened up completely new features for SparkSQL, which are alternatively only available in Apache Calcite. This work focuses on the new Spark Structured Streaming and analyses it by diving into its internal functionalities. With the help of a micro-benchmark consisting of streaming queries, we perform initial experiments evaluating the technology. Our results show that Spark Structured Streaming is able to run multiple queries successfully in parallel on data with changing velocity and volume sizes.

With the ever increasing number of IoT devices getting connected, an enormous amount of streaming data is being produced with very high velocity. In order to process these large number of data streams, a variety of stream processing platforms and query engines are emerging. In the stream query processing, an infinite data stream is divided into small chunks of finite data using a window operator. Window size and its type play an important role in the performance of any stream query engine. Due to the dynamic nature of IoT, data stream rate fluctuates very often, thus impeding the performance of query engines. In this work, we investigated the impact of any changes in data stream rates over the performance of a distributed query engine (e.g. Flink - https://flink.apache.org/). Our evaluation results indicate a direct impact of any changes in stream rate and window size over the performance of the engines. We propose an adaptive and dynamic query window size and type selector to impro...

An anomaly (deviant objects, exceptions, peculiar objects) is an important concept of the analysis. The volume and velocity of the data within many systems makes it difficult to detect and process anomalies for Big Data in real-time. Many anomaly detective systems count on the historical data for detecting behaviors'. Considering it as a problem to financial institutions in Ghana, the researcher proposed robust anomaly detection framework. The proposed frame work defines Spark stream, as part of Spark ecosystem, which stream data in real-time. Also, the proposed framework data model was build using SVM, Linear regression and Logistic regression as a package found in Spark MLlib. Additionally, the proposed framework was explained clearly to be implemented in real systems for financial institutions.

Advances in information technology have facilitated large volume, high-velocity of data, and the ability to store data continuously leading to several computational challenges. Due to the nature of big data in terms of volume, velocity, variety, variability, veracity, volatility, and value [1] that are being generated recently, big data computing is a new trend for future computing. Big data computing can be generally categorized into two types based on the processing requirements, which are big data batch computing and big data stream computing Abstract Recently, big data streams have become ubiquitous due to the fact that a number of applications generate a huge amount of data at a great velocity. This made it difficult for existing data mining tools, technologies, methods, and techniques to be applied directly on big data streams due to the inherent dynamic characteristics of big data. In this paper, a systematic review of big data streams analysis which employed a rigorous and methodical approach to look at the trends of big data stream tools and technologies as well as methods and techniques employed in analysing big data streams. It provides a global view of big data stream tools and technologies and its comparisons. Three major databases, Scopus, ScienceDirect and EBSCO, which indexes journals and conferences that are promoted by entities such as IEEE, ACM, SpringerLink, and Elsevier were explored as data sources. Out of the initial 2295 papers that resulted from the first search string, 47 papers were found to be relevant to our research questions after implementing the inclusion and exclusion criteria. The study found that scalability, privacy and load balancing issues as well as empirical analysis of big data streams and technologies are still open for further research efforts. We also found that although, significant research efforts have been directed to real-time analysis of big data stream not much attention has been given to the preprocessing stage of big data streams. Only a few big data streaming tools and technologies can do all of the batch, streaming, and iterative jobs; there seems to be no big data tool and technology that offers all the key features required for now and standard benchmark dataset for big data streaming analytics has not been widely adopted. In conclusion, it was recommended that research efforts should be geared towards developing scalable frameworks and algorithms that will accommodate data stream computing mode, effective resource allocation strategy and parallelization issues to cope with the ever-growing size and complexity of data.

CloudWave will revolutionise modern cloud infrastructures and tools by enabling agile development and delivery of adaptive cloud services which dynamically adjust to changes in their environment so as to optimise service quality and resource utilization. In doing so, CloudWave will advance the competitiveness of the EU economy by providing a powerful foundation, based on open standards, for creating innovative Future Internet services, realising the full promise of cloud computing. CloudWave will tangibly deliver (1) an open architecture and standards-based reference implementation of an advanced cloud software stack, with novel capabilities for adaptation across all cloud layers; and (2) tools and methods for agile development of reliable and adaptable cloud services, facilitated by the new stack. Technologically, CloudWave will advance the state-of-art along three dimensions: •Execution Analytics: A new framework where specialised algorithms dynamically analyse cloud infrastructur...

An anomaly (deviant objects, exceptions, peculiar objects) is an important concept of the analysis. The volume and velocity of the data within many systems makes it difficult to detect and process anomalies for Big Data in real-time. Many anomaly detective systems count on the historical data for detecting behaviors'. Considering it as a problem to financial institutions in Ghana, the researcher proposed robust anomaly detection framework. The proposed frame work defines Spark stream, as part of Spark ecosystem, which stream data in real-time. Also, the proposed framework data model was build using SVM, Linear regression and Logistic regression as a package found in Spark MLlib. Additionally, the proposed framework was explained clearly to be implemented in real systems for financial institutions.

An anomaly (deviant objects, exceptions, peculiar objects) is an important concept of the analysis. The volume and velocity of the data within many systems makes it difficult to detect and process anomalies for Big Data in real-time. Many anomaly detective systems count on the historical data for detecting behaviors'. Considering it as a problem to financial institutions in Ghana, the researcher proposed robust anomaly detection framework. The proposed frame work defines Spark stream, as part of Spark ecosystem, which stream data in real-time. Also, the proposed framework data model was build using SVM, Linear regression and Logistic regression as a package found in Spark MLlib. Additionally, the proposed framework was explained clearly to be implemented in real systems for financial institutions.

Many important "big data" applications need to process data arriving in real time. However, current programming models for distributed stream processing are relatively low-level, often leaving the user to worry about consistency of state across the system and fault recovery. Furthermore, the models that provide fault recovery do so in an expensive manner, requiring either hot replication or long recovery times. We propose a new programming model, discretized streams (D-Streams), that offers a high-level functional programming API, strong consistency, and efficient fault recovery. D-Streams support a new recovery mechanism that improves efficiency over the traditional replication and upstream backup solutions in streaming databases: parallel recovery of lost state across the cluster. We have prototyped D-Streams in an extension to the Spark cluster computing framework called Spark Streaming, which lets users seamlessly intermix streaming, batch and interactive queries.