Spark Ecosystem

Large-scale datasets are becoming more common, yet they can be challenging to understand and interpret. When dealing with big datasets, principal component analysis (PCA) is used to minimize the dimensionality of the data while maintaining interpretability and avoiding information loss. It accomplishes this by producing new uncorrelated variables that gradually reduce the variance of the system. In the field of data analysis, PCA is a multivariate statistical technique commonly used to obtain rules explaining the separation of groups in a given situation. Classes are predicted using a classification algorithm, a supervised learning technique that indicates which type of data points will be presented. Creating a classification model using classification algorithms is required before any successful classification can be achieved. It is possible to predict the future using a variety of categorized strategies. It is necessary to reduce the dimensionality of data sets using the PCA appro...

Many organizations are shifting to a data management paradigm called the "Lakehouse," which implements the functionality of structured data warehouses on top of unstructured data lakes. This presents new challenges for query execution engines. The engine needs to provide good performance on the raw uncurated datasets that are ubiquitous in data lakes, and excellent performance on structured data stored in popular columnar file formats like Apache Parquet. Toward these goals, we present Photon, a vectorized query engine for Lakehouse environments that we developed at Databricks. Photon can outperform existing warehouses on SQL workloads and also supports the Apache Spark API. We discuss the design choices we made in Photon (e.g., vectorization vs. code generation) and describe its integration with our existing SQL and Apache Spark runtimes, its task model, and its memory manager. Photon has accelerated some customer workloads by over 10× and has recently allowed Databricks to set a new audited performance record for the official 100TB TPC-DS benchmark. • Information systems → Database management system engines; Main memory engines; Database query processing.

Large-scale datasets are becoming more common, yet they can be challenging to understand and interpret. When dealing with big datasets, principal component analysis (PCA) is used to minimize the dimensionality of the data while maintaining interpretability and avoiding information loss. It accomplishes this by producing new uncorrelated variables that gradually reduce the variance of the system. In the field of data analysis, PCA is a multivariate statistical technique commonly used to obtain rules explaining the separation of groups in a given situation. Classes are predicted using a classification algorithm, a supervised learning technique that indicates which type of data points will be presented. Creating a classification model using classification algorithms is required before any successful classification can be achieved. It is possible to predict the future using a variety of categorized strategies. It is necessary to reduce the dimensionality of data sets using the PCA appro...

Large-scale datasets are becoming more common, yet they can be challenging to understand and interpret. When dealing with big datasets, principal component analysis (PCA) is used to minimize the dimensionality of the data while maintaining interpretability and avoiding information loss. It accomplishes this by producing new uncorrelated variables that gradually reduce the variance of the system. In the field of data analysis, PCA is a multivariate statistical technique commonly used to obtain rules explaining the separation of groups in a given situation. Classes are predicted using a classification algorithm, a supervised learning technique that indicates which type of data points will be presented. Creating a classification model using classification algorithms is required before any successful classification can be achieved. It is possible to predict the future using a variety of categorized strategies. It is necessary to reduce the dimensionality of data sets using the PCA appro...

Large-scale datasets are becoming more common, yet they can be challenging to understand and interpret. When dealing with big datasets, principal component analysis (PCA) is used to minimize the dimensionality of the data while maintaining interpretability and avoiding information loss. It accomplishes this by producing new uncorrelated variables that gradually reduce the variance of the system. In the field of data analysis, PCA is a multivariate statistical technique commonly used to obtain rules explaining the separation of groups in a given situation. Classes are predicted using a classification algorithm, a supervised learning technique that indicates which type of data points will be presented. Creating a classification model using classification algorithms is required before any successful classification can be achieved. It is possible to predict the future using a variety of categorized strategies. It is necessary to reduce the dimensionality of data sets using the PCA appro...

Large-scale datasets are becoming more common, yet they can be challenging to understand and interpret. When dealing with big datasets, principal component analysis (PCA) is used to minimize the dimensionality of the data while maintaining interpretability and avoiding information loss. It accomplishes this by producing new uncorrelated variables that gradually reduce the variance of the system. In the field of data analysis, PCA is a multivariate statistical technique commonly used to obtain rules explaining the separation of groups in a given situation. Classes are predicted using a classification algorithm, a supervised learning technique that indicates which type of data points will be presented. Creating a classification model using classification algorithms is required before any successful classification can be achieved. It is possible to predict the future using a variety of categorized strategies. It is necessary to reduce the dimensionality of data sets using the PCA appro...

Bioinformatics is an emerging interdisciplinary research area that deals with the computational management and analysis of biological information. Genomics is the most important domain in bioinformatics which compares genomic features like DNA sequences, genes, regulatory sequences, or other genomic structural components etc. of different organisms. Computers are used to gather, store, analyze and integrate biological and genetic information which can then be applied to gene-based drug discovery and development. Scientists may require weeks or months if they use their own workstations since biological big data is generated by several different bioinformatics/biological/biomedical experiments and it can be presented as structured or unstructured data. Each cell in the body contains a whole genome, yet the data packed into a few DNA molecules could fill a hard drive. Biological big data is now reaching the size of Terabytes, Petabytes and exa bytes and the different modes of represent...

Motor development is an important factor affecting the health status physically, psychologically and socially in both childhood and adulthood. It is important to develop motor skills starting from childhood and to participate in a variety of physical activities for this. In this study, it is aimed to reveal the motor development of 7-10 years old boys who receive multi-skill movement training outside of school and to examine the developmental differences. 61 volunteer college students participated in the study, the sample was divided into 31 students as a control group and 30 students as an intervention group receiving multi-skill movement training outside of school. The Test of Gross Motor Development-2 test, which includes locomotor and object control skill subtests, was applied to monitor the motor development of the children participating in the study. Gross motor skill scores from these two subtests were evaluated. At the end of the 12-week movement program, the study group showed a statistically significant improvement in gross motor skills, object control and locomotor sub-skills compared to the control group. A statistically significant improvement was found only in the object control sub-skill in the control group, whose in-school movement education continued. As a result, physical education activities at schools should be evaluated in terms of their duration and content, and children's participation in physical activities that show diversity should be evaluated in terms of motor development.

In the field of network security, the task of processing and analyzing huge amount of Packet CAPture (PCAP) data is of utmost importance for developing and monitoring the behavior of networks, having an intrusion detection and prevention system, firewall etc. In recent times, Apache Spark in combination with Hadoop Yet-Another-Resource-Negotiator (YARN) is evolving as a generic Big Data processing platform. While processing raw network packets, timely inference of network security is a primitive requirement. However, to the best of our knowledge, no prior work has focused on systematic study on fine-tuning the resources, scalability and performance of distributed Apache Spark cluster (while processing PCAP data). For obtaining best performance, various cluster parameters like number of cluster nodes, number of cores utilized from each node, total number of executors run in the cluster, amount of main-memory used from each node, executor memory overhead allotted for each node to handle garbage collection issue, etc., have been finetuned, which is the focus of the proposed work. Through the proposed strategy, we could analyze 85GB of data (provided by CSIR Fourth Paradigm Institute) in just 78 seconds, using 32 node (256 cores) Spark cluster. This would otherwise take around 30 minutes in traditional processing systems.

Large-scale datasets are becoming more common, yet they can be challenging to understand and interpret. When dealing with big datasets, principal component analysis (PCA) is used to minimize the dimensionality of the data while maintaining interpretability and avoiding information loss. It accomplishes this by producing new uncorrelated variables that gradually reduce the variance of the system. In the field of data analysis, PCA is a multivariate statistical technique commonly used to obtain rules explaining the separation of groups in a given situation. Classes are predicted using a classification algorithm, a supervised learning technique that indicates which type of data points will be presented. Creating a classification model using classification algorithms is required before any successful classification can be achieved. It is possible to predict the future using a variety of categorized strategies. It is necessary to reduce the dimensionality of data sets using the PCA appro...

Usage of big data which is related to medical filed is gaining popularity among healthcare services and for clinical research. Medical field is one of the largest areas which is generating enormous amount and varieties of data. Traditional systems are incapable of handling such big data which is characterized by volume, variety, velocity, veracity and values (5 V’s). To process this vast amount of data we need a framework which can parallel process the data by utilizing the clusters of commodity hardware. This hardware should be reliable, faulttolerant. Apache Spark is a fast, in-memory data processing engine with elegant and expressive development APIs to allow data workers to efficiently execute streaming, machine learning or SQL workloads that require fast iterative access to datasets. In the Hadoop framework we can develop MapReduce applications which can scale up from single node to thousands of machines. This paper investigates the big data which is used in clinical research t...

Large-scale datasets are becoming more common, yet they can be challenging to understand and interpret. When dealing with big datasets, principal component analysis (PCA) is used to minimize the dimensionality of the data while maintaining interpretability and avoiding information loss. It accomplishes this by producing new uncorrelated variables that gradually reduce the variance of the system. In the field of data analysis, PCA is a multivariate statistical technique commonly used to obtain rules explaining the separation of groups in a given situation. Classes are predicted using a classification algorithm, a supervised learning technique that indicates which type of data points will be presented. Creating a classification model using classification algorithms is required before any successful classification can be achieved. It is possible to predict the future using a variety of categorized strategies. It is necessary to reduce the dimensionality of data sets using the PCA appro...

Big data have acquired big attention in recent years. As big data makes its way into companies and business so there are some challenges in big data analytics. Apache spark framework becomes very popular for using in distributed data processing. Spark is an analytic machine for big data processing with various modules for SQL, streaming, graph processing and machine learning. Different scheduling algorithms vary with its behavior, design and also the goal required solving a problem like data locality, energy and time. The main goal in this research is to represent a comprehensive survey on job scheduling modes using in spark, the types of different scheduler, and existing algorithms with advantages and issues. In this paper, various adaptive ways to schedule jobs on spark and development algorithms to improve performance in Spark will be discussed, analyzed and evaluated. A comparison between different scheduling algorithms, strength and weakness points of them are provided. This can aid to the researchers understanding of which scheduling mechanisms best applied for Big Data.

Community detection is an important research topic in graph analytics that has a wide range of applications. A variety of static community detection algorithms and quality metrics were developed in the past few years. However, most real-world graphs are not static and often change over time. In the case of streaming data, communities in the associated graph need to be updated either continuously or whenever new data streams are added to the graph, which poses a much greater challenge in devising good community detection algorithms for maintaining dynamic graphs over streaming data. In this paper, we propose an incremental community detection algorithm for maintaining a dynamic graph over streaming data. The contributions of this study include (a) the implementation of a Distributed Weighted Community Clustering (DWCC) algorithm, (b) the design and implementation of a novel Incremental Distributed Weighted Community Clustering (IDWCC) algorithm, and (c) an experimental study to compare the performance of our IDWCC algorithm with the DWCC algorithm. We validate the functionality and efficiency of our framework in processing streaming data and performing large in-memory distributed dynamic graph analytics. The results demonstrate that our IDWCC algorithm performs up to three times faster than the DWCC algorithm for a similar accuracy.

In the field of network security, the task of processing and analyzing huge amount of Packet CAPture (PCAP) data is of utmost importance for developing and monitoring the behavior of networks, having an intrusion detection and prevention system, firewall etc. In recent times, Apache Spark in combination with Hadoop Yet-Another-Resource-Negotiator (YARN) is evolving as a generic Big Data processing platform. While processing raw network packets, timely inference of network security is a primitive requirement. However, to the best of our knowledge, no prior work has focused on systematic study on fine-tuning the resources, scalability and performance of distributed Apache Spark cluster (while processing PCAP data). For obtaining best performance, various cluster parameters like number of cluster nodes, number of cores utilized from each node, total number of executors run in the cluster, amount of main-memory used from each node, executor memory overhead allotted for each node to handle garbage collection issue, etc., have been finetuned, which is the focus of the proposed work. Through the proposed strategy, we could analyze 85GB of data (provided by CSIR Fourth Paradigm Institute) in just 78 seconds, using 32 node (256 cores) Spark cluster. This would otherwise take around 30 minutes in traditional processing systems.

The nuclear industry is experiencing a steady increase in maintenance costs even though plants are maintained under high levels of safety, capability and reliability. Nuclear power plants are expected to run every unit at maximum capacity at all times, efficiently utilizing assets with minimal downtime. Surveillance and maintenance of nuclear decommissioning infrastructure provides lot challenges with respect to maintenance or decommissioning of the buildings. There is a need for a framework to analyze the huge amount of data generated by the sensors on the nuclear reactor components as well as structures, to monitor the conditions of these building over a period of time. Emerging technologies such as big data analytics have become a requirement in the nuclear industry to improve structural health monitoring and diagnostics. FIU will make use of existing mature technologies in the areas of imaging, robotics, big data [1], and machine learning/deep learning [2] [3] [4] to assess the structural integrity of aging facilities at DOE sites. As these facilities await decommissioning, there is a need to understand the structural health of these structures. Many of these facilities were built over 50 years ago and in some cases these facilities have gone beyond the operational life expectancy. In other cases, the facilities have been placed in a state of "cold and dark" and they are sitting unused, awaiting decommissioning. Finally, some older facilities are one-of-a-kind operational/production facilities supporting critical DOE missions and cannot be shut down. In any of these scenarios, the structural integrity of these facilities may be compromised, so it is imperative that adequate inspections and data collection/analysis be performed on a continuous and ongoing basis. The primary goals of the research include collecting various formats of data such as structured, un-structured and streaming data from the various sensors deployed in buildings, collect videos/pictures from various imaging sources, ingest them using a Hadoop [5] distributed file system and process the data using Spark [6, 7] to perform batch processing and real time analytics. Research and development on various machine learning/deep learning algorithms will also be performed to analyze the heterogeneous data collected from nuclear decommission infrastructure. FIU will design the big data framework to ingest, store and process huge amounts of heterogeneous data collected from many sources and optimize the algorithms to provide insights into the data and predict anomalies observed when compared against baseline conditions. Various modules of this framework will include heterogeneous data sources, message broker, Hadoop distributed file system, Spark for stream and batch processing, machine learning/ deep learning, Cassandrapersistent data store and visualization.

Distributed in-memory processing frameworks accelerate iterative workloads by caching suitable datasets in memory rather than recomputing them in each iteration. Selecting appropriate datasets to cache as well as allocating a suitable cluster configuration for caching these datasets play a crucial role in achieving optimal performance. In practice, both are tedious, time-consuming tasks and are often neglected by end users, who are typically not aware of workload semantics, sizes of intermediate data, and cluster specification. To address these problems, we present Juggler, an end-to-end framework, which autonomously selects appropriate datasets for caching and recommends a correspondingly suitable cluster configuration to end users, with the aim of achieving optimal execution time and cost. We evaluate Juggler on various iterative, real-world, machine learning applications. Compared with our baseline, Juggler reduces execution time to 25.1 % and cost to 58.1 %, on average, as a result of selecting suitable datasets for caching. It recommends optimal cluster configuration in 50 % of cases and near-to-optimal configuration in the remaining cases. Moreover, Juggler achieves an average performance prediction accuracy of 90 %.

Large-scale datasets are becoming more common, yet they can be challenging to understand and interpret. When dealing with big datasets, principal component analysis (PCA) is used to minimize the dimensionality of the data while maintaining interpretability and avoiding information loss. It accomplishes this by producing new uncorrelated variables that gradually reduce the variance of the system. In the field of data analysis, PCA is a multivariate statistical technique commonly used to obtain rules explaining the separation of groups in a given situation. Classes are predicted using a classification algorithm, a supervised learning technique that indicates which type of data points will be presented. Creating a classification model using classification algorithms is required before any successful classification can be achieved. It is possible to predict the future using a variety of categorized strategies. It is necessary to reduce the dimensionality of data sets using the PCA appro...

Large-scale datasets are becoming more common, yet they can be challenging to understand and interpret. When dealing with big datasets, principal component analysis (PCA) is used to minimize the dimensionality of the data while maintaining interpretability and avoiding information loss. It accomplishes this by producing new uncorrelated variables that gradually reduce the variance of the system. In the field of data analysis, PCA is a multivariate statistical technique commonly used to obtain rules explaining the separation of groups in a given situation. Classes are predicted using a classification algorithm, a supervised learning technique that indicates which type of data points will be presented. Creating a classification model using classification algorithms is required before any successful classification can be achieved. It is possible to predict the future using a variety of categorized strategies. It is necessary to reduce the dimensionality of data sets using the PCA approach. This article will begin by introducing the basic ideas of PCA and discussing what it can and cannot do. It will then describe some variants of PCA and their applications, and then show how PCA improves the performance using a series of experiments.

Recently, efforts have been made to bring together the areas of high-performance computing (HPC) and massive data processing (Big Data). Traditional HPC frameworks, like COMPSs, are mostly task-based, while popular big-data environments, like Spark, are based on functional programming principles. The earlier are know for their good performance for regular, matrix-based computations; on the other hand, for fine-grained, data-parallel workloads, the later has often been considered more successful. In this paper we present our experience with the integration of some dataflow techniques into COMPSs, a task-based framework, in an effort to bring together the best aspects of both worlds. We present our API, called DDF, which provides a new data abstraction that addresses the challenges of integrating Big Data application scenarios into COMPSs. DDF has a functional-based interface, similar to many Data Science tools, that allows us to use dynamic evaluation to adapt the task execution in runtime. Besides the performance optimization it provides, the API facilitates the development of applications by experts in the application domain. In this paper we evaluate DDF's effectiveness by comparing the resulting programs to their original versions in COMPSs and Spark. The results show that DDF can improve COMPSs execution time and even outperform Spark in many use cases.

The complexity of Big Data analytics has long outreached the capabilities of current platforms, which fail to efficiently cope with the data and task heterogeneity of modern workflows due to their adhesion to a single data and/or compute model. As a remedy, we demonstrate IReS, the Intelligent Resource Scheduler for complex analytics workflows executed over multi-engine environments. IReS is able to optimize a workflow with respect to a user-defined policy by (a) allocating distinct parts of it to the most advantageous execution and/or storage engine among the available ones and (b) deciding on the exact amount of resources provisioned. Moreover, IReS can efficiently adapt to the current cluster/engine conditions and recover from failures by effectively monitoring the workflow execution in real-time. During the demo, the attendees will be able to create, optimize and execute workflows that match real use cases over multiple compute and data engines, imposing their preferred optimiza...

Acknowledgement and Disclaimer This publication is based upon work from the COST Action IC1406 High-Performance Modelling and Simulation for Big Data Applications (cHiPSet), supported by COST (European Cooperation in Science and Technology). The book reflects only the authors' views. Neither the COST Association nor any person acting on its behalf is responsible for the use, which might be made of the information contained in this publication. The COST Association is not responsible for external websites or sources referred to in this publication.

Access plan recommendation is a query optimization approach that executes new queries using prior created query execution plans (QEPs). The query optimizer divides the query space into clusters in the mentioned method. However, traditional clustering algorithms take a significant amount of execution time for clustering such large datasets. The MapReduce distributed computing model provides efficient solutions for storing and processing vast quantities of data. Apache Spark and Apache Hadoop frameworks are used in the present investigation to cluster different sizes of query datasets in the MapReduce-based access plan recommendation method. The performance evaluation is performed based on execution time. The results of the experiments demonstrated the effectiveness of parallel query clustering in achieving high scalability. Furthermore, Apache Spark achieved better performance than Apache Hadoop, reaching an average speedup of 2x.

There is a need to integrate SQL processing with more advanced machine learning (ML) analytics to drive actionable insights from large volumes of data. As a first step towards this integration, we study how to efficiently connect big SQL systems (either MPP databases or new-generation SQL-on-Hadoop systems) with distributed big ML systems. We identify two important challenges to address in the integrated data analytics pipeline: data transformation, how to efficiently transform SQL data into a form suitable for ML, and data transfer, how to efficiently handover SQL data to ML systems. For the data transformation problem, we propose an In-SQL approach to incorporate common data transformations for ML inside SQL systems through extended user-defined functions (UDFs), by exploiting the massive parallelism of the big SQL systems. We propose and study a general method for transferring data between big SQL and big ML systems in a parallel streaming fashion. Furthermore, we explore caching...

The hardware in the UMBC High Performance Computing Facility (HPCF) is supported by the U.S. National Science Foundation through the MRI program (grant nos. CNS–0821258, CNS–1228778, and OAC–1726023) and the SCREMS program (grant no. DMS–0821311), with additional substantial support from the University of Maryland, Baltimore County (UMBC). We would like to extend special thanks to our instructor Dr. Jianwu Wang for supporting us.

Lakehouse systems have reached in the past few years unprecedented size and heterogeneity and have been embraced by many industry players. However, they are often difficult to use as they lack the declarative language and optimization possibilities of relational engines. This paper introduces RumbleML, a high-level, declarative library integrated into the RumbleDB engine and with the JSONiq language. RumbleML allows using a single platform for data cleaning, data preparation, training, and inference, as well as management of models and results. It does it using a purely declarative language (JSONiq) for all these tasks and without any performance loss over existing platforms (e.g. Spark). The key insights of the design of RumbleML are that training sets, evaluation sets, and test sets can be represented as homogeneous sequences of flat objects; that models can be seamlessly embodied in function items mapping input test sets into prediction-augmented result sets; and that estimators ...

Healthcare informatics is undergoing a revolution because of the availability of safe, wearable sensors at low cost. Smart hospitals have exploited the development of the Internet of Things (IoT) sensors to create Remote Patients monitoring (RPM) models that observe patients at their homes. RPM is one of the Ambient Assisted Living (AAL) applications. The long-term monitoring of patients using the AALs generates big data. Therefore, AALs must adopt cloud-based architectures to store, process and analyze big data. The usage of big data analytics for handling and analyzing the massive amount of big medical data will make a big shift in the healthcare field. Advanced software frameworks such as Hadoop will promote the success of medical assistive applications because it allows the storage of data in its native form not only in the form of electronic medical records that can be stored in data warehouses. Also, Spark and its machine learning libraries accelerate the analysis of big medic...

In the field of network security, the task of processing and analyzing huge amount of Packet CAPture (PCAP) data is of utmost importance for developing and monitoring the behavior of networks, having an intrusion detection and prevention system, firewall etc. In recent times, Apache Spark in combination with Hadoop Yet-Another-Resource-Negotiator (YARN) is evolving as a generic Big Data processing platform. While processing raw network packets, timely inference of network security is a primitive requirement. However, to the best of our knowledge, no prior work has focused on systematic study on fine-tuning the resources, scalability and performance of distributed Apache Spark cluster (while processing PCAP data). For obtaining best performance, various cluster parameters like number of cluster nodes, number of cores utilized from each node, total number of executors run in the cluster, amount of main-memory used from each node, executor memory overhead allotted for each node to handle garbage collection issue, etc., have been finetuned, which is the focus of the proposed work. Through the proposed strategy, we could analyze 85GB of data (provided by CSIR Fourth Paradigm Institute) in just 78 seconds, using 32 node (256 cores) Spark cluster. This would otherwise take around 30 minutes in traditional processing systems.

Interventions that can successfully alter the trajectory toward obesity among high-risk children are critical if we are to effectively address this public health crisis. The goal of this pilot study was to implement and evaluate an innovative physical activity program with Hispanic-American (HA) preschool children attending Head Start. The Children in Action (CIA) program was a five month physical activity intervention. This intervention was a pilot study with 3-to 5year-olds enrolled in four HA Head Start centers. After baseline assessment, centers were matched by enrollment and randomly assigned to either the intervention or the control condition. A total of 295 preschool children were randomly selected across the four centers. The primary endpoints of this study were favorable changes in physical activity levels and gross motor skills. Using mixed effect time-series regression models, changes in weight was a secondary endpoint. We did not observe a statistical difference between intervention and control groups in physical activity levels during the awake time, gross motor skills, or weight status. Process evaluation data showed that there was adherence to protocols and the intervention was delivered 92% of the time, four times per week, during the five month intervention. We demonstrated that it is feasible to conduct the SPARK-Early Childhood (EC) curriculum among preschool children attending Head Start centers but that an increased dose and/or longer intervention duration will be required to impact gross motor skills, physical activity levels and weight status during this critical early childhood development stage.

I would like to express sincere thanks to my supervisor Ing. Adam Šenk for his helpful advices and comments that helped me to finish this master's thesis. Also I would like to thank Prof. Dr. Wolfgang Benn and Johannes Fliege from Technische Universität Chemnitz, Faculty of Computer Science for all their help and the opportunity to work on my master's thesis abroad at university in Chemnitz. Last but not the least, I would like to express my heartfelt gratitude to my parents, all my family and friends for the support not only during the work on the thesis, but during my whole university study.

There is a need to integrate SQL processing with more advanced machine learning (ML) analytics to drive actionable insights from large volumes of data. As a first step towards this integration, we study how to efficiently connect big SQL systems (either MPP databases or new-generation SQL-on-Hadoop systems) with distributed big ML systems. We identify two important challenges to address in the integrated data analytics pipeline: data transformation, how to efficiently transform SQL data into a form suitable for ML, and data transfer, how to efficiently handover SQL data to ML systems. For the data transformation problem, we propose an In-SQL approach to incorporate common data transformations for ML inside SQL systems through extended user-defined functions (UDFs), by exploiting the massive parallelism of the big SQL systems. We propose and study a general method for transferring data between big SQL and big ML systems in a parallel streaming fashion. Furthermore, we explore caching intermediate or final results of data transformation to improve the performance. Our techniques are generic: they apply to any big SQL system that supports UDFs and any big ML system that uses Hadoop InputFormats to ingest input data.

We introduce GraphFlow, a big graph framework that is able to encode complex data science experiments as a set of high-level workflows. GraphFlow combines the Spark big data processing platform and the Galaxy workflow management system to offer a set of components for graph processing using a novel interaction model for creating and using complex workflows. GraphFlow contributes an easy-to-use interface and scalable algorithms for big graph analytics. We demonstrate GraphFlow use in large social network analysis with several case studies.

This paper introduces Rumble, an engine that executes JSONiq queries on large, heterogenous and nested collections of JSON objects, leveraging the parallel capabilities of Spark so as to provide a high degree of data independence. The design is based on two key insights: (i) how to map JSONiq expressions to Spark transformations on RDDs and (ii) how to map JSONiq FLWOR clauses to Spark SQL on DataFrames. We have developed a working implementation of these mappings showing that JSONiq can efficiently run on Spark to query billions of objects into, at least, the TB range. The JSONiq code is concise in comparison to Spark's host languages while seamlessly supporting the nested, heterogeneous datasets that Spark SQL does not. The ability to process this kind of input, commonly found, is paramount for data cleaning and curation. The experimental analysis indicates that there is no excessive performance loss, occasionally even a gain, over Spark SQL for structured data, and a performa...

Thanks to the huge amount of sequenced data that is becoming available, building scalable solutions for supporting query processing and data analysis over genomics datasets is increasingly important. This paper presents GDMS, a scalable Genomic Data Management System for querying region-based genomic datasets; the focus of the paper is on the deployment of the system on a cluster hosted by CINECA.

I would like to express sincere thanks to my supervisor Ing. Adam Šenk for his helpful advices and comments that helped me to finish this master's thesis. Also I would like to thank Prof. Dr. Wolfgang Benn and Johannes Fliege from Technische Universität Chemnitz, Faculty of Computer Science for all their help and the opportunity to work on my master's thesis abroad at university in Chemnitz. Last but not the least, I would like to express my heartfelt gratitude to my parents, all my family and friends for the support not only during the work on the thesis, but during my whole university study.

Smart cities use digital technologies such as cloud computing, Internet of Things, or open data in order to overcome limitations of traditional representation and exchange of geospatial data. This concept ensures a significant increase in the use of data to establish new services that contribute to better sustainable development and monitoring of all phenomena that occur in urban areas. The use of the modern geoinformation technologies, such as sensors for collecting different geospatial and related data, requires adequate storage options for further data analysis. In this paper, we suggest the biG dAta sMart cIty maNagEment SyStem (GAMINESS) that is based on the Apache Spark big data framework. The model of the GAMINESS management system is based on the principles of the big data modeling, which differs greatly from standard databases. This approach provides the ability to store and manage huge amounts of structured, semi-structured, and unstructured data in real time. System perfo...

Clustering is a fundamental task in Knowledge Discovery and Data mining. It aims to discover the unknown nature of data by grouping together data objects that are more similar. While hundreds of clustering algorithms have been proposed, many are complex and do not scale well as more data become available, making then inadequate to analyze very large datasets. In addition, many clustering algorithms are sequential, thus inherently difficult to parallelize. We propose PatchWork, a novel clustering algorithm to address those issues. PatchWork is a distributed density clustering algorithm with linear computational complexity and linear horizontal scalability. It presents several desirable characteristics in knowledge discovery, in particular, it does not require a priori the number of clusters to identify, and offers a natural protection against outliers and noise. In addition, PatchWork makes it possible to discover spatially large clusters instead of dense clusters only. PatchWork relies on the map/reduce paradigm to parallelize computations and was implemented using Apache Spark, the distributed computation framework. As a result, PatchWork can cluster a billion points in a few minutes only, a 40x improvement over the distributed implementation of k-means in Spark MLLib.

The latest technological advances have allowed the development of smart home systems that establish a connection between humans and the devices that surround them, living in homes and working in fully automated companies. While these systems improve the quality of life of people in terms of comfort, safety and energy savings, as they automate aspects such as lighting, temperature, humidity, among others; they lack a Machine Learning system that manages the preferences, customs and behavior patterns of the inhabitants of the home or company they automate. The aforementioned lack shows an opportunity to improve these products, taking into account that the market trend in domestic devices, is to maximize automation, making them increasingly intelligent, to make decisions proactively, and to collaborate with each other and with the human being for his better quality of life. In this proposal it is shown that is possible to improve the management of the data stream captured by the devices, providing them with interoperability through the use of metadata, so that "intelligent modules" process the acquired knowledge automatically, with little intervention from the humans. Specifically, intends to use a Processing Architecture based on Measurement Metadata (PAbMM) and AWS Machine Learning and AWS IoT technology, to intelligently establish the environmental conditions of the home or trigger alarms, in order to improve more comfort, security and reduce energy consumption, on demand or proactively.

Today environment monitoring becomes important for humans to ensure a safe and wealthy life. Monitoring requirements are extremely different depending on the environment, leading to specially appointed usage that needs adaptability. The proposed system describes an implementation of Wireless Sensor Network that can be adjusted to various applications. And it also inserts the adaptability required to be conveyed and updated without necessity of arranging complex infrastructures. The solution is based on small autonomous wireless sensor nodes, small wireless receivers connected to the Internet, and a cloud architecture which provides data storage and delivery to remote clients. The solution permits supervisors on-site not only to monitor the current situation by using their smart-phones but also to monitor remote sites through the Internet. All measurements are stored at different levels to guarantee a safe back-trace and to access data stored in case of network failure or unavailability. The proposed system is useful in predicting environmental condition like rain, also, to measure the temperature, humidity, and presence of pressure.

The Big data is the name used ubiquitously now a day in distributed paradigm on the web. As the name point out it is the collection of sets of very large amounts of data in pet bytes, Exabyte etc. related systems as well as the algorithms used to analyze this enormous data. Hadoop technology as a big data processing technology has proven to be the go to solution for processing enormous data sets. MapReduce is a conspicuous solution for computations, which requirement one-pass to complete, but not exact efficient for use cases that need multi-pass for computations and algorithms. The Job output data between every stage has to be stored in the file system before the next stage can begin. Consequently, this method is slow, disk Input/output operations and due to replication. Additionally, Hadoop ecosystem doesn't have every component to ending a big data use case. Suppose we want to do an iterative job, you would have to stitch together a sequence of MapReduce jobs and execute them in sequence. Every this job has high-latency, and each depends upon the completion of the previous stage. Apache Spark is one of the most widely used open source processing engines for big data, with wealthy language-integrated APIs and an extensive range of libraries. Apache Spark is a usual framework for distributed computing that offers high performance for both batch and interactive processing. In this paper, we aimed to demonstrate a close-up view about Apache Spark and its features and working with Spark using Hadoop. We are in a nutshell discussing about the Resilient Distributed Datasets (RDD), RDD operations, features, and limitation. Spark can be used along with MapReduce in the same Hadoop cluster or can be used lonely as a processing framework. In the last comparative analysis between Spark and Hadoop and MapReduce in this paper.

The Big data is the name used ubiquitously now a day in distributed paradigm on the web. As the name point out it is the collection of sets of very large amounts of data in pet bytes, Exabyte etc. related systems as well as the algorithms used to analyze this enormous data. Hadoop technology as a big data processing technology has proven to be the go to solution for processing enormous data sets. MapReduce is a conspicuous solution for computations, which requirement one-pass to complete, but not exact efficient for use cases that need multi-pass for computations and algorithms. The Job output data between every stage has to be stored in the file system before the next stage can begin. Consequently, this method is slow, disk Input/output operations and due to replication. Additionally, Hadoop ecosystem doesn't have every component to ending a big data use case. Suppose we want to do an iterative job, you would have to stitch together a sequence of MapReduce jobs and execute them in sequence. Every this job has high-latency, and each depends upon the completion of the previous stage. Apache Spark is one of the most widely used open source processing engines for big data, with wealthy language-integrated APIs and an extensive range of libraries. Apache Spark is a usual framework for distributed computing that offers high performance for both batch and interactive processing. In this paper, we aimed to demonstrate a close-up view about Apache Spark and its features and working with Spark using Hadoop. We are in a nutshell discussing about the Resilient Distributed Datasets (RDD), RDD operations, features, and limitation. Spark can be used along with MapReduce in the same Hadoop cluster or can be used lonely as a processing framework. In the last comparative analysis between Spark and Hadoop and MapReduce in this paper.

The Big data is the name used ubiquitously now a day in distributed paradigm on the web. As the name point out it is the collection of sets of very large amounts of data in pet bytes, Exabyte etc. related systems as well as the algorithms used to analyze this enormous data. Hadoop technology as a big data processing technology has proven to be the go to solution for processing enormous data sets. MapReduce is a conspicuous solution for computations, which requirement one-pass to complete, but not exact efficient for use cases that need multi-pass for computations and algorithms. The Job output data between every stage has to be stored in the file system before the next stage can begin. Consequently, this method is slow, disk Input/output operations and due to replication. Additionally, Hadoop ecosystem doesn't have every component to ending a big data use case. Suppose we want to do an iterative job, you would have to stitch together a sequence of MapReduce jobs and execute them in sequence. Every this job has high-latency, and each depends upon the completion of the previous stage. Apache Spark is one of the most widely used open source processing engines for big data, with wealthy language-integrated APIs and an extensive range of libraries. Apache Spark is a usual framework for distributed computing that offers high performance for both batch and interactive processing. In this paper, we aimed to demonstrate a close-up view about Apache Spark and its features and working with Spark using Hadoop. We are in a nutshell discussing about the Resilient Distributed Datasets (RDD), RDD operations, features, and limitation. Spark can be used along with MapReduce in the same Hadoop cluster or can be used lonely as a processing framework. In the last comparative analysis between Spark and Hadoop and MapReduce in this paper.

—This study was conducted based on an assumption that Spark ML package has much better performance and accuracy than Spark MLlib package in dealing with big data. The used dataset in the comparison is for bank customers transactions. The Decision tree algorithm was used with both packages to generate a model for predicting the churn probability for bank customers depending on their transactions data. Detailed comparison results were recorded and conducted that the ML package and its new DataFrame-based APIs have better-evaluating performance and predicting accuracy.

The emerging phenomenon called ―Big Data‖ is pushing numerous changes in businesses and several other organizations, Domains, Fields, areas etc. Many of them are struggling just to manage the massive data sets. Big data management is about two things-―Big data‖ and ―Data Management‖ and these terms work together to achieve business and technology goals as well. In previous few years data generation have tremendously enhanced due to digitization of data. Day by day new computer tools and technologies for transmission of data among several computers through Internet is been increasing. It‗s relevance and importance in the context of applicability, usefulness for decision making, performance improvement etc in all areas have emerged very fast to be relevant in today‗s era. Big data management also has numerous challenges and common complexities include low organizational maturity relative to big data, weak business support, and the need to learn new technology approaches. This paper will discuss the impacts of Big Data and issues related to data management using current technologies.