Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Research Methods
Extracting Anti-Patterns for Big Data Applications
Results: Ostia Explained
Architecture Overview
Ostia Methodology
Topology Design Anti-Patterns Within Ostia
Persistent Data
Results
Discussion
Findings and Observations
Approach Limitations and Threats to Validity
Related Work
Conclusion
References

Verifying big data topologies by-design: a semi-automated approach

Profile image of Damian Andrew TamburriDamian Andrew Tamburri

Journal of Big Data

https://doi.org/10.1186/S40537-019-0199-Y
visibility

…

description

23 pages

link

1 file

Abstract

Big data or data-intensive applications (DIAs) process large amounts of data for the purpose of gaining key business intelligence through complex analytics using machinelearning techniques [20, 35]. These applications are receiving increased attention in the last years given their ability to yield competitive advantage by direct investigation of user needs and trends hidden in the enormous quantities of data produced daily by the average Internet user. According to Gartner [1] business intelligence and analytics applications will remain a top focus for Chief-Information Officers (CIOs) of most Fortune 500 companies until at least 2019-2021. However, the cost of ownership of the systems that process big data analytics are high due to infrastructure costs, steep learning curves for the different frameworks (such as Apache Storm [21], Apache Spark [2] or Apache Hadoop [3]) typically involved in design and development of big data applications and complexities in large-scale architectures. A key complexity of the above design and development activity lies in quickly and continuously refining the configuration parameters of the middleware and service platforms on top of which the DIA is running [12]. The process in question is especially complex as the number of middleware involved in DIAs design increases; the more middleware are involved the more parameters need co-evaluation (e.g., latency or beaconing times, caching policies, queue retention and more)-fine-tuning these "knobs" on so many

Related papers

Big Data and Cloud Computing: Trends and Challenges
Hosam El-Sofany

International Journal of Interactive Mobile Technologies (iJIM), 2017

Big data is currently one of the most critical emerging technologies. Big Data are used as a concept that refers to the inability of traditional data architectures to efficiently handle the new data sets. The 4V’s of big data – volume, velocity, variety and veracity makes the data management and analytics challenging for the traditional data warehouses. It is important to think of big data and analytics together. Big data is the term used to describe the recent explosion of different types of data from disparate sources. Analytics is about examining data to derive interesting and relevant trends and patterns, which can be used to inform decisions, optimize processes, and even drive new business models. Cloud computing seems to be a perfect vehicle for hosting big data workloads. However, working on big data in the cloud brings its own challenge of reconciling two contradictory design principles. Cloud computing is based on the concepts of consolidation and resource pooling, but big ...

View PDFchevron_right
On Designing a Generic Framework for Cloud-based Big Data Analytics
samiya khan

ArXiv, 2017

Big data analytics has gathered immense research attention lately because of its ability to harness useful information from heaps of data. Cloud computing has been adjudged as one of the best infrastructural solutions for implementation of big data analytics. This research paper proposes a five-layer model for cloud-based big data analytics that uses dew computing and edge computing concepts. Besides this, the paper also presents an approach for creation of custom big data stack by selecting technologies on the basis of identified data and computing models for the application

View PDFchevron_right
Big Data Platforms as a Service: Challenges and Approach
Edmon Begoli

usenix.org

Infrastructure-as-a-Service has revolutionized the manner in which users commission computing infrastructure. Coupled with Big Data platforms (Hadoop, Cassandra), IaaS has democratized the ability to store and process massive datasets. For users that need to customize or create new Big Data stacks, however, readily available solutions do not yet exist. Users must first acquire the necessary cloud computing infrastructure, and manually install the prerequisite software. For complex distributed services this can be a daunting challenge. To address this issue, we argue that distributed services should be viewed as a single application consisting of virtual machines. Users should no longer be concerned about individual machines or their internal organization. To illustrate this concept, we introduce Cloud-Get, a distributed package manager that enables the simple installation of distributed services in a cloud computing environment. Cloud-Get enables users to instantiate and modify distributed services, including Big Data services, using simple commands. Cloud-Get also simplifies creating new distributed services via standardized package definitions.

View PDFchevron_right
The big data Ecosystem and its Environs
Assoc Prof Dr Babak Bashari Rad

By the virtue of advanced devices, sensors, and social networks, big data arose to confront practitioners with a complete shift in the way they operationalize data. This has changed the context for many industries, and challenged leaders to adopt to big data ecosystem. This paper aims to explore big data ecosystem with attention to its architecture, key role players, and involving factors. In this study, firstly, the paradigm shift is discussed, and secondly the new ecosystem has been elaborated. This ecosystem is then dissected with attention to key role players, big data computation architecture, and skills required.

View PDFchevron_right
Optimizing Quality-Aware Big Data Applications in the Cloud
Danilo Ardagna

IEEE Transactions on Cloud Computing, 2021

The last years witnessed a steep rise in data generation worldwide and, consequently, the widespread adoption of software solutions able to support data-intensive applications. Competitiveness and innovation have strongly benefited from these new platforms and methodologies, and there is a great deal of interest around the new possibilities that Big Data analytics promise to make reality. Many companies currently engage in data-intensive processes as part of their core businesses; however, fully embracing the data-driven paradigm is still cumbersome, and establishing a production-ready, fine-tuned deployment is time-consuming, expensive, and resource-intensive. This situation calls for innovative models and techniques to streamline the process of deployment configuration for Big Data applications. In particular, the focus in this paper is on the rightsizing of Cloud deployed clusters, which represent a cost-effective alternative to installation on premises. This paper proposes a novel tool, integrated in a wider DevOps-inspired approach, implementing a parallel and distributed simulation-optimization technique that efficiently and effectively explores the space of alternative Cloud configurations, seeking the minimum cost deployment that satisfies quality of service constraints. The soundness of the proposed solution has been thoroughly validated in a vast experimental campaign encompassing different applications and Big Data platforms.

View PDFchevron_right
The big data system, components, tools, and technologies: a survey
Ravindara Bhatt

Knowledge and Information Systems, 2018

The traditional databases are not capable of handling unstructured data and high volumes of real-time datasets. Diverse datasets are unstructured lead to big data, and it is laborious to store, manage, process, analyze, visualize, and extract the useful insights from these datasets using traditional database approaches. However, many technical aspects exist in refining large heterogeneous datasets in the trend of big data. This paper aims to present a generalized view of complete big data system which includes several stages and key components of each stage in processing the big data. In particular, we compare and contrast various distributed file systems and MapReduce-supported NoSQL databases concerning certain parameters in data management process. Further, we present distinct distributed/cloud-based machine learning (ML) tools that play a key role to design, develop and deploy data models. The paper investigates case studies on distributed ML tools such as Mahout, Spark MLlib, and FlinkML. Further, we classify analytics based on the type of data, domain, and application. We distinguish various visualization tools pertaining three parameters: functionality, analysis capabilities, and supported development environment. Furthermore, we systematically investigate big data tools and technologies (Hadoop 3.0, Spark 2.3) including distributed/cloud-based stream processing tools in a comparative approach. Moreover, we discuss functionalities of several SQL Query tools on Hadoop based on 10 parameters. Finally, We present some critical points relevant to research directions and opportunities according to the current trend of big data. Investigating infrastructure tools for big data with recent developments provides a better understanding that how different tools and technologies apply to solve real-life applications.

View PDFchevron_right
The anatomy of big data computing
Raghavendra Rao chillarige

Software: Practice and Experience, 2015

Advances in information technology and its widespread growth in several areas of business, engineering, medical and scientific studies are resulting in information/data explosion. Knowledge discovery and decision making from such rapidly growing voluminous data is a challenging task in terms of data organization and processing, which is an emerging trend known as Big Data Computing; a new paradigm which combines large scale compute, new data intensive techniques and mathematical models to build data analytics. Big Data computing demands a huge storage and computing for data curation and processing that could be delivered from on-premise or clouds infrastructures. This paper discusses the evolution of Big Data computing, differences between traditional data warehousing and Big Data, taxonomy of Big Data computing and underpinning technologies, integrated platform of Big Data and Clouds known as Big Data Clouds, layered architecture and components of Big Data Cloud and finally discusses open technical challenges and future directions.

View PDFchevron_right
Big Picture of Big Data Software Engineering: With Example Research Challenges
Nazim Madhavji

2015 IEEE/ACM 1st International Workshop on Big Data Software Engineering, 2015

In the rapidly growing field of Big Data, we note that a disproportionately larger amount of effort is being invested in infrastructure development and data analytics in comparison to applications software development-approximately a 80:20 ratio. This prompted us to create a context model of Big Data Software Engineering (BDSE) containing various elementssuch as development practice, Big Data systems, corporate decision-making, and research-and their relationships. The model puts into perspective where various types of stakeholders fit in. From the research perspective, we describe example challenges in BDSE, specifically requirements, architectures, and testing and maintenance.

View PDFchevron_right
Service-generated Big Data and Big Data-as-a-Service: An Overview
Alain Correa

With the prevalence of service computing and cloud computing, more and more services are emerging on the Internet, generating huge volume of data, such as trace logs, QoS information, service relationship, etc. The overwhelming service-generated data become too large and complex to be effectively processed by traditional approaches. How to store, manage, and create values from the service-oriented big data become an important research problem. On the other hand, with the increasingly large amount of data, a single infrastructure which provides common functionality for managing and analyzing different types of service-generated big data is urgently required. To address this challenge, this paper provides an overview of service-generated big data and Big Data-as-a-Service. First, three types of service-generated big data are exploited to enhance system performance. Then, Big Data-as-a-Service, including Big Data Infrastructure-as-a-Service, Big Data Platform-as-a-Service, and Big Data Analytics Software-as-a-Service, is employed to provide common big data related services (e.g., accessing servicegenerated big data and data analytics results) to users to enhance efficiency and reduce cost.

View PDFchevron_right
The Role of Big Data in Systems Engineering: A Review
Ibrahim Akanbi

Engineering science & tecnology journal, 2024

This research review explores the transformative intersection of Big Data and Systems Engineering (SE), a convergence that leverages the vast capabilities of Big Data analytics to enhance the design, analysis, and management of complex systems. As the digital era ushers in unprecedented volumes of data, traditional systems engineering approaches are challenged to adapt, necessitating a paradigm shift towards integrating Big Data technologies. This review draws on recent scholarly contributions to highlight the implications, challenges, and innovative solutions emerging at this intersection. The integration of Big Data into Systems Engineering introduces significant opportunities for innovation. By processing and analyzing vast datasets, systems engineers can uncover hidden patterns and insights, leading to more efficient, reliable, and adaptable systems. However, this integration is not without its challenges. Information security emerges as a paramount concern, with the risk of insider attacks necessitating the development of new architectural solutions. Furthermore, the benchmarking and evaluation of Big Data systems present unique challenges due to the diversity of data and workloads involved. To address these challenges, this review examines several key contributions to the field. It discusses a formal framework for designing information systems that handle heterogeneous data, emphasizing the role of ontological models in separating system architecture from its

View PDFchevron_right

References (35)

  1. https ://githu b.com/maels tromd at/OSTIA . Accessed 1 Dec 2018.
  2. https ://githu b.com/socia lsens or. Accessed 1 Dec 2018.
  3. https ://githu b.com/senso rstor m/Storm CV. Accessed 1 Dec 2018.
  4. Balalaie A, Heydarnoori A, Jamshidi P. Microservices architecture enables devops: an experience report on migration to a cloud-native architecture. 2016.
  5. Bersani MM, Distefano S, Ferrucci L, Mazzara M. A timed semantics of workflows. In: ICSOFT (Selected Papers), com- munications in computer and information Science, vol. 555. Berlin: Springer; 2014. p. 365-83.
  6. Bersani MM, Marconi F, Tamburri DA, Jamshidi P, Nodari A. Continuous architecting of stream-based systems. In: Muccini H, Harper EK, editors. Proceedings of the 25th IFIP/IEEE working conference on software architectures. Washington, DC: IEEE Computer Society; 2016. p. 131-42.
  7. Bersani MM, Rossi M, San Pietro P. A tool for deciding the satisfiability of continuous-time metric temporal logic. Acta Informatica. 2015:1-36. https ://doi.org/10.1007/s0023 6-015-0229-y.
  8. Brunnert A, van Hoorn A, Willnecker F, Danciu A, Hasselbring W, Heger C, Herbst N, Jamshidi P, Jung R, von Kistowski J, et al. Performance-oriented devops: a research agenda. 2015. arXiv preprint arXiv :1508.04752 .
  9. Camilli M. Formal verification problems in a big data world: towards a mighty synergy. In: Companion proceedings of the 36th international conference on software engineering, ICSE companion. New York: ACM; 2014. p. 638-41. https ://doi.org/10.1145/25910 62.25910 88
  10. Chandrasekaran K, Santurkar S, Arora A. Stormgen -a domain specific language to create ad-hoc storm topologies. In: FedCSIS. 2014. p. 1621-8.
  11. Clements P, Kazman R, Klein M. Evaluating software architectures: methods and case studies. Boston: Addison- Wesley; 2001.
  12. Demri S, D'Souza D. An automata-theoretic approach to constraint LTL. Inf Comput. 2007;205(3):380-415.
  13. Di Nitto E, Jamshidi P, Guerriero M, Spais I, Tamburri DA. A software architecture framework for quality-aware devops. In: Proceedings of the 2nd international workshop on quality-aware DevOps, QUDOS@ISSTA 2016, Saarbrücken, Germany, July 21, 2016. 2016. p. 12-7. https ://doi.org/10.1145/29454 08.29454 11.
  14. Emani CK, Cullot N, Nicolle C. Understandable big data: a survey. Comput Sci Rev. 2015;17:70-81.
  15. Evans R. Apache storm, a hands on tutorial. In: IC2E. New York: IEEE; 2015. p. 2.
  16. Frankel D. Model driven architecture: applying MDA to enterprise computing. New York: Wiley; 2002.
  17. Furia CA, Mandrioli D, Morzenti A, Rossi M. Modeling time in computing: a taxonomy and a comparative survey. ACM Comput Surv. 2010;42(2):6:1-59.
  18. Hirzel M, Andrade H, Gedik B, Jacques-Silva G, Khandekar R, Kumar V, Mendell MP, Nasgaard H, Schneider S, Soulé R, Wu KL. Ibm streams processing language: analyzing big data in motion. IBM J Res Dev. 2013;57(3/4):7.
  19. Kalantari A, Kamsin A, Kamaruddin H, Ale Ebrahim N, Gani A, Ebrahimi A, Shamshirband S. A bibliometric approach to tracking big data research trends. J Big Data. 2017;4(1):30. https ://doi.org/10.1186/s4053 7-017-0088-1.
  20. Krippendorff K. Content analysis: an introduction to its methodology. 2nd ed. Thousand Oaks: Sage Publications; 2004.
  21. Marconi F, Bersani MM, Erascu M, Rossi M. Towards the formal verification of DIA through MTL models. In: Lecture notes in computer science.
  22. Morgan DL. Focus groups as qualitative research. Thousand Oaks: Sage Publications; 1997.
  23. Olshannikova E, Ometov A, Koucheryavy Y, Olsson T. Visualizing big data with augmented and virtual reality: chal- lenges and research agenda. J Big Data. 2015;2(1):22. https ://doi.org/10.1186/s4053 7-015-0031-2.
  24. Peng S, Gu J, Wang XS, Rao W, Yang M, Cao Y. Cost-based optimization of logical partitions for a query workload in a hadoop data warehouse. In: Chen L, Jia Y, Sellis TK, Liu G, editors. APWeb, Lecture notes in computer science, vol. 8709. Berlin: Springer; 2014. p. 559-67.
  25. Pnueli A. The temporal logic of programs. In: Proceedings of the 18th annual symposium on foundations of computer science, SFCS '77. Washington, DC: IEEE Computer Society; 1977. p. 46-57. https ://doi.org/10.1109/ SFCS.1977.32
  26. Pradella M, Morzenti A, Pietro PS. Bounded satisfiability checking of metric temporal logic specifications. ACM Trans Softw Eng Methodol. 2013;22(3):201-2054. https ://doi.org/10.1145/24915 09.24915 14.
  27. Quartulli M, Lozano J, Olaizola IG. Beyond the lambda architecture: effective scheduling for large scale eo informa- tion mining and interactive thematic mapping. In: IGARSS. 2015. p. 1492-5.
  28. Rajeev A, Dill DL. A theory of timed automata. Theor Comput Sci. 1994;126:183-235.
  29. Ratner B. Statistical and machine-learning data mining: techniques for better predictive modeling and analysis of big data. Boca Raton: CRC Press Inc; 2012.
  30. Snášel V, Nowaková J, Xhafa F, Barolli L. Geometrical and topological approaches to big data. Futur Gener Comput Syst. 2017;67:286-96. https ://doi.org/10.1016/j.futur e.2016.06.005.
  31. Tamura Y, Yamada S. Reliability analysis based on a jump diffusion model with two wiener processes for cloud computing with big data. Entropy. 2015;17(7):4533-46.
  32. Tommaso Di Noia MM, Sciascio ED. A computational model for mapreduce job flow. 2014.
  33. Toshniwal A, Taneja S, Shukla A, Ramasamy K, Patel JM, Kulkarni S, Jackson J, Gade K, Fu M, Donham J et al. Storm@ twitter. In: Proceedings of the 2014 ACM SIGMOD international conference on management of data. New York: ACM; 2014. p. 147-56.
  34. Wang D, Liu J. Optimizing big data processing performance in the public cloud: opportunities and approaches. IEEE Netw. 2015;29(5):31-5.
  35. Yang F, Su W, Zhu H, Li Q. Formalizing mapreduce with csp. In: Proceedings of ECBS. Washington, DC: IEEE Computer Society; 2010. p. 358-67. https ://doi.org/10.1109/ECBS.2010.50.

Related papers

A Fast and Incremental Development Life Cycle for Data Analytics as a Service
Beniamino Di Martino

2018 IEEE International Congress on Big Data (BigData Congress), 2018

Big Data does not only refer to a huge amount of diverse and heterogeneous data. It also points to the management of procedures, technologies, and competencies associated with the analysis of such data, with the aim of supporting high-quality decision making. There are, however, several obstacles to the effective management of a Big Data computation, such as data velocity, variety, and veracity, and technological complexity, which represent the main barriers towards the full adoption of the Big Data paradigm. The goal of this work is to define a new software Development Life Cycle for the design and implementation of a Big Data computation. Our proposal integrates two model-driven methods: a first method based on pre-configured services that reduces the cost of deployment and a second method based on custom component development that provides an incremental process of refinement and customization. The proposal is experimentally evaluated by clustering a data set of the distribution of the population in the United States based on contextual criteria.

View PDFchevron_right
Deciphering Big Data Stacks: An Overview of Big Data Tools
Tomislav Lipic

2014

With its ability to ingest, process, and decipher an abundance of incoming data, the Big Data is considered by many a cornerstone of future research and development. However, the large number of available tools and the overlap between those are impeding their technological potential. In this paper, we present a systematic grouping of the available tools and present a network of dependencies among those with the aim of composing individual tools into functional software stacks required to perform Big Data analyses. Keywords—Big Data; distributed systems; Cloud computing; data processing platform trends; interdependency

View PDFchevron_right
Toward a Framework for Improving the Execution of the Big Data Applications
Wissem Abbes

Procedia Computer Science, 2015

In this paper, we propose a new framework based on learning techniques to improve the execution of a cloud application, especially the big-data application that requires significant computing capacity. We propose a new metric to detect the available capacity in the cloud client.

View PDFchevron_right
Big data: Evaluation criteria for big data analytics technologies
Andre De La Harpe

Proceedings of the International Conference on Business and Management Dynamics 2016: Sustainable economies in the information economy, 2016

View PDFchevron_right
Runtime Performance Challenges in Big Data Systems
Ian Gorton

Proceedings of the 2015 Workshop on Challenges in Performance Methods for Software Development - WOSP '15, 2015

Big data systems are becoming pervasive. They are distributed systems that include redundant processing nodes, replicated storage, and frequently execute on a shared "cloud" infrastructure. For these systems, design-time predictions are insufficient to assure runtime performance in production. This is due to the scale of the deployed system, the continually evolving workloads, and the unpredictable quality of service of the shared infrastructure. Consequently, a solution for addressing performance requirements needs sophisticated runtime observability and measurement. Observability gives real-time insights into a system's health and status, both at the system and application level, and provides historical data repositories for forensic analysis, capacity planning, and predictive analytics. Due to the scale and heterogeneity of big data systems, significant challenges exist in the design, customization and operations of observability capabilities. These challenges include economical creation and insertion of monitors into hundreds or thousands of computation and data nodes, efficient, low overhead collection and storage of measurements (which is itself a big data problem), and application-aware aggregation and visualization. In this paper we propose a reference architecture to address these challenges, which uses a model-driven engineering toolkit to generate architecture-aware monitors and application-specific visualizations.

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved