Academia.eduAcademia.edu

Outline

Title
Abstract
Related Work
Big Data Architectures
Big Data Computational Model
Experimental Testbed
Delay Variance Results
Summary of Insights
Conclusion
References
All Topics
Computer Science
Distributed Systems

Delay-Resistant Geo-Distributed Analytics

Profile image of Habib MostafaeiHabib Mostafaei

IEEE Transactions on Network and Service Management

https://doi.org/10.1109/TNSM.2022.3192710
visibility

…

description

16 pages

link

1 file

Abstract

Big data analytics platforms have played a critical role in the unprecedented success of data-driven applications. However, real-time and streaming data applications, and recent legislation, e.g., GDPR in Europe, have posed constraints on exchanging and analyzing data, especially personal data, across geographic regions. To address such constraints data has to be processed and analyzed in-situ and aggregated results have to be exchanged among the different sites for further processing. This introduces additional network delays due to the geographic distribution of the sites and potentially affecting the performance of analytics platforms that are designed to operate in datacenters with low network delays. In this paper, we show that the three most popular big data analytics systems (Apache Storm, Apache Spark, and Apache Flink) fail to tolerate round-trip times more than 30 milliseconds even when the input data rate is low. The execution time of distributed big data analytics tasks degrades substantially after this threshold, and some of the systems are more sensitive than others. A closer examination and understanding of the design of these systems show that there is no winner in all wide-area settings. However, we show that it is possible to improve the performance of all these popular big data analytics systems significantly amid even transcontinental delays (where inter-node delay is more than 30 milliseconds) and achieve performance comparable to this within a datacenter for the same load. Index Terms-Wide-area analytics, big data analytics, geodistributed systems, networked systems. I. INTRODUCTION B IG DATA analytics platforms [1], [2], [3], [4], [5], [6] have played a critical role in the unprecedented success of data-driven applications. Such platforms are typically deployed

Related papers

Emerging Technology and Architecture for Big-data Analytics
Chip Hong Chang

2017

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

View PDFchevron_right
Enabling Strategies for Big Data Analytics in Hybrid Infrastructures
Kassiano Matteussi

2018 International Conference on High Performance Computing & Simulation (HPCS), 2018

A huge volume of data is produced every day by social networks (e.g. Facebook, Instagram, Whatsapp, etc.), sensors, mobile devices and other applications. Although the Cloud computing scenario has grown rapidly in recent years, it still suffers from a lack of the kind of standardization that involves the resource management for Big Data applications, such as the case of MapReduce. In this context, the users face a big challenge in attempting to understand the requirements of the application and how to consolidate the resources properly. This scenario raises significant challenges in the different areas: systems, infrastructure, platforms as well as providing several research opportunities in Big Data Analytics. This work proposes the use of hybrid infrastructures such as Cloud and Volunteer Computing for Big Data processing and analysis. In addition, it provides a data distribution model that improves the resource management of Big Data applications in hybrid infrastructures. The results indicate the feasibility of hybrid infrastructures since it supports the reproducibility and predictability of Big Data processing by low and high-scale simulation within Hybrid infrastructures.

View PDFchevron_right
Big Data Streaming Platforms to Support Real-time Analytics
Eliana Fernandes

2020

In recent years data has grown exponentially due to the evolution of technology. The data flow circulates in a very fast and continuous way, so it must be processed in real time. Therefore, several big data streaming platforms have emerged for processing large amounts of data. Nowadays, companies have difficulties in choosing the platform that best suits their needs. In addition, the information about the platforms is scattered and sometimes omitted, making it difficult for the company to choose the right platform. This work focuses on helping companies or organizations to choose a big data streaming platform to analyze and process their data flow. We provide a description of the most popular platforms, such as: Apache Flink, Apache Kafka, Apache Samza, Apache Spark and Apache Storm. To strengthen the knowledge about these platforms, we also approached their architectures, advantages and limitations. Finally, a comparison among big data streaming platforms will be provided, using as...

View PDFchevron_right
Big Data Analytics Beyond Hadoop: Real-Time Applications with Storm, Spark, and More Hadoop Alternatives
Vijay Agneeswaran

2014

Master alternative Big Data technologies that can do what Hadoop can't: real-time analytics and iterative machine learning. When most technical professionals think of Big Data analytics today, they think of Hadoop. But there are many cutting-edge applications that Hadoop isn't well suited for, especially real-time analytics and contexts requiring the use of iterative machine learning algorithms. Fortunately, several powerful new technologies have been developed specifically for use cases such as these. Big Data Analytics Beyond Hadoop is the first guide specifically designed to help you take the next steps beyond Hadoop. Dr. Vijay Srinivas Agneeswaran introduces the breakthrough Berkeley Data Analysis Stack (BDAS) in detail, including its motivation, design, architecture, Mesos cluster management, performance, and more. He presents realistic use cases and up-to-date example code for: Spark, the next generation in-memory computing technology from UC Berkeley Storm, the parall...

View PDFchevron_right
An Algorithmic Framework for Geo-Distributed Analytics
Srikanth Kandula

Static & dynamic game theory: foundations & application, 2019

Large scale cloud enterprises operate tens to hundreds of datacenters, running a variety of services that produce enormous amounts of data, such as search clicks and infrastructure operation logs. A recent research direction in both academia and industry is to attempt to process the "big data" in multiple datacenters, as the alternative of centralized processing might be too slow and costly (e.g., due to transferring all the data to a single location). Running such geo-distributed analytics jobs at scale gives rise to key resource management decisions: Where should each of the computations take place? Accordingly, which data should be moved to which location, and when? Which network paths should be used for moving the data, etc. These decisions are complicated not only because they involve the scheduling of multiple types of resources (e.g., compute and network), but also due to the complicated internal data flow of the jobs-typically structured as a DAG of tens of stages, each of which with up to thousands of tasks. Recent work [17, 22, 25] has dealt with the resource management problem by abstracting away certain aspects of the problem, such as the physical network connecting the datacenters, the DAG structure of the jobs and/or the compute capacity constraints at the (possibly heterogeneous) datacenters. In this paper, we provide the first analytical model that includes all aspects of the problem, with the objective of minimizing the makespan of multiple geo-distributed jobs. We provide exact and approximate algorithms for certain practical scenarios, and suggest principled heuristics for other scenarios of interest.

View PDFchevron_right
A survey on platforms for big data analytics
Rehan Sarwar

The primary purpose of this paper is to provide an in-depth analysis of different platforms available for performing big data analytics. This paper surveys different hardware platforms available for big data analytics and assesses the advantages and drawbacks of each of these platforms based on various metrics such as scalability, data I/O rate, fault tolerance, real-time processing, data size supported and iterative task support. In addition to the hardware, a detailed description of the software frameworks used within each of these platforms is also discussed along with their strengths and drawbacks. Some of the critical characteristics described here can potentially aid the readers in making an informed decision about the right choice of platforms depending on their computational needs. Using a star ratings table, a rigorous qualitative comparison between different platforms is also discussed for each of the six characteristics that are critical for the algorithms of big data analytics. In order to provide more insights into the effectiveness of each of the platform in the context of big data analytics, specific implementation level details of the widely used k-means clustering algorithm on various platforms are also described in the form pseudocode.

View PDFchevron_right
The Stratosphere platform for big data analytics
Jyotir Chatterjee

Presented Stratosphere, an open-source software stack for parallel data analysis. Stratosphere brings together a unique set of features that allow the expressive, easy, and efficient programming of analytical applications at very large scale. Stratosphere’s features include “in situ” data processing, a declarative query language, treatment of user-defined functions as first-class citizens, automatic program parallelization and optimization, support for iterative programs, and a scalable and efficient execution engine. Stratosphere covers a variety of “Big Data” use cases, such as data warehousing, information extraction and integration, data cleansing, graph analysis, and statistical analysis applications. In this paper, we present the overall system architecture design decisions, introduce Stratosphere through example queries, and then dive into the internal workings of the system’s components that relate to extensibility, programming model, optimization, and query execution. We experimentally compare Stratosphere against popular open-source alternatives, and we conclude with a research outlook for the next years.

View PDFchevron_right
Pushing Analytics to the Edge
Maria Valero

2016 IEEE Global Communications Conference (GLOBECOM), 2016

Edge or Fog computing is emerging as a new computing paradigm where the data processing, networking, storage and analytics are performed closer to the devices (IoT) and applications. The edge of a network plays an important role in the IoT system. It is an optimal site for off-loading bandwidth hungry IoT data. In order to generate business value out of the large volume of data on the edge, we need decentralized machine learning (ML) algorithms. These algorithms must enable edge devices to communicate autonomously and deliver information seamlessly to the decision makers. In this paper, we present EdgeSGD a decentralized stochastic gradient descent method to solve a large linear regression problem on the edge network. The solution is applied to the seismic imaging use-case and evaluated using an edge computing testbed. The proposed algorithm avoids sending raw data to the cloud, and offers faster and balanced computation. We compare the algorithm with the existing methods such as MapReduce, DGD and EXTRA. The results show that the EdgeSGD converges faster to the optimal value and is robust towards node/link failure. Finally, we test the algorithm using real-world seismic data trace from Parkfield, CA and show that our proposed algorithm can illuminate the underlying fault region of San-Andreas.

View PDFchevron_right
SMART: An Application Framework for Real Time Big Data Analysis on Heterogeneous Cloud Environments
Rubem Pereira

2015 IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing, 2015

HAL is a multidisciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

View PDFchevron_right
A Fast and Incremental Development Life Cycle for Data Analytics as a Service
Beniamino Di Martino

2018 IEEE International Congress on Big Data (BigData Congress), 2018

Big Data does not only refer to a huge amount of diverse and heterogeneous data. It also points to the management of procedures, technologies, and competencies associated with the analysis of such data, with the aim of supporting high-quality decision making. There are, however, several obstacles to the effective management of a Big Data computation, such as data velocity, variety, and veracity, and technological complexity, which represent the main barriers towards the full adoption of the Big Data paradigm. The goal of this work is to define a new software Development Life Cycle for the design and implementation of a Big Data computation. Our proposal integrates two model-driven methods: a first method based on pre-configured services that reduces the cost of deployment and a second method based on custom component development that provides an incremental process of refinement and customization. The proposal is experimentally evaluated by clustering a data set of the distribution of the population in the United States based on contextual criteria.

View PDFchevron_right

References (67)

  1. "Apache storm." 2020. [Online]. Available: https://storm.apache.org/
  2. A. Toshniwal et al., "Storm@twitter," in Proc. ACM SIGMOD, 2014, pp. 147-156.
  3. "Apache spark." 2020. [Online]. Available: https://spark.apache.org/
  4. M. Zaharia et al., "Spark: Cluster computing with working sets," in Proc. HotCloud, vol. 10, 2010, p. 95.
  5. "Apache Flink." 2020. [Online]. Available: https://flink.apache.org/
  6. P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas, "Apache flink: Stream and batch processing in a single engine," Bull. IEEE Comput. Soc. Tech. Committee Data Eng., vol. 36, no. 4, pp. 28-38, 2015.
  7. "Keystone real-time stream processing platform." 2021. [Online]. Available: https://bit.ly/3ngPoeF (Accessed: Jan. 6, 2021).
  8. "Video access log processing with apache Flink." 2021. [Online]. Available: https://bit.ly/3ol4Vvz (Accessed: Jan. 6, 2021).
  9. "2020 identity fraud study: Genesis of the identity fraud crisis." 2021. [Online]. Available: https://bit.ly/3hSTkBn (Accessed: Jan. 6, 2021).
  10. "The state of online retail performance, spring 2017, Akamai." 2021. [Online]. Available: https://bit.ly/3hSVp09 (Accessed: Jan. 6, 2021).
  11. "Data protection in the EU, the general data protection regulation (GDPR); regulation (EU) 2016/679." 2016. [Online]. Available: http:/ /bit.ly/3qdVUVo
  12. L. Kalman, "New European data privacy and cyber security laws: One year later," Commun. ACM, vol. 62, no. 4, p. 38, 2019.
  13. S. Greengard, "Weighing the impact of GDPR," Commun. ACM, vol. 61, no. 11, pp. 16-18, 2018.
  14. State of California. "California consumer privacy act-Assembly bill no. 375." 2018. [Online]. Available: http://bit.ly/2K5qOjo
  15. Office of the Privacy Commissioner of Canada. "Amended act on the personal information protection and electronic documents act." 2018. [Online]. Available: https://bit.ly/3oPDSJ7
  16. The Privacy Protection Authority of Israel. "Protection of privacy reg- ulations (data security) 5777-2017." 2018. [Online]. Available: https:// bit.ly/2LMwcIA
  17. Personal Information Protection Commission, Japan. "Amended act on the protection of personal information." 2017. [Online]. Available: https:/ /www.ppc.go.jp/en/
  18. Office of the Australian Information Commissioner. "Australian privacy principles guidelines; Australian privacy principle 5-Notification of the collection of personal information." 2018. [Online]. Available: https:// bit.ly/38BBrUP
  19. A. Rabkin, M. Arye, S. Sen, V. S. Pai, and M. J. Freedman, "Aggregation and degradation in JetStream: Streaming Analytics in the wide area," in Proc. NSDI, 2014, pp. 275-288.
  20. B. Zhang, X. Jin, S. Ratnasamy, J. Wawrzynek, and E. A. Lee, "AWStream: Adaptive wide-area streaming analytics," in Proc. SIGCOMM, 2018, pp. 236-252.
  21. R. Viswanathan, G. Ananthanarayanan, and A. Akella, "CLARINET: WAN-aware optimization for analytics queries," in Proc. OSDI, 2016, pp. 435-450.
  22. Q. Pu et al., "Low latency geo-distributed data analytics," in Proc. ACM SIGCOMM, 2015, pp. 421-434.
  23. C.-C. Hung, G. Ananthanarayanan, L. Golubchik, M. Yu, and M. Zhang, "Wide-area analytics with multiple resources," in Proc. EuroSys, 2018, pp. 1-16.
  24. W. Xiao, W. Bao, X. Zhu, and L. Liu, "Cost-aware big data processing across geo-distributed datacenters," IEEE Trans. Parallel Distrib. Syst., vol. 28, no. 11, pp. 3114-3127, Nov. 2017.
  25. A. Vulimiri, C. Curino, B. Godfrey, K. Karanasos, and G. Varghese, "WANalytics: Analytics for a Geo-distributed data-intensive world," in Proc. CIDR, 2015, pp. 1-7.
  26. D. Kumar, J. Li, A. Chandra, and R. Sitaraman, "A TTL-based approach for data aggregation in Geo-distributed streaming Analytics," Proc. ACM Meas. Anal. Comput. Syst., vol. 3, no. 2, pp. 1-27, 2019.
  27. W. Li, D. Niu, Y. Liu, S. Liu, and B. Li, "Wide-area spark streaming: Automated routing and batch sizing," IEEE Trans. Parallel Distrib. Syst., vol. 30, no. 6, pp. 1434-1448, Jun. 2019.
  28. A. Jonathan, A. Chandra, and J. Weissman, "Multi-query optimization in wide-area streaming analytics," in Proc. SoCC, 2018, pp. 412-425.
  29. F. Lai, J. You, X. Zhu, H. V. Madhyastha, and M. Chowdhury, "Sol: Fast distributed computation over slow networks," in Proc. NSDI, 2020, pp. 273-288.
  30. B. Heintz, A. Chandra, and R. K. Sitaraman, "Optimizing timeliness and cost in Geo-distributed streaming Analytics," IEEE Trans. Cloud Comput., vol. 8, no. 1, pp. 232-245, Jan.-Mar. 2020.
  31. A. Jonathan, A. Chandra, and J. Weissman, "Rethinking adaptability in wide-area stream processing systems," in Proc. HotCloud, 2018, pp. 1-9.
  32. H. Mostafaei, S. Afridi, and J. Abawajy, "Network-aware worker place- ment for wide-area streaming analytics," Future Gener. Comput. Syst., vol. 136, pp. 270-281, Nov. 2022.
  33. L. Chen, S. Liu, and B. Li, "Optimizing network transfers for data analytic jobs across Geo-distributed Datacenters," IEEE Trans. Parallel Distrib. Syst., vol. 33, no. 2, pp. 403-414, Feb. 2022.
  34. Y. Chen, L. Luo, D. Guo, O. Rottenstreich, and J. Wu, "SDTP: Accelerating wide-area data analytics with simultaneous data transfer and processing," IEEE Trans. Cloud Comput., early access, Oct. 15, 2021, doi: 10.1109/TCC.2021.3119991.
  35. K. Beedkar, D. Brekardin, J.-A. Quiané-Ruiz, and V. Markl, "Compliant geo-distributed data processing in action," Proc. VLDB Endow., vol. 14, no. 12, pp. 2843-2846, Jul. 2021.
  36. A. Jonathan, A. Chandra, and J. Weissman, "WASP: Wide-area adap- tive stream processing," in Proc. ACM/IFIP/USENIX Middleware, 2020, pp. 221-235.
  37. Y. Jin et al., "Zooming in on wide-area latencies to a global cloud provider," in Proc. SIGCOMM, 2019, pp. 104-116.
  38. W. Reda et al., "Path persistence in the cloud: A study of the effects of inter-region traffic engineering in a large cloud provider's network," SIGCOMM Comput. Commun. Rev., vol. 50, no. 2, pp. 11-23, 2020.
  39. C.-Y. Hong et al., "Achieving high utilization with software-driven WAN," in Proc. ACM SIGCOMM, 2013, pp. 15-26.
  40. S. Chintapalli et al., "Benchmarking streaming computation engines: Storm, Flink and spark streaming," in Proc. IPDPSW, 2016, pp. 1789-1792.
  41. J. Karimov, T. Rabl, A. Katsifodimos, R. Samarev, H. Heiskanen, and V. Markl, "Benchmarking distributed stream data processing systems," in Proc. ICDE, 2018, pp. 1507-1518.
  42. M. A. Lopez, A. G. P. Lobato, and O. C. M. B. Duarte, "A performance comparison of open-source stream processing platforms," in Proc. GLOBECOM, 2016, pp. 1-6.
  43. "Apache Zookeeper." 2020. [Online]. Available: https:// zookeeper.apache.org/
  44. "Apache Hadoop." 2020. [Online]. Available: https://hadoop.apache.org/
  45. "What is/are the main difference(s) between Flink and storm?" 2015. [Online]. Available: https://bit.ly/3qiaQSF
  46. S. Saxena and S. Gupta, Practical Real-Time Data Processing and Analytics: Distributed Computing and Event Processing Using Apache Spark, Flink, Storm, and Kafka. London, U.K.: Packt, 2017.
  47. "Trident API overview." 2021. [Online]. Available: https://bit.ly/ 3win8NC
  48. S. Zeuch et al., "Analyzing efficient stream processing on modern hardware," Proc. VLDB Endow., vol. 12, no. 5, pp. 516-530, 2019.
  49. "Extending the Yahoo streaming benchmarks." 2020. [Online]. Available: https://github.com/dataArtisans/yahoo-streaming-benchmark
  50. "Apache kafka." 2020. [Online]. Available: https://kafka.apache.org/
  51. "Redis." 2020. [Online]. Available: https://redis.io/
  52. J. Dean and L. A. Barroso, "The tail at scale," Commun. ACM, vol. 56, no. 2, pp. 74-80, 2013.
  53. "Traffic control (TC)." 2020. [Online]. Available: https://wiki.debian.org/ TrafficControl
  54. "The state of the Internet." 2020. [Online]. Available: https://bit.ly/ 39gKAS4
  55. "Apache storm: Performance tuning." 2021. [Online]. Available: https:/ /bit.ly/3bLzYfM
  56. "Equinix." 2020. [Online]. Available: https://www.equinix.com/data- centers/
  57. "Amazon global infrastructure." 2020. [Online]. Available: http:// amzn.to/38zsFq4
  58. "Data residency in azure." 2020. [Online]. Available: http://bit.ly/ 3qdWimV
  59. "Google Datacenters." 2020. [Online]. Available: https://about.google/ locations/
  60. "AT&T network delay." 2020. [Online]. Available: https:// ipnetwork.bgtmo.ip.att.net/pws/network_delay.html (Accessed: Apr. 11, 2020).
  61. "Global ping statistics: Ping times between Wonder Network servers." 2020. [Online]. Available: https://wondernetwork.com/pings(Accessed: Sep. 10, 2020).
  62. "Analysis of network flow control and back pressure: Flink advanced tutorials." 2020. [Online]. Available: https://bit.ly/340lvZ7
  63. "Spark streaming programming guide." 2020. [Online]. Available: http:/ /bit.ly/3bw3bfb (Accessed: Dec. 28, 2020).
  64. "Socket statistics (SS)." 2020. [Online]. Available: http://bit.ly/39nj2dm
  65. "Can apache spark use TCP listener as input?" 2019. [Online]. Available: https://bit.ly/3bJIUSV
  66. "vmStat-Report virtual memory statistics." 2020. [Online]. Available: http://bit.ly/38BlTAd
  67. A. Uta et al., "Is big data performance reproducible in modern cloud networks?" in Proc. NSDI, 2020, pp. 513-527.

Related papers

Low Latency Geo-distributed Data Analytics
Qifan Pu

Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication - SIGCOMM '15, 2015

Low latency analytics on geographically distributed datasets (across datacenters, edge clusters) is an upcoming and increasingly important challenge. The dominant approach of aggregating all the data to a single datacenter significantly inflates the timeliness of analytics. At the same time, running queries over geo-distributed inputs using the current intra-DC analytics frameworks also leads to high query response times because these frameworks cannot cope with the relatively low and variable capacity of WAN links. We present Iridium, a system for low latency geo-distributed analytics. Iridium achieves low query response times by optimizing placement of both data and tasks of the queries. The joint data and task placement optimization, however, is intractable. Therefore, Iridium uses an online heuristic to redistribute datasets among the sites prior to queries' arrivals, and places the tasks to reduce network bottlenecks during the query's execution. Finally, it also contains a knob to budget WAN usage. Evaluation across eight worldwide EC2 regions using production queries show that Iridium speeds up queries by 3× − 19× and lowers WAN usage by 15% − 64% compared to existing baselines.

View PDFchevron_right
A survey on bandwidth-aware geo-distributed frameworks for big-data analytics
Said Najah

2021

In the era of global-scale services, organisations produce huge volumes of data, often distributed across multiple data centres, separated by vast geographical distances. While cluster computing applications, such as MapReduce and Spark, have been widely deployed in data centres to support commercial applications and scientific research, they are not designed for running jobs across geo-distributed data centres. The necessity to utilise such infrastructure introduces new challenges in the data analytics process due to bandwidth limitations of the inter-data-centre communication. In this article, we discuss challenges and survey the latest geo-distributed big-data analytics frameworks and schedulers (based on MapReduce and Spark) with WAN-bandwidth awareness.

View PDFchevron_right
Distributed Data Strategies to Support Large-Scale Data Analysis Across Geo-Distributed Data Centers
Joshua Zhexue Huang

IEEE Access, 2020

As the volume of data grows rapidly, storing big data in a single data center is no longer feasible. Hence, companies have developed two scenarios to store their big data in multiple data centers. In the first scenario, the company's big data are distributed in multiple data centers without data replication. In the second scenario, data are also stored in multiple data centers but important data are replicated in these data centers to increase data safety and availability. However, in these scenarios, analyzing big data distributed in multiple data centers becomes a challenging task. In this paper, we propose two data distribution strategies to support big data analysis across geo-distributed data centers. In these strategies, we use the recent Random Sample Partition data model to convert big data into sets of random sample data blocks and distribute these data blocks into multiple data centers either without replication or with replication. In analyzing big data in multiple data centers without replication, we randomly select samples of data blocks from multiple data centers and download the sample data blocks to one data center for analysis. In the second strategy with replication of data blocks, we can analyze big data on any data center by randomly selecting a sample of data blocks replicated from other data centers. This strategy avoids data transformation between data centers. We demonstrate the performance of the two strategies in big data analysis by using simulation results produced on one local data center and four AWS data centers in North America, Asia, and Australia. INDEX TERMS Big data analysis, cloud data centers, distributed computing, random sample partition, wide area analytics.

View PDFchevron_right
SURVEY ON DISTRIBUTED DATA STRATEGIES TO SUPPORT LARGE-SCALE DATA ANALYSIS ACROSS GEO-DISTRIBUTED DATA CENTERS
International Research Journal of Modernization in Engineering Technology and Science (IRJMETS)

IRJMETS PUblication, 2021

The storage of big data in a single data center is no longer possible when the volume of data rises fast. As a result, businesses have devised two scenarios for storing their large data across several data centers. The company's large data is scattered across numerous data centers in the first scenario, with no data replication. Data is kept in numerous data centers in the second scenario, but vital data is replicated in these data centers to ensure data safety and availability. Analyzing massive data dispersed across several data centers. In this we study two data distribution algorithms from our referenced base paper [1] to assist big data analysis across geo-distributed data.

View PDFchevron_right
The big data system, components, tools, and technologies: a survey
Ravindara Bhatt

Knowledge and Information Systems, 2018

The traditional databases are not capable of handling unstructured data and high volumes of real-time datasets. Diverse datasets are unstructured lead to big data, and it is laborious to store, manage, process, analyze, visualize, and extract the useful insights from these datasets using traditional database approaches. However, many technical aspects exist in refining large heterogeneous datasets in the trend of big data. This paper aims to present a generalized view of complete big data system which includes several stages and key components of each stage in processing the big data. In particular, we compare and contrast various distributed file systems and MapReduce-supported NoSQL databases concerning certain parameters in data management process. Further, we present distinct distributed/cloud-based machine learning (ML) tools that play a key role to design, develop and deploy data models. The paper investigates case studies on distributed ML tools such as Mahout, Spark MLlib, and FlinkML. Further, we classify analytics based on the type of data, domain, and application. We distinguish various visualization tools pertaining three parameters: functionality, analysis capabilities, and supported development environment. Furthermore, we systematically investigate big data tools and technologies (Hadoop 3.0, Spark 2.3) including distributed/cloud-based stream processing tools in a comparative approach. Moreover, we discuss functionalities of several SQL Query tools on Hadoop based on 10 parameters. Finally, We present some critical points relevant to research directions and opportunities according to the current trend of big data. Investigating infrastructure tools for big data with recent developments provides a better understanding that how different tools and technologies apply to solve real-life applications.

View PDFchevron_right

Related topics

  • Computer Science
  • Distributed Computing
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved