Academia.eduAcademia.edu

Outline

Title
Abstract
Background and Related Work
Big Data Stream Analysis
Key Issues in Big Data Stream Analysis
Related Work
Data Sources
Data Retrieval
Result
Shape of the Data
Data Access
Methods and Techniques Article
Discussion
Limitation of the Review
Conclusion and Further Work
References
All Topics
Computer Science

Big data stream analysis: a systematic literature review

Profile image of Taiwo KolajoTaiwo Kolajo

Journal of Big Data

https://doi.org/10.1186/S40537-019-0210-7
visibility

…

description

30 pages

link

1 file

Abstract

Advances in information technology have facilitated large volume, high-velocity of data, and the ability to store data continuously leading to several computational challenges. Due to the nature of big data in terms of volume, velocity, variety, variability, veracity, volatility, and value [1] that are being generated recently, big data computing is a new trend for future computing. Big data computing can be generally categorized into two types based on the processing requirements, which are big data batch computing and big data stream computing Abstract Recently, big data streams have become ubiquitous due to the fact that a number of applications generate a huge amount of data at a great velocity. This made it difficult for existing data mining tools, technologies, methods, and techniques to be applied directly on big data streams due to the inherent dynamic characteristics of big data. In this paper, a systematic review of big data streams analysis which employed a rigorous and methodical approach to look at the trends of big data stream tools and technologies as well as methods and techniques employed in analysing big data streams. It provides a global view of big data stream tools and technologies and its comparisons. Three major databases, Scopus, ScienceDirect and EBSCO, which indexes journals and conferences that are promoted by entities such as IEEE, ACM, SpringerLink, and Elsevier were explored as data sources. Out of the initial 2295 papers that resulted from the first search string, 47 papers were found to be relevant to our research questions after implementing the inclusion and exclusion criteria. The study found that scalability, privacy and load balancing issues as well as empirical analysis of big data streams and technologies are still open for further research efforts. We also found that although, significant research efforts have been directed to real-time analysis of big data stream not much attention has been given to the preprocessing stage of big data streams. Only a few big data streaming tools and technologies can do all of the batch, streaming, and iterative jobs; there seems to be no big data tool and technology that offers all the key features required for now and standard benchmark dataset for big data streaming analytics has not been widely adopted. In conclusion, it was recommended that research efforts should be geared towards developing scalable frameworks and algorithms that will accommodate data stream computing mode, effective resource allocation strategy and parallelization issues to cope with the ever-growing size and complexity of data.

Related papers

A Software Chain Approach to Big Data Stream Processing and Analytics
Leonard Barolli

2015 Ninth International Conference on Complex, Intelligent, and Software Intensive Systems, 2015

Big Data Stream processing is among the most important computing trends nowadays. The growing interest on Big Data Stream processing comes from the need of many Internet-based applications that generate huge data streams, whose processing can serve to extract useful analytics and inform for decision making systems. For instance, an IoTbased monitoring systems for a supply-chain, can provide real time data analytics for the business delivery performance. The challenges of processing Big Data Streams reside on coping with real-time processing of an unbounded stream of data, that is, the computing system should be able to compute at high throughput to accommodate the high data stream rate generation in input. Clearly, the higher the data stream rate, the higher should be the throughput to achieve consistency of the processing results (e.g. preserving the order of events in the data stream). In this paper we show how to map the data stream processing phases (from data generation to final results) to a software chain architecture, which comprises five main components: sensor, extractor, parser, formatter and outputter. We exemplify the approach using the Yahoo!S4 for processing the Big Data Stream from FlightRadar24 global flight monitoring system.

View PDFchevron_right
Big Data Streaming Platforms to Support Real-time Analytics
Eliana Fernandes

2020

In recent years data has grown exponentially due to the evolution of technology. The data flow circulates in a very fast and continuous way, so it must be processed in real time. Therefore, several big data streaming platforms have emerged for processing large amounts of data. Nowadays, companies have difficulties in choosing the platform that best suits their needs. In addition, the information about the platforms is scattered and sometimes omitted, making it difficult for the company to choose the right platform. This work focuses on helping companies or organizations to choose a big data streaming platform to analyze and process their data flow. We provide a description of the most popular platforms, such as: Apache Flink, Apache Kafka, Apache Samza, Apache Spark and Apache Storm. To strengthen the knowledge about these platforms, we also approached their architectures, advantages and limitations. Finally, a comparison among big data streaming platforms will be provided, using as...

View PDFchevron_right
Real-time stream processing for Big Data
Wolfram Wingerath

it - Information Technology, 2016

With the rise of the web 2.0 and the Internet of things, it has become feasible to track all kinds of information over time, in particular fine-grained user activities and sensor data on their environment and even their biometrics. However, while efficiency remains mandatory for any application trying to cope with huge amounts of data, only part of the potential of today's Big Data repositories can be exploited using traditional batch-oriented approaches as the value of data often decays quickly and high latency becomes unacceptable in some applications. In the last couple of years, several distributed data processing systems have emerged that deviate from the batch-oriented approach and tackle data items as they arrive, thus acknowledging the growing importance of timeliness and velocity in Big Data analytics. In this article, we give an overview over the state of the art of stream processors for low-latency Big Data analytics and conduct a qualitative comparison of the most po...

View PDFchevron_right
PROCESSING & ANALYSING LARGE & COMPLEX DATA STREAMS USING BIG DATA
ASHAD U L L A H QURESHI

Concepts Books Publication, 2018

The emerging large datasets have made efficient data processing a much more difficult task for the traditional methodologies. Invariably, datasets continue to increase rapidly in size with time. The purpose of this research is to give an overview of some of the tools and techniques that can be utilized to manage and analyze large datasets. We propose a faster way to catalogue and retrieve data by creating a directory file – more specifically, an improved method that would allow file retrieval based on its time and date. This method eliminates the process of searching the entire content of files and reduces the time it takes to locate the selected data. We also implement the nearest search algorithm in an event where the searched query is not found. The algorithm sorts through data to find the closest points that are within close proximity to the searched query. We also offer an efficient data reduction method that effectively condenses the amount of data. The algorithm enables users to store the desired amount of data in a file and decrease the time in which observations are retrieved for processing. This is achieved by using a reduced standard deviation range to minimize the original data and keeping the dataset to a significant smaller dataset size.

View PDFchevron_right
A survey of systems for massive stream analytics
SHREY UPADHYAY

The immense growth of data demands switching from traditional data processing solutions to systems, which can process a continuous stream of real time data. Various applications employ stream processing systems to provide solutions to emerging Big Data problems. Open-source solutions such as Storm, Spark Streaming, and S4 are the attempts to answer key stream processing questions. The recent introduction of real time stream processing commercial solutions such as Amazon Kinesis, IBM Infosphere Stream reflect industry requirements. The system and application related challenges to handle massive stream of real time data analytics are an active field of research. In this paper, we present a comparative analysis of the existing state-of-the-art stream processing solutions. We also include various application domains, which are transforming their business model to benefit from these large scale stream processing systems.

View PDFchevron_right
Handling Big Data Stream Analytics using SAMOA Framework - A Practical Experience
Sonali Agarwal

International Journal of Database Theory and Application, 2014

Data analytics and machine learning has always been of great importance in almost every field especially in business decision making and strategy building, in healthcare domain, in text mining and pattern identification on the web, in meteorological department, etc. The daily exponential growth of data today has shifted the normal data analytics to new paradigm of Big Data Analytics and Big Data Machine Learning. We need tools to perform online data analysis on streaming data for achieving faster learning and faster response in data analytics as well as maintaining scalability in terms of huge volume of data. SAMOA (Scalable Advanced Massive Online Analysis) is a recent framework in this reference. This paper discusses the architecture of this SAMOA framework and its directory structure. Also it expresses a practical experience of configuring and deployment of the tool for handling massive online analysis on Big Data.

View PDFchevron_right
A review on big data real-time stream processing and its scheduling techniques
Stavros Souravlas

International Journal of Parallel, Emergent and Distributed Systems, 2019

Over the last decade, several interconnected disruptions have happened in the large scale distributed and parallel computing landscape. The volume of data currently produced by various activities of the society has never been so big and is generated at an increasing speed. Data that is received in real-time can become way too valuable at the time it arrives and supports valuable decision making. Systems for managing data streams is not a recently developed concept but its becoming more important due to the multiplication of data stream sources in the context of IoT. This paper refers to the unique processing challenges posed by the nature of streams, and the related mechanisms used to face them in the big data era. Several cloud systems emerged to enable distributed processing of streams of big data. Distributed stream management systems (DSMS) along with their strengths and limitations are presented and compared. Computations in these systems demand elaborate orchestration over a collection of machines. Consequently, a classification and literature review on these systems' scheduling techniques and their enhancements is also provided.

View PDFchevron_right
Big data: From beginning to future
Salimah Mokhtar

International Journal of Information Management, 2016

Big data is a potential research area receiving considerable attention from academia and IT communities. In the digital world, the amounts of data generated and stored have expanded within a short period of time. Consequently, this fast growing rate of data has created many challenges. In this paper, we use structuralism and functionalism paradigms to analyze the origins of big data applications and its current trends. This paper presents a comprehensive discussion on state-of-the-art big data technologies based on batch and stream data processing. Moreover, strengths and weaknesses of these technologies are analyzed. This study also discusses big data analytics techniques, processing methods, some reported case studies from different vendors, several open research challenges, and the opportunities brought about by big data. The similarities and differences of these techniques and technologies based on important parameters are also investigated. Emerging technologies are recommended as a solution for big data problems.

View PDFchevron_right
A Survey on Big Data in Real Time
International Journal IJRITCC

Big data is data whose characteristics force us to look beyond the traditional methods that are prevalent at the time. Online news, micro-blogs, search queries are just a few examples of these continuous streams of user activities. Evolving data streams methods are becoming a low-cost, green methodology for real time online prediction and analysis.Heterogeneity,scale,timeliness,complexity, and privacy problems with Big Data impede progress at all phases of the pipeline that can create value from data. The problems start right away during data acquisition, when the data tsunami requires us to make decisions, currently in an ad hoc manner, about what data to keep and what to discard, and how to store what we keep reliably with the right metadata.

View PDFchevron_right
A Comparative Study on Streaming Frameworks for Big Data
Haithem Mezni

2018

Recently, increasingly large amounts of data are generated from a variety of sources. Existing data processing technologies are not suitable to cope with the huge amounts of generated data. Yet, many research works focus on streaming in Big Data, a task referring to the processing of massive volumes of structured/unstructured streaming data. Recently proposed streaming frameworks for Big Data applications help to store, analyze and process the continuously captured data. In this paper, we discuss the challenges of Big Data and we survey existing streaming frameworks for Big Data. We also present an experimental evaluation and a comparative study of the most popular streaming platforms.

View PDFchevron_right

References (151)

  1. Mavragani A, Ochoa G, Tsagarakis KP. Assessing the methods, tools, and statistical procedures in Google trends research: systematic review. J Med Internet Res. 2018;20(11):e270.
  2. Sun D, Zhang G, Zheng W, Li K. Key technologies for big data stream computing. In: Li K, Jiang H, Yang LT, Guz- zocrea A, editors. Big data algorithms, analytics and applications. New York: Chapman and Hall/CRC; 2015. p. 193-214. ISBN 978-1-4822-4055-9.
  3. Qian ZP, He Y, Su CZ et al. TimeStream: Reliable stream computation in the cloud. In: Proc. 8th ACM European conference in computer system, EuroSys 2013. Prague: ACM Press; 2013. p. 1-4.
  4. Liu R, Li Q, Li F, Mei L, Lee, J. Big data architecture for IT incident management. In: Proceedings of IEEE international conference on service operations and logistics, and informatics (SOLI), Qingdao, China. 2014. p. 424-9.
  5. Sakr S. An introduction to Infosphere streams: A platform for analysing big data in motion. IBM. 2013. https :// www.ibm.com/devel operw orks/libra ry/bd-strea msint ro/index .html. Accessed 7 Oct 2018.
  6. Xhafa F, Naranjo V, Caballé S. Processing and analytics of big data stream with Yahoo!S4. In: 2015 IEEE 29th inter- national conference on advanced information networking and applications, Gwangiu, South Korea, 24-27 March 2015. 2015. https ://doi.org/10.1109/aina.2015.194.
  7. Marz N. Storm: distributed and fault-tolerant real-time computation. In: Paper presented at Strata conference on making data work, Santa Clara, California, 28 Feb-1 March 2012. 2012. https ://cdn.oreil lysta tic.com/en/asset s/1/ event /75/Storm _%20dis tribu ted%20and %20fau lt-toler ant%20rea ltime %20com putat ion%20Pre senta tion.pdf. Accessed 25 Jan 2018.
  8. Ballard C, Farrell DM, Lee M, Stone PD, Thibault S, Tucker S. IBM InfoSphere Streams: harnessing data in motion. IBM Redbooks. 2010.
  9. Joseph S, Jasmin EA, Chandran S. Stream computing: opportunities and challenges in smart grid. Procedia Tech- nol. 2015;21:49-53.
  10. IBM Research (no date) Stream computing platforms, applications and analytics. IBM. http://resea rcher .watso n.ibm.com/resea rcher /view_grp.php?id=2531 Accessed 5 Mar 2019.
  11. Gantz J, Reinsel D. The digital universe in 2020: big data, bigger digital shadows, and biggest growth in the Far East. New York: IDC iView: IDC Analyse future; 2012.
  12. Cortes R, Bonnaire X, Marin O, Sens P. Stream processing of healthcare sensor data: studying user traces to identify challenges from a big data perspective. The 4th international workshop on body area sensor networks (BAS- Net-2015). Procedia Comput Sci. 2015;52:1004-9.
  13. Chung D, Shi H. Big data analytics: a literature review. J Manag Anal. 2015;2(3):175-201.
  14. Lu J, Li D. Bias correction in a small sample from big data. IEEE Trans Knowl Data Eng. 2013;25(11):2658-63.
  15. Garzo A, Benczur AA, Sidlo CI, Tahara D, Ywatt EF. Real-time streaming mobility analytics. In: Proc. 2013 IEEE interna- tional conference on big data, big data, Santa Clara, CA, United States, IEEE Press. 2013. p 697-702.
  16. Zaharia M, Das T, Li H, Hunter T, Shenker S, Stoica I. Discretized streams: fault-tolerant streaming computation at scale. In: Proc. the 24th ACM symposium on operating system principles, SOSP 2013, Farmington, PA, United States. New York: ACM Press; 2013. p. 423-38.
  17. Fan J, Liu H. Statistical analysis of big data on pharmacogenomics. Adv Drug Deliv Rev. 2013;65(7):987-1000.
  18. Bifet A, Holmes G, Kirkby R, Pfahringer B. Moa: massive online analysis. J Mach Learn Res. 2010;11:1601-4.
  19. Akter S, Fosso WS. Big data analytics in e-commerce: a systematic review and agenda for future research. Electr Markets. 2016;26:173-94.
  20. Sivarajah U, Kamal MM, Irani Z, Weerakkody V. Critical analysis of big data challenges and analytical methods. J Bus Res. 2016;70:263-86.
  21. Wienhofen LW, Mathisen BM, Roman D. Empirical big data research: a systematic literature mapping. CoRR, abs/1509.03045. 2015.
  22. Habeeb RAA, Nasaruddin F, Gani A, Hashem IAT, Ahmed E, Imran M. Real-time big data processing for anomaly detection: a survey. Int J Inform Manage. 2018;45:289-307. https ://doi.org/10.1016/j.ijinf omgt.2018.08.006.
  23. Mehta N, Pandit A. Concurrence of big data analytics in healthcare: a systematic review. Int J Med Inform. 2018;114:57-65.
  24. Kitchenham BA, Charters S. Guidelines for performing systematic literature review in software engineering. Techni- cal report 2(3), EBSE-2007-01, Keele University and University of Durham. 2007.
  25. Host M, Orucevic-Alagic A. A systematic review of research on open source software in commercial software prod- uct development. 2013. http://www.bcs.org/uploa d/pdf/ewic_ea10_sessi on5pa per2.pdf. Accessed 2 Mar 2018.
  26. Millman N. Analytics for business. Computerworld. 2014. https ://www.compu terwo rld.com/artic le/24758 40/big- data/8-consi derat ions-when-selec ting-big-data-techn ology .html. Accessed 7 Oct 2018.
  27. Oussous A, Benjelloun F, Lachen AA, Belfkih S. Big data technologies: a survey. J King Saud Univ Comput Inform Sci. 2018;30:431-48.
  28. Becker H, Naaman M, Gravano L. Learning similarity metrics for event identification in social media. In: Proceed- ings of the third ACM international conference on web search and data mining (WSDM'10), ACM New York, NY, USA, 4-6 Feb 2010. 2010. p. 291-300.
  29. Aggarwal CC, Zhai C. A survey of text clustering algorithms. In: Aggarwal CC, Zhai C, editors. Mining text data. New York: Springer; 2012. p. 77-128.
  30. Panagiotou N, Katakis I, Gunopulos D. Detecting events in online social networks: Definitions, trends and chal- lenges. In: Michaelis S, et al., editors. Solving large scale learning tasks: challenges and algorithms. Lecture Notes in Computer Science, vol. 9850. Cham: Springer; 2016. p. 42-84. https ://doi.org/10.1007/978-3-319-41706 -6_2.
  31. Deepa MS, Sujatha N. Comparative study of various clustering techniques and its characteristics. Int J Adv Netw Appl. 2014;5(6):2104-16.
  32. Reddy KSS, Bindu CS. A review of density-based clustering algorithms for big data analysis. In: International confer- ence on I-SMAC (IoT in Social, Mobile, Analytic, and Cloud), Palladam, India 10-11 February 2017, IEEE. 2017. https ://doi.org/10.1109/i-smac.2017.80583 22.
  33. Pelkowitz L. A continuous relaxation labelling algorithm for Markov random fields. IEEE Trans Syst Man Cybern. 1990;20:709-15.
  34. Li SZ. Markov random field modelling in image analysis. New York: Springer; 2001.
  35. Zhong S. Efficient streaming text clustering. Neural Netw. 2005;18:5-6.
  36. Aggarwal CC, Yu PS. A framework for clustering massive text and categorical data streams. In: Proceedings of the sixth SIAM international conference on data mining, Bethesda, MD, USA, 20-22 Apr 2016. 2006. https ://doi. org/10.1137/1.97816 11972 764.44.
  37. Li H, Jiang X, Xiong L, Liu J. Differentially private histogram publication for dynamic datasets: an adaptive sampling approach. Proc ACM Int Conf Knowl Manag. 2015. p. 1001-10. https ://doi.org/10.1145/28064 16.28064 41.
  38. Deng JD. Outline detection energy data streams using incremental and kernel PCA algorithms. 2016 IEEE 16th international conference on data mining workshops. 2016. p. 390-7. https ://doi.org/10.1109/icdmw .2016.158.
  39. Limsopatham N, Collier N. Adapting phrase-based machine translation to normalise medical terms in social media messages. In: Proceedings of the 2015 conference on empirical methods in natural language processing, EMNLP 2015, Lisbon. 2015. p.ρ 1675-80.
  40. Kaushik R, Apoorva CS, Mallya D, Chaitanya JNVK, Kamath SS. Sociopedia: an interactive system for event detec- tion and trend analysis for Twitter data. In: Nagar A, Mohapatra D, Chaki N (eds) Smart innovation, systems and technologies, proceedings of 3rd international conference on advanced computing, networking and informatics. New Delhi: Springer; 2016.
  41. Carter S, Weerkamp W, Tsagkias E. Microblog language identification: overcoming the limitations of short, unedited and idiomatic text. Lang Resour Eval J. 2013;47(1):195-215.
  42. Pooja P, Pandey A. Impact of memory intensive applications on performance of cloud virtual machine. In: Pro- ceedings of 2014 recent advances in engineering and computational sciences (RAECS), UIET Panjab University Chandigarh, 6-8 March 2014. 2014. p. 1-6. https ://doi.org/10.1109/raecs .2014.67996 29.
  43. Chang M, Choi IS, Niu D, Zheng H. Performance impact of emerging memory technologies on big data applica- tions: a latency-programmable system emulation approach. In: Proceedings of 2018 on great lake symposium on VLSI (GLSVLSI'18), Chicago, IL, USA, ACM New York, NY, USA, 23-25 May 2018. 2018. p. 439-42. https ://doi. org/10.1145/31945 54.31946 33.
  44. Yang W, Da Silva A, Picard ML. Computing data quality indicators on big data streams using a CEP. In: International workshop on computational intelligence for multimedia understanding IWCIM, Prague, Czech Republic, 29-30 October 2015. 2015.
  45. Neumeyer L, Robbins B, Nair A, Kesari A. S4: Distribute stream computing platform. In: Proceedings of the 2010 IEEE international conference on data mining workshops. 2010. p. 170-7. https ://doi.org/10.1109/icdmw .2010.172.
  46. Inoubli W, Aridhi S, Mezni H, Maddouri M, Nguifo E. A comparative study on streaming frameworks for big data. In: 44th international conference on very large databases: workshop LADaS-Latin American Data Science, Aug 2018, Rio de Janeiro, Brazil. 2018. p. 1-8.
  47. Peng D, Dabek F Large-scale incremental processing using distributed transactions and notifications. In: Proc 9th USENIX conf oper sys. des implement, Vancouver, BC, Canada, 4-6 Oct 2010. 2010. p. 1-15.
  48. Marz N. Trident. 2012. https ://githu b.com/natha nmarz /storm /wiki/Tride nt-tutor ial. Accessed 8 Mar 2018.
  49. Babcock B, Babu S, Datar M, Motwani R, Widom J. Models and issues in data stream systems. In: Proc of the 21st ACM SIGACT-SIGMOD-SIGART symposium on principles of database systems (PODS), Madison, Wisconsin, 3-5 June 2002. 2002. p. 1-16.
  50. Chandrasekaran S, Cooper O, Deshpande A, Franklin MJ, Hellerstein JM, Hong W, Krishnamurthy S, Madden SR, Reiss F, Shah MA. TelegraphCQ: Continuous dataflow processing. In: Proceedings of the 2003 ACM SIGMOD inter- national conference on management of data, San Diego, California, 9-12 Jun 2003. 2003. p. 668.
  51. Abadi DJ, Ahmad Y, Balazinska M, Cherniack M, Hwang JH, Lindner W, Maskey AS, Rasin E, Ryvkina E, Tatbul N, Xing Y, Zdonik S. The design of the borealis stream processing engine. Second biennial conference on innovative data systems research (CIDR 2005). CA: Asilomar; 2005. p. 277-89.
  52. Groleat T. High-performance traffic monitoring for network security and management. Human-computer interac- tion [cs.HC]. Télécom Bretagne; Université de Bretagne Occidentale; 2014.
  53. Kamburugamuve S, Fox G, Leake D, Qiu J. Survey of distributed stream processing for large stream sources. Grids UCS Indiana Educ. 2013. https ://doi.org/10.13140 /rg.2.1.3856.2968.
  54. Murthy S. What are the disadvantages of Redis? 2016. https ://www.quora .com/What-are-the-disad vanta ges-of- Redis . Accessed 8 Mar 2018.
  55. Su X, Gilman E, Wetz P, Riekki J, Zuo Y, Leppanen T. Stream reasoning for the internet of things: challenges and gap analysis. WIMS '16 proceedings of the 6th international conference on web intelligence, mining and semantics, Nîmes, France-June 13-15, New York: ACM. Article no 1. 2016. https ://doi.org/10.1145/29128 45.29128 53.
  56. Morales GDF, Bifet A. SAMOA: scalable advanced massive online analysis. J Mach Learn Res. 2015;16(1):149-53.
  57. Amazon Web Services. Lambda architecture for batch and stream processing. 2018. https ://d1.awsst atic.com/ white paper s/lambd a-archi tecur e-on-for-batch -aws.pdf Accessed 2 May 2019.
  58. Kreps J. Questioning the Lambda architecture. 2014. https ://www.oreil ly.com/ideas /quest ionin g-the-lambd a-archi tectu re. Accessed 2 May 2019.
  59. Tay Y. Data generation for application-specific benchmarking. Proc VLDB Endowment. 2011;4(12):1470-3.
  60. HiBench big data benchmark suite. https ://githu b.com/intel -hadoo p/HiBen ch. Accessed 21 Dec 2018.
  61. Hadoop 1.2.1 Documentation. GridMix. https ://hadoo p.apach e.org/docs/r1.2.1/gridm ix.html. Accessed 8 Mar 2018.
  62. Ouaknine K, Carey M, KirkPatrick S. The PigMix benchmark on Pig, MapReduce, and HPCC systems. 2015 IEEE international conference on big data, New York, NY, USA, 27 June-2 July 2015. p. 643-8. https ://doi.org/10.1109/ bigda tacon gress .2015.99.
  63. Ghazal A, Rabl T, Hu M, Raab F, Poess M, Crolotte A, Jacobson H. BigBench: towards an industry standard bench- mark for big data analytics. In: Proceedings of the 2013 ACM SIGMOID international conference on management of data, New York, NY, USA, 22-27 Jun 2013. p. 1197-203.
  64. Bergamaschi S, Gagliardelli L, Simonini G, Zhu S. BigBench workload executed by using apache flink. Procedia Manuf. 2017;11:695-702. https ://doi.org/10.1016/j.promf g.2017.07.169.
  65. Wang L, Zhan J, Luo C, Zhu Y, Yang Q, He Y, et al. BigDataBench: a big data benchmark suite from internet services. In: 2014 IEEE 20th international symposium on high performance architecture (HPCA), Orlando, FL, USA: IEEE, 15-19 February 2014. 2014. https ://doi.org/10.1109/hpca.2014.68359 58.
  66. Gao W, Zhan J, Wang L, Luo C, Zheng D, Wen X, et al. BigDataBench: A scalable and unified big data and AI bench- mark suite. 2018. arXiv.org > cs > arXiv :1802.08254 v2. https ://arxiv .org/abs/1802.08254 v2.
  67. Liao X, Gao Z, Ji W, Wang Y. An enforcement of real-time scheduling in Spark Streaming. 6th international green and sustainable computing conference, IEEE. 2016. https ://doi.org/10.1109/igcc.2015.73937 30. p. 1-6.
  68. Agerri R, Artola X, Beloki Z, Rigau G, Soroa A. Big data for natural language processing: a streaming approach. Knowledge-based systems. 2015;79:36-42 ISSN 0950-7051.
  69. Krawczyk B, Woźniak M. Incremental weighted one-class classifier for mining stationary data streams. J Comput Sci. 2015;9:19-25.
  70. Chan SWK, Chong MWC. Sentiment analysis in financial texts. Decis Support Syst. 2017;94:53-64.
  71. Rakthanmanon T, Campana B, Mueen A, Batista G, Westover B, Zhu Q, Zakaria J, Keogh E. Addressing big data time series: mining trillions of time series subsequences under dynamic time warping. ACM Trans Knowl Discov Data. 2013;7(3):31. https ://doi.org/10.1145/25004 89.
  72. Hadian A, Shahrivari S. High-performance parallel k-means clustering for disk-resident datasets on multi-core CPUs. J Supercomput. 2014;69(2):845-63.
  73. Mozafari B, Zeng K, D' Antoni L, Zaniolo C. High-performance complex event processing over hierarchical data. ACM Trans Datab Syst. 2013;38(4):39. https ://doi.org/10.1145/25367 79.
  74. Sun Y, Wang Z, Liu H, Du C, Yuan J. Online ensemble using adaptive windowing for data streams with concept drift. Int J Distrib Sens Netw. Article ID 4218973, 9 pages. 2016. http://dx.doi.org/10.1155/2016/42189 73.
  75. Nguyen DT, Jung JJ. Real-time event detection on social data stream. Mobile Netw Appl. 2014;20(4):475-86.
  76. Tsagkatakis G, Beferull-Lozano B, Tsakalides P. Singular spectrum-based matrix completion for time series recovery and prediction. EURASIP J Adv Signal Proces. 2016;2016:66. https ://doi.org/10.1186/s1363 4-016-0360-0.
  77. Papapetrou O, Garofalakis M, Deligiannakis A. Sketching distributed sliding-window data streams. VLDB J. 2015;24:345-68. https ://doi.org/10.1007/s0077 8-015-0380-7.
  78. Elkhoukhi H, NaitMalek Y, Berouine A, Bakhouya M, Elouadghiri D, Essaaidi M. Towards a real-time occupancy detection approach for smart buildings. Procedia Comput Sci. 2018;134:114-20.
  79. Chakrabarti C. Delivering interactive access to data at massive scale at Barclays. Austin. 2016.
  80. Kovacevc I, Mekterovic I. Novel BI data architectures. MIPRO 2018, Opatija, Croatia. 2018. p. 1191-6.
  81. Veiga J, Enes J, Exposito RR, Tourino J. BDEv 3.0: energy efficiency and microarchitectural characterization of big data processing frameworks. Fut Generat Comput Syst. 2018;86:565-81.
  82. Tozzi, C. Dummy's guide to batch vs. streaming. Trillium Software. 2017. http://blog.syncs ort.com/2017/07/big- data/big-data-101-batch -strea m-proce ssing /. Accessed 2 Mar 2018.
  83. Dusi M, D'Heureuse N, Huici F, Trammell B, Niccolini S. Blockmon: flexible and high performance big data stream analytics platform and its use cases. NEC Tech J. 2012;7:102-6.
  84. Puthal D, Nepal S, Ranjan R, Chen J. A dynamic prime number based efficient security mechanism for big sensing data streams. J Comput Syst Sci. 2017;83:22-42.
  85. Vanathi R, and Khadir ASA. A robust architectural framework for big data stream computing in personal healthcare real-time analytics. World Congress on Computing and Communication Technologies. 2017. p. 97-104. https ://doi. org/10.1109/wccct .2016.32.
  86. Ma K, Yang B. Stream-based live entity resolution approach with adaptive duplicate count strategy. Int J Web Grid Serv. 2017;13(3):351-73.
  87. Murphy BM, O'Driscoll C, Boylan GB, Lightbody G, Marnane WP. Stream computing for biomedical signal process- ing: A QRS complex detection case study. In: Conf proc IEEE eng med biol soc. 2015. https ://doi.org/10.1109/ embc.2015.73197 41. p. 5928-31.
  88. Apache Spark Streaming-Spark 2.1.0 Documentation. http://spark .apach e.org/strea ming.
  89. Sun H, Birke R, Bjorkqvist M, Chen LY. AccStream: accuracy-aware overload management for stream processing systems. In: IEEE conference on autonomic computing. New York: Elsevier; 2017. p. 39-48.
  90. Canbay Y, Sağıroğlu S. Big data anonymization with spark (UBMK'17). In: 2nd IEEE international conference on computer science and engineering. 2017. p. 833-8.
  91. Sahana RG, Babu BS. Converting an E-commerce prospect into a customer using streaming analytics. In: 2nd international conference on applied and theoretical computing and communication technology (iCATccT) IEEE. 2016. p. 312-7. https ://doi.org/10.1109/icatc ct.2016.79120 14.
  92. Troiano L, Vaccaro A, Vitelli MC. On-line smart grids optimization by case-based reasoning on big data. In: 2016 IEEE workshop on environmental, energy, and structural monitoring systems (EESMS), Bari, Italy, 13-14 Jun 2016.
  93. Joseph S, Jasmin EA. Stream computing framework for outage detection in smart grid. In: Proceedings of 2015 IEEE international conference on power, instrumentation, control and computing (PICC), Thrissur, India, 9-11 Dec 2015. 2015. https ://doi.org/10.1109/picc.2015.74557 44.
  94. Apache. Apache Storm. 2016. http://storm .apach e.org. Accessed 10 Oct 2018.
  95. Gokalp MO, Kocyigit A, Eren PE. A visual programming framework for distributed Internet of Things centric com- plex event processing. Comput Elect Eng. 2018;74:581-604.
  96. Maio CD, Fenza G, Loia E, Orciuoli F. Distributed online temporal fuzzy concept analysis for stream processing in smart cities. J Parallel Distrib Comput. 2017;110:31-41.
  97. Val PB, Garcia NF, Sanchez-Fernandez L, Arias-Fisteus J. Patterns for distributed real-time stream processing. IEEE Trans Parallel Distrib Syst. 2017;2(11):3243-57. https ://doi.org/10.1109/TPDS.2017.27169 29.
  98. Fernandez-Rodrigues JY, Alvarez-Garcia JA, Fisteus JA, Luaces MR, Magana VC. Benchmarking real-time vehicle data streaming models for a smart city. Inform Syst. 2017;72:62-76.
  99. Bifet A. Mining big data in real time. Informatica (Slovenia). 2013;37:15-20.
  100. Apache. Apache Samza-What is Samza? 2016. http://samza .apach e.org. Accessed 8 Oct 2018.
  101. Ananthanarayanan R, Basker V, Das S, Gupta A, Jiang H, Qiu T, Reznichenko A, Ryabkov D, Singh M, Venkataraman S. Photon: fault-tolerant and scalable joining of continuous data streams. In: Proceedings of 2013 ACM SIGMOD international conference on management of data, New York, New York, USA, 22-27 June 2013. 2013. p. 577-88.
  102. Apache Apache Aurora. 2016. http://auror a.apach e.org. Accessed 7 Oct 2018.
  103. Jiang Q, Adaikkalavan R, Chakravarthy S. MavEStream: synergistic integration of stream and event processing. In: 2007 second international conference on digital telecommunications (ICDT'07) San Jose, CA, USA. 2007. p 29-361. https ://doi.org/10.1109/icdt.2007.21 IEEE Xplore.
  104. Yang W, Da Silva A, Picard ML. Computing data quality indicators on big data streams using a CEP. In: 2015 Inter- national workshop on computational intelligence for multimedia understanding (IWCIM), Prague, Czech Republic, 29-30 Oct 2015. 2015.
  105. EsperTech. http://www.esper tech.com. Accessed 8 Oct 2018.
  106. Song M, Kim MC. RT 2 M: real-time twitter trend mining system. In: Proceedings of international conference on social intelligence and technology (SOCIETY), State College, PA, USA, 8-10 May 2013. 2013. p. 64-71.
  107. Barbieri DF, Braga D, Ceri S. Querying RDF streams with C-SPARQL. ACM Sigmoid. 2010;39(1):20-36. https ://doi. org/10.1145/18607 02.18607 05.
  108. Ren X, Khrouf H, Kazi-Aoul Z, ChabChoub Y, Cure O. On measuring performances of C-SPARQL and CQELS. CoRR, abs/1611.08269. 2016.
  109. Morales GF. SAMOA: A platform for mining big data streams. WWW 2013 Companions, Rio de Janeiro, Brazil, 13-17 May 2013. 2013.
  110. Keeney J, Fallon L, Tai W, O'Sullivan D. Towards composite semantic reasoning for real-time network management data enrichment. In: Proceedings of the 11th international conference on network and service management (CNSM), Barcelona, Spain, 9-13 Nov 2013. 2015. p. 182-6.
  111. Le-Phuoc D, Dao-Tran M, Parreira JX, Hauswirth M. A native and adaptive approach for unified processing of linked streams and linked data. In: International semantic web conference, Koblenz, Germany, 23-27 October 2011. 2011. p. 370-88.
  112. Anicic D, Rudolph S, Fodor P, Stojanovic N. Stream reasoning and complex event processing in ETALIS. Sem Web Linked Spatiotemp Data Geo-Ontolo. 2012;3(4):397-407.
  113. Apache Kylin. Kylin cube from streaming (Kafka). 2015. http://kylin .apach e.org/docs1 5/tutor ial/cube_strea ming. html. Accessed 2 Oct 2018.
  114. Splunk. Splunk Stream. 2017. https ://splun kbase .splun k.com/app/1809/. Accessed 2 Oct 2018.
  115. Shnayder V, Chen B, Lorincz K, Fulford-Jones TRF, Welsh M. Sensor networks for medical care. Technical report TR-08-05, Division of Engineering and Applied Sciences, Harvard University. 2005. https ://www.eecs.harva rd.edu/~shnay der/paper s/codeb lue-techr ept05 .pdf. Accessed 8 Oct 2018.
  116. Dror Y. Practical elastic search anomaly detection made powerful with anodot. 2017. https ://www.anodo t.com/ blog/pract ical-elast icsea rch-anoma lydet ectio n-made-owerf ul-with-anodo t/. Accessed 8 Mar 2019.
  117. Baciu G, Li C, Wang Y, Zhang X. Cloudets: Cloud-based cognition for large streaming data. In: Ge N, Lu J, Wang Y, Howard N, Chen P, Tao X, Zhang B, Zadeh LA (eds) Proceedings of IEEE 14th international conference on cognitive informatics and cognitive computing (ICCI*CC'15), Tsinghua, Univ., Beijing, China, 6-8 Jul 2015. 2015. p. 333-8.
  118. Tedeschi A, Benedetto F. A cloud-based big data sentiment analysis application for enterprises' brand monitor- ing in social media streams. In: 2015 IEEE 1st international forum on research and technologies for society and industry leveraging a better tomorrow (RTSI), Turing, Italy, 16-18 Sept 2015. 2015. p 186-91.
  119. Lavin A, Ahmad S. Evaluating real-time anomaly detection algorithms-the Numenta anomaly benchmark. In: 2015 IEEE 14th international conference on machine learning and applications (ICMLA), Miami, FL, USA, 9-11 Dec 2015. 2015. https ://doi.org/10.1109/icmla .2015.141.
  120. Chen X, Chen H, Zhang N, Huang J, Zhang W. Large-scale real-time semantic processing framework for Internet of Things. Int J Distrib Sens Netw. 2015;365372:11. https ://doi.org/10.1155/2015/36537 2.
  121. Branscombe M. How Microsoft's fast track Azure will help businesses conquer IoT. 2015. http://www.techr adar. com/news/inter net/cloud -servi ces/howmi croso ft-s-fast-track -azure -will-help-busin esses -conqu er-iot-12910 25. Accessed 8 Mar 2018.
  122. Biem A, Bouillet E, Feng H, Ranganathan A, Riabov A, Verscheure O, Koutsopoulos H, Moran C. IBM InfoSphere streams for scalable, real-time, intelligent transportation services. SIGMOID'10 Indianapolis, Indiana, USA, 6-11 Jun 2010. 2010. p. 1093-100.
  123. Akidau T, Balikov A, Bekiroglu K, Chernyak S, Haberman J, Lax R, McVeety S, Mills D, Nordstrom P, Whittle S. Mill- Wheel: fault-tolerant stream processing at internet scale. Proc VLDB Endowment. 2013;6(11):1033-44.
  124. Blount M, Ebling MR, Eklund JM, James AG, McGregor C, Percival N, Smith KP, Sow D. Real-time analysis for inten- sive care: development and deployment of the artemis analytic system. IEEE Eng Med Biol Mag. 2010;29(2):110-8. https ://doi.org/10.1109/MEMB.2010.93645 4.
  125. Introducing WSO2 Data Analytics Server. 2015. https ://docs.wso2.com/displ ay/DAS30 0/Intro ducin g+DAS. Accessed 8 Mar 2019.
  126. Ali M, Chandramouli B, Goldstein J, Schindlauer R. The extensibility framework in Microsoft StreamInsight. In: Pro- ceedings of the 2011 IEEE 27th international conference on data engineering (ICDE), Washington, DC, USA, 11-16 Apr 2011. 2011. p. 1242-53.
  127. TIBCO StreamBase Documentation. https ://docs.tibco .com. Accessed 8 Mar 2018.
  128. Wilkes S. Making in-memory computing enterprise-grade-overview-Striim. 2016. http://www.strii m.com/ blog/2016/06/makin g-in-memor ycomp uting -enter prise -grade -overv iew/ Accessed 8 Mar 2019.
  129. Kyvos Insights. Kyvos insights 2018. 2018. https ://www.kyvos insig hts.com/. Accessed 1 Feb 2018.
  130. AtScale. AtScale overview (version 4.1). 2017. http://info.atsca le.com/atsca le-overv iew. Accessed 2 Feb 2018.
  131. AtScale. AtScale. 2018. http://atsca le.com/produ ct/. Accessed 2 Feb 2018.
  132. Gedik B, Andrade H, Wu K, Yu PS, Doo M. Spade: the S declarative stream processing engine. In: 2008 ACM SIG- MOID international conference on management of data, Vancouver, Canada, 9-12 Jun 2008. 2008. p. 1123-34.
  133. Mimic, II. http://physi onet.org/physi obank /datab ase/mimic 2db/. Accessed 4 Nov 2016.
  134. Wu Z, Zou M. An incremental community detection method for social tagging systems using locality sensitive hashing. Neural Netw. 2014;58:14-28. https ://doi.org/10.1016/j.neune t.2014.05.019.
  135. O'Callaghan L, Mishra N, Meyerson A, Guha S, Motwani R. Streaming-data algorithms for high-quality clustering. In: Proceedings of IEEE international conference on data engineering, San Jose, CA, USA, 26 Feb-1 Mar 2002. 2002. p. 685-94.
  136. Aggarwal CC, Han JW, Wang JY. A framework for clustering evolving data streams. In: Proceedings of the 29th VLDB conference, vol. 29, Berlin, Germany, 9-12 Sep 2003. 2003. p. 81-92.
  137. Backhoff O, Ntoutsi E. Scalable online-offline stream clustering in apache spark. In: 2016 IEEE 16th international conference on data mining workshops (ICDMW), Barcelona, Spain, 12-15 Dec 2016. 2016. p. 37-44. https ://doi. org/10.1109/icdmw .2016.0014.
  138. Aggarwal CC, Han J, Wang J, Yu PS. A framework for projected clustering of high dimensional data streams. In: Proceedings of the 30th international conference on very large data bases, 30, Toronto, Canada, 31 Aug-3 Sep 2004. 2004. p. 852-63.
  139. Cao F, Ester M, Qian W, Zhou A. Density-based clustering over an evolving data stream with noise. In: 2006 SIAM conference on data mining. 2006. p. 328-39.
  140. Chen Y, Tu L. Density-based clustering for real-time stream data. In: Proceedings of the 13th ACM SIGKDD interna- tional conference on knowledge discovery and data mining, San Jose, CA, USA, 12-15 Aug 2007. 2007. p. 133-42.
  141. Zhu WH, Yin J, Xie YH. Arbitrary shape cluster algorithm for clustering data stream. J Softw. 2006;17(3):379-87.
  142. Khalilian M, Mustapha N, Sulaiman N. Data stream clustering by divide and conquer approach based on vector model. J Big Data. 2016;3:1. https ://doi.org/10.1186/s4053 7-015-0036-x.
  143. Dai DB, Zhao G, Sun SL. Effective clustering algorithm for probabilistic data stream. J Softw. 2009;20(5):1313-28.
  144. Ding S, Zhang J, Jia H, Qian J. An adaptive density data stream clustering algorithm. Cogn Comput. 2016;8(1):1-9. https ://doi.org/10.1007/s1255 9-015-9342-z.
  145. Choi D, Song S, Kim B, Bae I. Processing moving objects and traffic events based on spark streaming. In: Proceed- ings of the 8th international conference on disaster recovery and business continuity (DRBC), Jeju, South Korea, 25-28 Nov 2015. 2015. p. 4-7.
  146. Chen XJ, Ke J. Fast processing of conversion time data flow in cloud computing via weighted FPtree mining algorithms. In: 2015 IEEE 12th intl conf on ubiquitous intelligence and computing and 2015 IEEE 12th intl conf on autonomic and trusted computing and 2015 IEEE 15th intl conf on scalable computing and communications and its associated workshops (UIC-ATC-ScalCom), Beijing, China, 10-14 Aug 2015. 2015.
  147. Li T, Wang L. Key technology of online auditing data stream processing. In: 2015 IEEE 12th intl conf on ubiquitous intelligence and computing and 2015 IEEE 12th intl conf on autonomic and trusted computing and 2015 IEEE 15th intl conf on scalable computing and communications and its associated workshops (UIC-ATC-ScalCom), Beijing, China, 10-14 Aug 2015. 2015.
  148. Xiao F, Aritsugi M, Wang Q, Zhang R. Efficient processing of multiple nested event pattern queries over multi- dimensional event streams based on a triaxial hierarchical model. Artif Intell Med. 2016;72(1):56-71. https ://doi. org/10.1016/j.artme d.2016.08.002.
  149. Wang Z, Zhao Z, Weng S, Zhang C. Incremental multiple instance outlier detection. Neural Comput Appl. 2015;26:957-68. https ://doi.org/10.1007/s0052 1-014-1750-6.
  150. Ruiz E, Casillas J. Adaptive fuzzy partitions for evolving association rules in big data stream. Int J Approx Reasoning. 2018;93:463-86.
  151. Jadhav SA, Kosbatwar SP. Concept-adapting very fast decision tree with misclassification error. Int J Adv Res Com- put Eng Technol (IJARCET). 2016;5(6):1763-7.

Related papers

A Review on Big Data Stream Processing Applications: Contributions, Benefits, and Limitations
Ronal Hadi

JOIV : International Journal on Informatics Visualization, 2021

The amount of data in our world has been rapidly keep growing from time to time. In the era of big data, the efficient processing and analysis of big data using machine learning algorithm is highly required, especially when the data comes in form of streams. There is no doubt that big data has become an important source of information and knowledge in making decision process. Nevertheless, dealing with this kind of data comes with great difficulties; thus, several techniques have been used in analyzing the data in the form of streams. Many techniques have been proposed and studied to handle big data and give decisions based on off-line batch analysis. Today, we need to make a constructive decision based on online streaming data analysis. Many researchers in recent years proposed some different kind of frameworks for processing the big data streaming. In this work, we explore and present in detail some of the recent achievements in big data streaming in term of contributions, benefi...

View PDFchevron_right
Mapping the Big Data Landscape: Technologies, Platforms and Paradigms for Real-Time Analytics of Data Streams
Frederic Stahl

IEEE Access, 2021

The 'Big Data' of yesterday is the 'data' of today. As technology progresses, new challenges arise and new solutions are developed. Due to the emergence of Internet of Things applications within the last decade, the field of Data Mining has been faced with the challenge of processing and analysing data streams in real-time, and under high data throughput conditions. This is often referred to as the Velocity aspect of Big Data. Whereas there are numerous reviews on Data Stream Mining techniques and applications, there is very little work surveying Data Stream processing paradigms and associated technologies, from data collection through to pre-processing and feature processing, from the perspective of the user, not that of the service provider. In this article, we evaluate a particular type of solution, which focuses on streaming data, and processing pipelines that permit online analysis of data streams that cannot be stored as-is on the computing platform. We review foundational computational concepts such as distributed computation, faulttolerant computing, and computational paradigms/architectures. We then review the available technological solutions, and applications that pertain to data stream mining as case studies of these theoretical concepts. We conclude with a discussion of the field of data stream processing/analytics, future directions and research challenges. INDEX TERMS Big data applications, Internet of Things (IoT), edge computing, distributed computing, pipeline processing.

View PDFchevron_right
A Systematic Study for Big Data Stream Processing Frameworks
Ali Yazici

Journal on Advances in Theoretical and Applied Informatics

The choice of the most effective stream processing framework (SPF) for Big Data has been an important issue among the researchers and practioners. Each of the SPFs has different cutting edge technologies in their steps of processing the data in motion that gives them a better advantage over the others. Even though, these technologies used in each SPF might better them, it is still difficult to conclude which framework berforms better under different scenarios and conditions. In this paper, we aim to show trends and differences about several SPFs for Big Data by using the so called Systematic Mapping (SM) approach using the related research outcomes. To achieve this objective, nine research questions (RQs) were raised, in which 91 studies that were conducted between 2010 and 2015 were evaluated. We present the trends by classifying the research on SPFs with respect to the proposed RQs which can direct researchers in getting an state-of-art overview of the field.

View PDFchevron_right
Data Stream Mining Algorithms in Big Data: A Survey
Nilesh Vispute

The infrastructure build in the big data platform is reliable to challenge the commercial and non-commercial IT development communities of data streams in high dimensional data cluster modeling. The APSO ie., Accelerated Particle Swarm Optimization is a technique which commonly known for data's are sourced to accumulate their continuation in the batch model induction algorithms which is not feasible for the real time applications[8]. In this project, a new technique has been introduced ie., supervised machine learning methods for developing dynamic resource allocation which targets a user defined learning method to identify the workload patterns and also feature selection is used to process the loaded data in the searched space to form the subset of the optimal solution in size to interact their demands in computation. The main theme of this project is to feed up the data in a lightweight feature selection and to designed the streaming data by using APSO, which enables the swarm search layered forms related query dependent performance in the process scheduling and data accuracy in the iterative manner. Thus the Big data in APSO are put under the test of new feature selection algorithm for performance evaluation. I. INTRODUCTION Recently a lot of news in the media advocates the hype of Big Data that are manifested in three problematic issues. They are the 3V challenges known as: Velocity problem that gives rise to a huge amount of data to be handled at an escalating high speed; Variety problem that makes data processing and integration difficult because the data come from various sources and they are formatted differently; and Volume problem that makes storing, processing, and analysis over them both computational and archiving challenging. In views of these 3V challenges, the traditional data mining approach which are based on the full batch-mode learning may run short in meeting the demand of analytic efficiency. That is simply because the traditional data mining model construction techniques require loading in the full set of data, and then the data are partitioned according to some divide-and-conquer strategy; two classical algorithms are Classification And Regression Tree algorithm (CART) for decision tree induction and Rough-set discrimination[8]. Each time when fresh data arrive, which is typical in the data collection process that makes the big data inflate to bigger data, the traditional induction method needs to rerun and the model that was built needs to be built again with the inclusion of new data.

View PDFchevron_right
Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review
Erum Mehmood

IEEE Access

Contribution: Recently, real-time data warehousing (DWH) and big data streaming have become ubiquitous due to the fact that a number of business organizations are gearing up to gain competitive advantage. The capability of organizing big data in efficient manner to reach a business decision empowers data warehousing in terms of real-time stream processing. A systematic literature review for real-time stream processing systems is presented in this paper which rigorously look at the recent developments and challenges of real-time stream processing systems and can serve as a guide for the implementation of real-time stream processing framework for all shapes of data streams. Background: Published surveys and reviews either cover papers focusing on stream analysis in applications other than real-time DWH or focusing on extraction, transformation, loading (ETL) challenges for traditional DWH. This systematic review attempts to answer four specific research questions. Research Questions: 1)Which are the relevant publication channels for realtime stream processing research? 2) Which challenges have been faced during implementation of real-time stream processing? 3) Which approaches/tools have been reported to address challenges introduced at ETL stage while processing real-time stream for real-time DWH? 4) What evidence have been reported while addressing different challenges for processing real-time stream? Methodology: A systematic literature was conducted to compile studies related to publication channels targeting real-time stream processing/joins challenges and developments. Following a formal protocol, semi-automatic and manual searches were performed for work from 2011 to 2020 excluding research in traditional data warehousing. Of 679,547 papers selected for data extraction, 74 were retained after quality assessment. Findings: This systematic literature highlights implementation challenges along with developed approaches for real-time DWH and big data stream processing systems and provides their comparisons. This study found that there exists various algorithms for implementing real-time join processing at ETL stage for structured data whereas less work for un-structured data is found in this subject matter. INDEX TERMS Real-time stream processing, big data streaming, structured/un-structured data, ETL, systematic literature review.

View PDFchevron_right

Related topics

  • Computer Science

    • Cited by

    Disruptive Technologies for Disruptive Innovations: Challenges and Opportunities
    Jamal Al-karaki

    Advances in Intelligent Systems and Computing

    Disruptive technologies continuously and significantly alter the way people communicate and collaborate as well as the way industries operate today and in the future. To create new business models and opportunities, several combinations of disruptive technologies are being introduced nowadays. Among these technologies, cloud computing, IoT, Blockchain, artificial intelligence, social networks and media, big data, and 5G are mostly used. For instance, Blockchain technology made distributed solutions feasible and popular. On the other hand, big data and the social media are two contemporary technologies which lament significant impact on business and society. This paper presents a holistic approach to integration perspectives of these technologies considering many challenges like security and privacy. This paper also surveys the most relevant work in order to analyze how some of these technologies could potentially improve each other.

    View PDFchevron_right
    Mining frequent itemsets from streaming transaction data using genetic algorithms
    Sikha Bagui

    Journal of Big Data

    This paper presents a study of mining frequent itemsets from streaming data in the presence of concept drift. Streaming data, being volatile in nature, is particularly challenging to mine. An approach using genetic algorithms is presented, and various relationships between concept drift, sliding window size, and genetic algorithm constraints are explored. Concept drift is identified by changes in frequent itemsets. The novelty of this work lies in determining concept drift using frequent itemsets for mining streaming data, using the genetic algorithm framework. Formulas have been presented for calculating minimum support counts in streaming data using sliding windows. Testing highlighted that the ratio of the window size to transactions per drift was a key to good performance. Getting good results when the sliding window size was too small was a challenge since normal fluctuations in the data could appear to be a concept drift. Window size must be managed in conjunction with support...

    View PDFchevron_right
    Machine Learning Research Trends in Africa: A 30 Years Overview with Bibliometric Analysis Review
    Abiodun Ikotun

    Archives of Computational Methods in Engineering

    The machine learning (ML) paradigm has gained much popularity today. Its algorithmic models are employed in every field, such as natural language processing, pattern recognition, object detection, image recognition, earth observation and many other research areas. In fact, machine learning technologies and their inevitable impact suffice in many technological transformation agendas currently being propagated by many nations, for which the already yielded benefits are outstanding. From a regional perspective, several studies have shown that machine learning technology can help address some of Africa’s most pervasive problems, such as poverty alleviation, improving education, delivering quality healthcare services, and addressing sustainability challenges like food security and climate change. In this state-of-the-art paper, a critical bibliometric analysis study is conducted, coupled with an extensive literature survey on recent developments and associated applications in machine lea...

    View PDFchevron_right
    A scalable and real-time system for disease prediction using big data processing
    Aziza EL OUAAZIZI

    Multimedia Tools and Applications, 2023

    The growing chronic diseases patients and the centralization of medical resources cause significant economic impact resulting in hospital visits, hospital readmission, and other healthcare costs. This paper proposes a scalable and real-time system for disease prediction from medical data streams. This is carried out by integrating Twitter, Apache Kafka, Apache Spark and Apache Cassandra. Thus, Twitter users tweet attributes related to health, Kafka streaming receives all desired tweets attributes and ingest them to Spark streaming. Here, a machine learning algorithm is applied to predict health status and send back a response message through Kafka. The heart disease dataset, obtained from the UCI repository, was used for experiments. In order to enhance prediction accuracy, Relief algorithm is used for features selection. We compared sex types of relevant machine learning algorithms implemented by Spark MLlib such as Random Forest (RF), Naive Bayes, Support Vector Machine, Multilayer Perceptron, Decision Tree and Logistic Regression with the full features as well as selected features. The highest classification accuracy of 92.05% was reported using RF with selected features. The scalability of RF using Spark MLlib and WEKA framework for both training and application stages was measured. The results show significantly better performances of Spark in terms of scalability and computing times.

    View PDFchevron_right
    Time series big data: a survey on data stream frameworks, analysis and algorithms
    Filipe Cabral Pinto

    Journal of Big Data, 2023

    View PDFchevron_right
    580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved