Academia.eduAcademia.edu

Figure 8 – uploaded by Fahim Ullah

See full PDFdownloadDownload figure
Table 8. Comparison of NoSQL data storage tools. CouchDB is a clustered database which means that it enables the execution of one logical database server on multiple servers or virtual machines (VMs). This set-up improves the capacity and availability of the database without modifying the APIs. Terratore is a database for storing documents, which is accessible through the HTTP protocol. It supports both single-cluster and multi-cluster deployments and offers advanced data scaling features. The documents are stored by partitioning and then distributing them across various nodes. Hive is a data warehouse which is built on top of the Hadoop framework and offers data query features by providing an interface such as the SQL for different files and data stored within the Hadoop database [98]. Hbase is a distributed and scalable database for big data which allows random and real-time access to the data for both reading and writing. Neo4j is a graph database which enables the user to perform graphical modeling of big data. It allows developers to handle data by using a graph query language called Cypher which enables them to perform create, read, update, and delete (CRUD) operations on data.

Table 8 Comparison of NoSQL data storage tools. CouchDB is a clustered database which means that it enables the execution of one logical database server on multiple servers or virtual machines (VMs). This set-up improves the capacity and availability of the database without modifying the APIs. Terratore is a database for storing documents, which is accessible through the HTTP protocol. It supports both single-cluster and multi-cluster deployments and offers advanced data scaling features. The documents are stored by partitioning and then distributing them across various nodes. Hive is a data warehouse which is built on top of the Hadoop framework and offers data query features by providing an interface such as the SQL for different files and data stored within the Hadoop database [98]. Hbase is a distributed and scalable database for big data which allows random and real-time access to the data for both reading and writing. Neo4j is a graph database which enables the user to perform graphical modeling of big data. It allows developers to handle data by using a graph query language called Cypher which enables them to perform create, read, update, and delete (CRUD) operations on data.

Related Figures (32)

Figure 1. Methodology for shortlisting research articles for the study.
Figure 1. Methodology for shortlisting research articles for the study.
Note: S1: “Big Data” OR “Technology for big data filtering” OR “Refining big data”, S1*: (TITLE-ABS-KEY(Tools for big data analysis) OR (big data analytics tools) OR (big data visualization technologies) AND PUBYEAR > 2009, S2: (TITLE-ABS-KEY (big data real estate ) OR (big data property management) OR (big data real estate management) OR (big data real estate development) OR (big data property development)) AND PUBYEAR > 2009, $3: (TITLE-ABS-KEY (big data disaster management) OR (big data disaster)) AND PUBYEAR > 2009. Table 1. Initial article retrieval—phase 1 (year 2010-2020).
Note: S1: “Big Data” OR “Technology for big data filtering” OR “Refining big data”, S1*: (TITLE-ABS-KEY(Tools for big data analysis) OR (big data analytics tools) OR (big data visualization technologies) AND PUBYEAR > 2009, S2: (TITLE-ABS-KEY (big data real estate ) OR (big data property management) OR (big data real estate management) OR (big data real estate development) OR (big data property development)) AND PUBYEAR > 2009, $3: (TITLE-ABS-KEY (big data disaster management) OR (big data disaster)) AND PUBYEAR > 2009. Table 1. Initial article retrieval—phase 1 (year 2010-2020).
Figure 2. Most frequent keywords in the big data articles from 2010 to 2020.
Figure 2. Most frequent keywords in the big data articles from 2010 to 2020.
Figure 3. Most frequent keywords used in the big data articles on real estate and property from 2010 to 2020.
Figure 3. Most frequent keywords used in the big data articles on real estate and property from 2010 to 2020.
Figure 5 presents the rough trend that was initially observed when narrowing down papers needed for the temporal review. A steep rise in big data can be seen in the years 2013-2014, 2015-2016, and 2017-2018, while a less substantial incline was seen in 2016-2017. From here onward, the search was further refined, and only those papers which truly suited the purpose of this review were selected.
Figure 5 presents the rough trend that was initially observed when narrowing down papers needed for the temporal review. A steep rise in big data can be seen in the years 2013-2014, 2015-2016, and 2017-2018, while a less substantial incline was seen in 2016-2017. From here onward, the search was further refined, and only those papers which truly suited the purpose of this review were selected.
Figure 5. Big data papers published per year as indexed on Scopus and Web of Science from 2010-2019.
Figure 5. Big data papers published per year as indexed on Scopus and Web of Science from 2010-2019.
Figure 6. Worldwide big data, disaster big data, and real estate big data search trends in last decade. The next stage was based on screening the retrieved articles based on well-defined criteria based on four rules. the aim was Firstly, only articles published from 1 January 2010 and onward were selected, because to keep a recent focus and to cover articles published in the last decade, as the concept of big data and its usage became common only recently, and the last few years saw a rapid rise in technologies being develo ped for big data management and analysis. Secondly, only articles written in the English language were selected; thus, articles written in any other language were excluded Thirdly, only journal articles including original research papers and reviews were included. Articles written as letters, editoria s, conference papers, webpages, or any other nonstandard format were eliminated. Lastly, no duplicate or redundant articles could be present and, thus, when the same article was retrieved from multiple search engines or sources, it was discarded. Finally, a total of 182 published articles were narrowed down after the screening phase for 51 (135), 18 for S1* and 28 for S2, and 19 for S3. These papers were then critically analyzed one by one to determine their fit within the scope of the research objectives and questions, with the aim of bringing the existence of
Figure 6. Worldwide big data, disaster big data, and real estate big data search trends in last decade. The next stage was based on screening the retrieved articles based on well-defined criteria based on four rules. the aim was Firstly, only articles published from 1 January 2010 and onward were selected, because to keep a recent focus and to cover articles published in the last decade, as the concept of big data and its usage became common only recently, and the last few years saw a rapid rise in technologies being develo ped for big data management and analysis. Secondly, only articles written in the English language were selected; thus, articles written in any other language were excluded Thirdly, only journal articles including original research papers and reviews were included. Articles written as letters, editoria s, conference papers, webpages, or any other nonstandard format were eliminated. Lastly, no duplicate or redundant articles could be present and, thus, when the same article was retrieved from multiple search engines or sources, it was discarded. Finally, a total of 182 published articles were narrowed down after the screening phase for 51 (135), 18 for S1* and 28 for S2, and 19 for S3. These papers were then critically analyzed one by one to determine their fit within the scope of the research objectives and questions, with the aim of bringing the existence of
Note: The filters applied were publications from 2010 and onward, the presence of keywords in the title or abstract, English language, and no duplications. Exclusions included short papers, editorial notes, calls for issues, errata, discussions, and closures. Table 2. Articles retrieved after each phase, as well as those filtered and shortlisted for final content analyses.
Note: The filters applied were publications from 2010 and onward, the presence of keywords in the title or abstract, English language, and no duplications. Exclusions included short papers, editorial notes, calls for issues, errata, discussions, and closures. Table 2. Articles retrieved after each phase, as well as those filtered and shortlisted for final content analyses.
Table 3. Most repeated keywords in the big data papers from 2010-2020. After highlighting the most repeated keywords, journals contributing the most to the shortlisted papers were studied. Table 4 shows the top five journals/sources from which the articles were retrieved. An inclusion criterion of at least 15 documents was applied as the filter for shortlisting the top sources. Consequently, the majority of articles hailed from lecture notes in computer science followed by IOP conference series and others.
Table 3. Most repeated keywords in the big data papers from 2010-2020. After highlighting the most repeated keywords, journals contributing the most to the shortlisted papers were studied. Table 4 shows the top five journals/sources from which the articles were retrieved. An inclusion criterion of at least 15 documents was applied as the filter for shortlisting the top sources. Consequently, the majority of articles hailed from lecture notes in computer science followed by IOP conference series and others.
Figure 7. Article types reviewed in the study.
Figure 7. Article types reviewed in the study.
Table 4. Top sources based on number of papers reviewed from 2010-2020.
Table 4. Top sources based on number of papers reviewed from 2010-2020.
Figure 8. Authors’ names, as well as the number of documents and citations, of the 139 reviewed papers from 2010 to 2020. Similarly, once the sources were highlighted, the following analyses were aimed at highlighting the top contributing authors, countries, and organizations contributing to the study area. Figure 8 shows the contributions by authors in terms of the number of documents and their citations. A minimum number of six documents with at least six citations was the filter applied to shortlist these authors.
Figure 8. Authors’ names, as well as the number of documents and citations, of the 139 reviewed papers from 2010 to 2020. Similarly, once the sources were highlighted, the following analyses were aimed at highlighting the top contributing authors, countries, and organizations contributing to the study area. Figure 8 shows the contributions by authors in terms of the number of documents and their citations. A minimum number of six documents with at least six citations was the filter applied to shortlist these authors.
Figure 9. Country-wise contributions to and citations of the 139 reviewed articles from 2010 to 2020 After highlighting the top contributing authors, countries with top contributions to the field of big data were investigated, as shown in Figure 9. A minimum inclusion criterion was set at 10 documents from a specific country among the shortlisted papers. The race is led by China with 34 papers, followed by the United States of America (USA) with 24 papers among the shortlist. However, when it comes to the citations, the USA is leading with 123 citations, followed by China with 58 citations.
Figure 9. Country-wise contributions to and citations of the 139 reviewed articles from 2010 to 2020 After highlighting the top contributing authors, countries with top contributions to the field of big data were investigated, as shown in Figure 9. A minimum inclusion criterion was set at 10 documents from a specific country among the shortlisted papers. The race is led by China with 34 papers, followed by the United States of America (USA) with 24 papers among the shortlist. However, when it comes to the citations, the USA is leading with 123 citations, followed by China with 58 citations.
Table 5. List of affiliated organizations and the number of contributing documents included in the 135 reviewed articles from 2010 to 2020. 3.2. Big Data and Its Seven Vs Big data is the name given to datasets containing large, varied, and complex structures with issues related to storage, analysis, and visualization for data processing [7]. Massive amounts of data are generated from a variety of sources like audios, videos, social networking, sensors, and mobile phones, which are stored in the form of databases that require different applications for the analyses [38]. Big data is characterized by its high volume, sharing, creation, and removal in seconds, along with the high inherent variations and complexities [16]. Thus, it can be structured, unstructured, or semi-structured and vary in the form of text, audio, image, or video [39]. Previously, methods used for the storage and analysis of big data were slow in speed because of the low processing capabilities and lack of technology. Until 2003, humans were able to create a mere five exabytes, whereas, today, in the era of disruption and technological advancements, the same amount of data is created in the span of two days. The rapidness of data creation comes with a set of difficulties that occur in storage, sorting, and categorization of such big data. The expansion of data usage and generation reaches its heights today, and, in 2013, the data were reported to be 2.72 zettabytes, exponentially increasing to date [6].
Table 5. List of affiliated organizations and the number of contributing documents included in the 135 reviewed articles from 2010 to 2020. 3.2. Big Data and Its Seven Vs Big data is the name given to datasets containing large, varied, and complex structures with issues related to storage, analysis, and visualization for data processing [7]. Massive amounts of data are generated from a variety of sources like audios, videos, social networking, sensors, and mobile phones, which are stored in the form of databases that require different applications for the analyses [38]. Big data is characterized by its high volume, sharing, creation, and removal in seconds, along with the high inherent variations and complexities [16]. Thus, it can be structured, unstructured, or semi-structured and vary in the form of text, audio, image, or video [39]. Previously, methods used for the storage and analysis of big data were slow in speed because of the low processing capabilities and lack of technology. Until 2003, humans were able to create a mere five exabytes, whereas, today, in the era of disruption and technological advancements, the same amount of data is created in the span of two days. The rapidness of data creation comes with a set of difficulties that occur in storage, sorting, and categorization of such big data. The expansion of data usage and generation reaches its heights today, and, in 2013, the data were reported to be 2.72 zettabytes, exponentially increasing to date [6].
Initially, big data was characterized by its variety, volume, and velocity, which were known as e three Vs of data [6]; however, later value and veracity were later added to the previously defined pects of the data [40]. Recently, variability and visualization were also added to the characteristics of ¢ data by Sheddon et al. [41]. These seven Vs along with the hierarchy, integrity, and correlation n help integrate the functions of smart real estate including safe, economical, and more intelligent eration, to help the customers make better and more informed decisions [21]. These seven Vs for fining the characteristics of big data are illustrated and summarized in Figure 10. Each of these Vs is plained in the subsequent sections. Variety is one of the important characteristics of big data that refers to the collection of dat ‘om different sources. Data vary greatly in the form of images, audio, videos, numbers, or text [39 yrming heterogeneity in the d atasets [42]. Structured data refer to the data present in tabular form it oreadsheets, and these data are easy to sort because they are already tagged, whereas text, image: nd audio are examples of uns ructured data that are random and relatively difficult to sort [6]. Variet ot only exist in formats and data types but also in different kinds of uses and ways of analyzing th ata [43]. Different aspects of t he variety attribute of big data are summarized in Table 6. The existenc f data in diverse shapes and forms adds to its complexity. Therefore, the concept of a relationa atabase is becoming absurd with the growing diversity in the forms of data. Thus, integration o sing the big data directly in a system is quite challenging. For example, on the worldwide wel NWW), people use various browsers and applications which change the data before sending then ) the cloud [44]. Furthermore, these data are entered manually on the interface and are, therefor 1ore prone to errors, which affects the data integrity. Thus, variety in data implies more chances o rrors. To address this, the concept of data lakes was proposed to manage the big data, which provide schema-less repository for raw data with a common access interface; however, this is prone to dat wamping if the data are just dumped into a data lake without any metadata management. Tools suc ; Constance were proposed and highlighted by Hai et al. [45] for sophisticated metadata managemen ver raw data extracted from heterogeneous data sources. Based on three functional layers of ingestiorz
Initially, big data was characterized by its variety, volume, and velocity, which were known as e three Vs of data [6]; however, later value and veracity were later added to the previously defined pects of the data [40]. Recently, variability and visualization were also added to the characteristics of ¢ data by Sheddon et al. [41]. These seven Vs along with the hierarchy, integrity, and correlation n help integrate the functions of smart real estate including safe, economical, and more intelligent eration, to help the customers make better and more informed decisions [21]. These seven Vs for fining the characteristics of big data are illustrated and summarized in Figure 10. Each of these Vs is plained in the subsequent sections. Variety is one of the important characteristics of big data that refers to the collection of dat ‘om different sources. Data vary greatly in the form of images, audio, videos, numbers, or text [39 yrming heterogeneity in the d atasets [42]. Structured data refer to the data present in tabular form it oreadsheets, and these data are easy to sort because they are already tagged, whereas text, image: nd audio are examples of uns ructured data that are random and relatively difficult to sort [6]. Variet ot only exist in formats and data types but also in different kinds of uses and ways of analyzing th ata [43]. Different aspects of t he variety attribute of big data are summarized in Table 6. The existenc f data in diverse shapes and forms adds to its complexity. Therefore, the concept of a relationa atabase is becoming absurd with the growing diversity in the forms of data. Thus, integration o sing the big data directly in a system is quite challenging. For example, on the worldwide wel NWW), people use various browsers and applications which change the data before sending then ) the cloud [44]. Furthermore, these data are entered manually on the interface and are, therefor 1ore prone to errors, which affects the data integrity. Thus, variety in data implies more chances o rrors. To address this, the concept of data lakes was proposed to manage the big data, which provide schema-less repository for raw data with a common access interface; however, this is prone to dat wamping if the data are just dumped into a data lake without any metadata management. Tools suc ; Constance were proposed and highlighted by Hai et al. [45] for sophisticated metadata managemen ver raw data extracted from heterogeneous data sources. Based on three functional layers of ingestiorz
3.2.2. Volume Table 6. The seven Vs, as well as their key aspects and context of usage. maintenance, and querying, Constance can implement the interface between the data sources and enable the major human-machine interaction, as well as dynamically and incrementally extract and summarize the current metadata of the data lake that can help address and manage disasters and the associated risks [46]. Such data lakes can be integrated with urban big data for smarter real estate management, where, just like the human and non-human resources of smart real estate, urban big data also emerge as an important strategic resource for the development of intelligent cities and strategic directions [21]. Such urban big data can be converged, analyzed, and mined with depth via the Internet of things, cloud computing, and artificial intelligence technology to achieve the goal of intelligent administration of smart real estate.
3.2.2. Volume Table 6. The seven Vs, as well as their key aspects and context of usage. maintenance, and querying, Constance can implement the interface between the data sources and enable the major human-machine interaction, as well as dynamically and incrementally extract and summarize the current metadata of the data lake that can help address and manage disasters and the associated risks [46]. Such data lakes can be integrated with urban big data for smarter real estate management, where, just like the human and non-human resources of smart real estate, urban big data also emerge as an important strategic resource for the development of intelligent cities and strategic directions [21]. Such urban big data can be converged, analyzed, and mined with depth via the Internet of things, cloud computing, and artificial intelligence technology to achieve the goal of intelligent administration of smart real estate.
Table 7. Comparison of data collection tools. 3.4.2. Data Storage
Table 7. Comparison of data collection tools. 3.4.2. Data Storage
Table 9. Comparison of data filtering tools. Import.io is a web data integration tool which transforms unstructured data into a structured format so that they can be integrated into various business applications. After specifying the target website URL, the web data extraction module provides a visual environment for designing automated workflows for harvesting data, going beyond HTML parsing of static content to automate end-user interactions yielding data that would otherwise not be immediately visible. ParseHub is a free, easy to use, and powerful web scraping tool which allows users to get data from multiple pages, as well as interact with AJAX, forms, dropdowns, etc. Mozenda is a web scraping tool which allows a user to scrape text, files, images, and PDF content from web pages with a point-and-click feature. It organizes data files for publishing and exporting them directly to TSV, comma-separated values (CSV), extensible markup language (XML), Excel (XLSX), or JavaScript object notation (JSON) through an API. Content Grabber is a cloud-based web scraping tool that helps businesses of all sizes with data extraction. Primary features of Content Grabber include agent logging, notifications, a customizable user interface, scripting capabilities, scripting, agent debugger, error handling, and data export. Octoparse is a cloud-based data scraping tool which turns web pages into structured spreadsheets within clicks without coding. Scraped data can be downloaded in CSV, Excel, or API format or saved to databases. In order to extract structured data from unstructured data, the data are filtered through some tools which filter out the useful information necessary for the analyses. Some data filtering tools and their features are compared in Table 9. website URL, the web data extraction module provides a visual environment for designing automated
Table 9. Comparison of data filtering tools. Import.io is a web data integration tool which transforms unstructured data into a structured format so that they can be integrated into various business applications. After specifying the target website URL, the web data extraction module provides a visual environment for designing automated workflows for harvesting data, going beyond HTML parsing of static content to automate end-user interactions yielding data that would otherwise not be immediately visible. ParseHub is a free, easy to use, and powerful web scraping tool which allows users to get data from multiple pages, as well as interact with AJAX, forms, dropdowns, etc. Mozenda is a web scraping tool which allows a user to scrape text, files, images, and PDF content from web pages with a point-and-click feature. It organizes data files for publishing and exporting them directly to TSV, comma-separated values (CSV), extensible markup language (XML), Excel (XLSX), or JavaScript object notation (JSON) through an API. Content Grabber is a cloud-based web scraping tool that helps businesses of all sizes with data extraction. Primary features of Content Grabber include agent logging, notifications, a customizable user interface, scripting capabilities, scripting, agent debugger, error handling, and data export. Octoparse is a cloud-based data scraping tool which turns web pages into structured spreadsheets within clicks without coding. Scraped data can be downloaded in CSV, Excel, or API format or saved to databases. In order to extract structured data from unstructured data, the data are filtered through some tools which filter out the useful information necessary for the analyses. Some data filtering tools and their features are compared in Table 9. website URL, the web data extraction module provides a visual environment for designing automated
Table 10. Comparison of data cleaning tools. 3.4.5. Data Analysis and Visualization
Table 10. Comparison of data cleaning tools. 3.4.5. Data Analysis and Visualization
Table 11. Comparison of data visualization tools.
Table 11. Comparison of data visualization tools.
Figure 11. Comparison of performance for (a) PageRank and (b) logistic regression algorithm.
Figure 11. Comparison of performance for (a) PageRank and (b) logistic regression algorithm.
Table 12. Parameter comparison of Hadoop and Apache Spark. RDD—Resilient Distributed Datasets. 3.6. Machine Learning in Data Analytics
Table 12. Parameter comparison of Hadoop and Apache Spark. RDD—Resilient Distributed Datasets. 3.6. Machine Learning in Data Analytics
Table 13. Sub-domains of machine learning. All machine learning techniques are efficient in processing data; however, as the size of the data grows, the extraction and organization of discriminative information from the data pose a challenge to the traditional methods of machine learning. Thus, to cope up with the growing demand of data processing, advanced methods for machine learning are being developed that are intelligent and much efficient for solving big data problems [113]. As such, one developed method is representation learning [114], which eases the task of information extraction by capturing a greater number of input configurations from a reasonably small data size. Furthermore, deep belief networks (DBNs) and convolution neural networks (CNNs) are used extensively for speech and hand-written digit recognition [115]. Deep learning methods with higher processing power and advanced graphic processors are used on large databases [113]. Traditional methods of machine learning possess centralized processing, which is addressed with the use of distributed learning that distributes the data among various workstations, making the process of data analysis much faster. Classical methods of machine learning mostly use the same feature space for training and testing of the dataset, which
Table 13. Sub-domains of machine learning. All machine learning techniques are efficient in processing data; however, as the size of the data grows, the extraction and organization of discriminative information from the data pose a challenge to the traditional methods of machine learning. Thus, to cope up with the growing demand of data processing, advanced methods for machine learning are being developed that are intelligent and much efficient for solving big data problems [113]. As such, one developed method is representation learning [114], which eases the task of information extraction by capturing a greater number of input configurations from a reasonably small data size. Furthermore, deep belief networks (DBNs) and convolution neural networks (CNNs) are used extensively for speech and hand-written digit recognition [115]. Deep learning methods with higher processing power and advanced graphic processors are used on large databases [113]. Traditional methods of machine learning possess centralized processing, which is addressed with the use of distributed learning that distributes the data among various workstations, making the process of data analysis much faster. Classical methods of machine learning mostly use the same feature space for training and testing of the dataset, which
Table 14. Issues and possible solutions of machine learning for big data. AI and machine learning methods are being increasingly integrated in systems dealing with a wide variety of issues related to disasters. This includes disaster prediction, risk assessment, detection, susceptibility mapping, and disaster response activities such as damage assessment after the occurrence of a disaster. In Nepal, in April 2015, an earthquake of 7.8 magnitude hit 21 miles off the southeast coast of Lamjung. The standby task force was successful in mobilizing 3000 volunteers across the country within 12 hours after the quake, which was possible due to the revolutionized AI system in Nepal. Volunteers in that area started tweeting and uploading crisis-related photographs on social media. Artificial Intelligence for Disaster Response (AIDR) used those tagged tweets to identify the needs of people based on categories such as urgent need, damage to infrastructure, or even help regarding resource deployment. Similarly, Qatar developed a tool known as the Qatar Computing Research Institute (QCRI) for disaster management. The tool was developed by the Qatar Foundation to increase awareness and to develop education and science in a community. For disaster risk management, QCRI aims to provide its services by increasing the efficiency of agencies and volunteer facilities. The tool has an AI system installed which helps in recognizing tweets and texts regarding any devastated area or crisis. The QCRI then provides an immediate solution to overcome the crisis [122]. OneConcern is a tool developed o analyze disaster situations. The tool creates a comprehensive picture of the location during an emergency operation. This image is used by emergency centers to investigate the situation and provide an immediate response in the form of relief goods or other rescue efforts. The tool also helps in the creation of a planning module that can be useful in identifying and determining the areas prone to a disaster. The vulnerable areas can then be evacuated to avoid loss of life. Until now, OneConcern identified 163,696 square miles area and arranged shelter for 39 million people. It
Table 14. Issues and possible solutions of machine learning for big data. AI and machine learning methods are being increasingly integrated in systems dealing with a wide variety of issues related to disasters. This includes disaster prediction, risk assessment, detection, susceptibility mapping, and disaster response activities such as damage assessment after the occurrence of a disaster. In Nepal, in April 2015, an earthquake of 7.8 magnitude hit 21 miles off the southeast coast of Lamjung. The standby task force was successful in mobilizing 3000 volunteers across the country within 12 hours after the quake, which was possible due to the revolutionized AI system in Nepal. Volunteers in that area started tweeting and uploading crisis-related photographs on social media. Artificial Intelligence for Disaster Response (AIDR) used those tagged tweets to identify the needs of people based on categories such as urgent need, damage to infrastructure, or even help regarding resource deployment. Similarly, Qatar developed a tool known as the Qatar Computing Research Institute (QCRI) for disaster management. The tool was developed by the Qatar Foundation to increase awareness and to develop education and science in a community. For disaster risk management, QCRI aims to provide its services by increasing the efficiency of agencies and volunteer facilities. The tool has an AI system installed which helps in recognizing tweets and texts regarding any devastated area or crisis. The QCRI then provides an immediate solution to overcome the crisis [122]. OneConcern is a tool developed o analyze disaster situations. The tool creates a comprehensive picture of the location during an emergency operation. This image is used by emergency centers to investigate the situation and provide an immediate response in the form of relief goods or other rescue efforts. The tool also helps in the creation of a planning module that can be useful in identifying and determining the areas prone to a disaster. The vulnerable areas can then be evacuated to avoid loss of life. Until now, OneConcern identified 163,696 square miles area and arranged shelter for 39 million people. It
Table 15. Organizations using big data in smart real estate, focusing on stakeholders and required resources.
Table 15. Organizations using big data in smart real estate, focusing on stakeholders and required resources.
Figure 12. Conceptual model for big data utilization in real estate transactions with decentralizec validation to ensure data integrity. Big data can be generated by software and tools owned by agencies and the sellers of properties, which gives personalized suggestions and recommendations to the prospective buyers or users of the service to make better and informed decisions. However, it is important to have a centralized independent validation system in check that can be operated by the government or assessment agencies to protect the privacy of the users, along with verification of the data and information provided to the prospective buyers. In this way, trust can be generated between the key real estate stakeholders, i.e., the sellers and buyers, which can reduce, if not eliminate, the regrets related to ill-informed decisions made by the buyers or users. A conceptual model is presented in Figure 12 for this purpose. As highlighted by Joseph and Varghese [158], there is a risk of big data brokers misleading the consumers and exploiting their interests; therefore, regulators and legislators should begin to develop consumer protection strategies against the strong growth for big data brokers. The model in Figure 12 supports this argument and presents an intermediary organization for keeping an eye on the misuse of data and manipulations by big data agents and brokers.
Figure 12. Conceptual model for big data utilization in real estate transactions with decentralizec validation to ensure data integrity. Big data can be generated by software and tools owned by agencies and the sellers of properties, which gives personalized suggestions and recommendations to the prospective buyers or users of the service to make better and informed decisions. However, it is important to have a centralized independent validation system in check that can be operated by the government or assessment agencies to protect the privacy of the users, along with verification of the data and information provided to the prospective buyers. In this way, trust can be generated between the key real estate stakeholders, i.e., the sellers and buyers, which can reduce, if not eliminate, the regrets related to ill-informed decisions made by the buyers or users. A conceptual model is presented in Figure 12 for this purpose. As highlighted by Joseph and Varghese [158], there is a risk of big data brokers misleading the consumers and exploiting their interests; therefore, regulators and legislators should begin to develop consumer protection strategies against the strong growth for big data brokers. The model in Figure 12 supports this argument and presents an intermediary organization for keeping an eye on the misuse of data and manipulations by big data agents and brokers.
Table 15. Cont.
Table 15. Cont.
Table 16. Big data sources, features, and phases specific to various disaster types.
Table 16. Big data sources, features, and phases specific to various disaster types.
Table 16. Cont. Social media is one of the best resources to gather real-time data at the time of crisis. It is being increasingly used for communication and coordination during emergencies [184]. This calls for a system to be able to effectively manage these data and filter the data related to the needs and requests of the people during the post-disaster period. To be able to provide timely help, the big data generated from the social networks should be mined and analyzed to determine factors like which areas need the most relief services and should be prioritized by the relief workers, and what services are required by the people there [137]. In this section, we propose a framework that extracts the data from various social media networks like Facebook, Twitter, news APIs, and other sources. The extracted data are mostly in the unstructured form and need to undergo cleaning and pre-processing to remove irrelevant and redundant information. This also involves removing URLs, emoticons, symbols, hashtags, and words from a foreign language. After applying these pre-processing steps, the data need to be filtered so that only relevant data are retained. During a post-disaster period, the basic needs of the people are related to food, water, medical aid, and accommodation. Hence, some keywords related to these four categories must be defined, so that only the data related to them are extracted. For example, the terms related to the keyword “food” may be “hunger, starved, eat”. A wide range of terms related to each keyword need to be defined so that maximum data related to them are extracted. It is also crucial to gather these data along with information related to the geographical location, so that location-wise aid could be provided. After gathering these data, the next step will be to train a machine learning model, to predict which area needs emergency services and which facilities are needed by the people over there. Before supplying data for classification, the data must be represented in the form of a feature vector so that they can be interpreted by the algorithm. A unigram-, bigram-, or trigram-based approach can be used for generation of a feature vector from the data. The basic workflow of the system is presented in Figure 13.
Table 16. Cont. Social media is one of the best resources to gather real-time data at the time of crisis. It is being increasingly used for communication and coordination during emergencies [184]. This calls for a system to be able to effectively manage these data and filter the data related to the needs and requests of the people during the post-disaster period. To be able to provide timely help, the big data generated from the social networks should be mined and analyzed to determine factors like which areas need the most relief services and should be prioritized by the relief workers, and what services are required by the people there [137]. In this section, we propose a framework that extracts the data from various social media networks like Facebook, Twitter, news APIs, and other sources. The extracted data are mostly in the unstructured form and need to undergo cleaning and pre-processing to remove irrelevant and redundant information. This also involves removing URLs, emoticons, symbols, hashtags, and words from a foreign language. After applying these pre-processing steps, the data need to be filtered so that only relevant data are retained. During a post-disaster period, the basic needs of the people are related to food, water, medical aid, and accommodation. Hence, some keywords related to these four categories must be defined, so that only the data related to them are extracted. For example, the terms related to the keyword “food” may be “hunger, starved, eat”. A wide range of terms related to each keyword need to be defined so that maximum data related to them are extracted. It is also crucial to gather these data along with information related to the geographical location, so that location-wise aid could be provided. After gathering these data, the next step will be to train a machine learning model, to predict which area needs emergency services and which facilities are needed by the people over there. Before supplying data for classification, the data must be represented in the form of a feature vector so that they can be interpreted by the algorithm. A unigram-, bigram-, or trigram-based approach can be used for generation of a feature vector from the data. The basic workflow of the system is presented in Figure 13.
Figure 13. Proposed framework for utilizing social media big data for emergency and disaster relief.
Figure 13. Proposed framework for utilizing social media big data for emergency and disaster relief.
Figure 14. Big data utilization for disaster risk management
Figure 14. Big data utilization for disaster risk management
Figure 15. Smart real estate big data as an input to disaster management. the data from building integration, maintenance, and facility management constitutes smart real estate big data controlled by the real estate managers. These data, if refined and shared with the disaster managers and response teams by the smart real estate management agencies and managers can help in planning for disaster response. For example, the data related to available facilities at the building can help prepare the occupants for upcoming disasters through proper training and awareness, who can respond to these disasters in an efficient way. Similarly, knowledge of smart building components and the associated building management data can help address the four key areas of disaster risk management: prevent, prepare, respond, and recover. The proposed merging framework is inspired by the works of Grinberger et al. [185], Lv et al. [186], Hashem et al. [187], and Shah et al. [30]. Grinberger et al. [185] used data obtained from smart real estate in terms of occupant data in terms of socioeconomic attributes such as income, age, car ownership, and building data based on value and floor space to investigate the disaster preparedness response for a hypothetical earthquake in downtown Jerusalem. Lv et al. [186] proposed a model for using big data obtained from multimedia usage by real estate users to develop a disaster management plan for service providers such as traffic authorities, fire, and other emergency departments. Hashem et al. [187] proposed an integrated model based on wireless sensing technologies that can integrate various components of smart cities for industrial process monitoring and control, machine health monitoring, natural disaste1 prevention, and water quality monitoring. Similarly, Shah et al. [30] proposed a disaster-resilient smart city concept that integrates IoT and big data technologies and offers a generic solution for disaster risk management activities in smart city incentives. Their framework is based on a combination of the Hadoop Ecosystem and Apache Spark that supports both real-time and offline analysis, and the implementation model consists of data harvesting, data aggregation, data pre-processing, and a big data analytics and service platform. A variety of datasets from smart buildings, city pollution, traffic simulators, and social media such as Twitter are utilized for the validation and evaluation of the system to detect and generate alerts for a fire in a building, pollution level in the city, emergency evacuation path, and the collection of information about natural disasters such as earthquakes and tsunamis Furthermore, Yang et al. [25] proposed real-time feedback loops on nature disasters to help real estate and city decision-makers make real-time updates, along with a precision and dynamic rescue plan that helps in in all four phases of disaster risk management: prevention, mitigation, response, and recovery; this can help the city and real estate planners and managers to take prompt and accurate actions to improve the city’s resilience to disasters.
Figure 15. Smart real estate big data as an input to disaster management. the data from building integration, maintenance, and facility management constitutes smart real estate big data controlled by the real estate managers. These data, if refined and shared with the disaster managers and response teams by the smart real estate management agencies and managers can help in planning for disaster response. For example, the data related to available facilities at the building can help prepare the occupants for upcoming disasters through proper training and awareness, who can respond to these disasters in an efficient way. Similarly, knowledge of smart building components and the associated building management data can help address the four key areas of disaster risk management: prevent, prepare, respond, and recover. The proposed merging framework is inspired by the works of Grinberger et al. [185], Lv et al. [186], Hashem et al. [187], and Shah et al. [30]. Grinberger et al. [185] used data obtained from smart real estate in terms of occupant data in terms of socioeconomic attributes such as income, age, car ownership, and building data based on value and floor space to investigate the disaster preparedness response for a hypothetical earthquake in downtown Jerusalem. Lv et al. [186] proposed a model for using big data obtained from multimedia usage by real estate users to develop a disaster management plan for service providers such as traffic authorities, fire, and other emergency departments. Hashem et al. [187] proposed an integrated model based on wireless sensing technologies that can integrate various components of smart cities for industrial process monitoring and control, machine health monitoring, natural disaste1 prevention, and water quality monitoring. Similarly, Shah et al. [30] proposed a disaster-resilient smart city concept that integrates IoT and big data technologies and offers a generic solution for disaster risk management activities in smart city incentives. Their framework is based on a combination of the Hadoop Ecosystem and Apache Spark that supports both real-time and offline analysis, and the implementation model consists of data harvesting, data aggregation, data pre-processing, and a big data analytics and service platform. A variety of datasets from smart buildings, city pollution, traffic simulators, and social media such as Twitter are utilized for the validation and evaluation of the system to detect and generate alerts for a fire in a building, pollution level in the city, emergency evacuation path, and the collection of information about natural disasters such as earthquakes and tsunamis Furthermore, Yang et al. [25] proposed real-time feedback loops on nature disasters to help real estate and city decision-makers make real-time updates, along with a precision and dynamic rescue plan that helps in in all four phases of disaster risk management: prevention, mitigation, response, and recovery; this can help the city and real estate planners and managers to take prompt and accurate actions to improve the city’s resilience to disasters.
Related topics:
Computer ScienceMachine LearningDisaster risk managementReal estate managementProperty ManagementBig Data StorageBig Data Management
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved