Big data and remote sensing: A new software of ingestion

—Searching info on the web in today's world can be considered as dragging a net across the surface of the earth. While a great deal may be caught in the net, there is still a huge amount of information that is deep, and therefore, missed. The reason is simple: Most of the Web's information is buried down on dynamically produced sites, and standard search engines never find it, where data are hidden behind query interfaces. But a direct query is a "one at a time" laborious way to find info.Several factors contribute to making this problem particularly challenging. The Web is changing at a constant pace – new sources are added, and old sources are removed and modified. The remote wireless senses generate very massive amount real-time data from the Satellite or from the Aircraft with the assistance of the sensors. Technology trends for Big Data accept open source software, commodity servers, and massively parallel-distributed processing platforms. Analytics is at the core of exploiting values from Big Data to produce consumable insights for business and government. This paper presents architecture for Big Data Analytics and explores Big Data technologies offering SQL databases, Hadoop Distributed File System and Map-Reduce. The intended architecture has the aptness of storing incoming unprepared data to dispatch offline analysis on largely stored dumps when required. Concluding, a detailed analysis of remotely sensed earth observatory Big Data for ground or sea level are offered using Hadoop. The proposed architecture possess the ability of dividing, load balancing, and parallel processing of only useful data. Thus, it results in efficient analysis of real-time remote sensing Big Data using earth observatory system.

Enormous data generated by Satellite sensors, Storage and Processing of Remote Sensing Data is a challenging task due to its variety and volume. This paper studied on real-time Big Data Analytical architecture for remote sensing satellite application. To handle Remote Sensing Data proposed architecture comprises three main units, such as Data Pre-Processing Unit (DPREU), Data Analysis Unit (DAU) and Data Post-Processing Unit (DPOSTU). First, DPREU acquires the required data from satellite sensors by using filtration, balanced distributed storage and parallel processing using Hadoop environment. Second, DAU identifies the hidden patterns from data stored in distributed File System using Map functions followed by Reduce functions in Map-Reduce paradigm. Finally, DPOSTU is the upper layer unit of the proposed architecture, which is responsible for compiling storage of the results, and generation of decision based on the results received from DAU.

Journal of Applied Remote Sensing, 2020

Satellite data are used in several environmental applications, particularly in air quality supervising, climate change monitoring, and natural disaster predictions. However, remote sensing (RS) data occur in huge volume, in near-real time, and are stored inside complex structures. We aim to prove that satellite data are big data (BD). Accordingly, we propose a software as an extract-transform-load tool for satellite data preprocessing. We focused on the ingestion layer that will enable an efficient RSBD integration. As a result, the developed software layer receives data continuously and removes ∼86% of the unused files. This layer also eliminates nearly 20% of erroneous datasets. Thanks to the proposed approach, we successfully reduced storage space consumption, enhanced the RS data accuracy, and integrated preprocessed datasets into a Hadoop distributed file system.

BigSkyEarth conference: AstroGeoInformatics, Tenerife, Spain, December 17-19, 2018, 2018

The ever-growing need of data preservation and their systematic analysis contributing to sustainable development of the society spurred in the past decade, numerous Big Data projects and initiatives are focusing on the Earth Observation (EO). The number of Big Data EO applications has grown extremely worldwide almost simultaneously with other scientific and technological areas of the human knowledge due to the revolutionary technological pr ogress in the space and information technology sciences. The substantial contribution to this development are the space programs of the renowned space agencies, such as NASA, ESA, Roskosmos, JAXA, DLR, INPE, ISRO, CNES etc. A snapshot of the current Big Data sets from available satellite missions covering the Bulgarian territory is also presented. This short overview of the geoscience Big Data collection with a focus on EO will emphasize to the multiple Vs of EO in order to provide a snapshot on the current state-of-the-art in EO data preservation and manipulation. Main modern approaches for compressing, clustering and modelling EO in the geoinformation science for Big Data analysis, interpretation and visualization for a variety of applications are outlined. Special attention is paid to the contemporary EO data modelling and visualization systems.

INTERNATIONAL JOURNAL OF COMPUTERS COMMUNICATIONS & CONTROL, 2021

Multi-area and multi-faceted remote sensing (SAR) datasets are widely used due to the increasing demand for accurate and up-to-date information on resources and the environment for regional and global monitoring. In general, the processing of RS data involves a complex multi-step processing sequence that includes several independent processing steps depending on the type of RS application. The processing of RS data for regional disaster and environmental monitoring is recognized as computationally and data demanding.Recently, by combining cloud computing and HPC technology, we propose a method to efficiently solve these problems by searching for a large-scale RS data processing system suitable for various applications. Real-time on-demand service. The ubiquitous, elastic, and high-level transparency of the cloud computing model makes it possible to run massive RS data management and data processing monitoring dynamic environments in any cloud. via the web interface. Hilbert-based da...

Image processing algorithms related to remote sensing have been tested and utilized on the Hadoop MapReduce parallel platform by using an experimental 112-core high-performance cloud computing system that is situated in the Environmental Studies Center at the University of Qatar. Although there has been considerable research utilizing the Hadoop platform for image processing rather than for its original purpose of text processing, it had never been proved that Hadoop can be successfully utilized for high-volume image files. Hence, the successful utilization of Hadoop for image processing has been researched using eight different practical image processing algorithms. We extend the file approach in Hadoop to regard the whole TIFF image file as a unit by expanding the file format that Hadoop uses. Finally, we apply this to other image formats such as the JPEG, BMP, and GIF formats. Experiments have shown that the method is scalable and efficient in processing multiple large images used mostly for remote sensing applications, and the difference between the single PC runtime and the Hadoop runtime is clearly noticeable.

International Journal of Advanced Computer Science and Applications, 2021

With the rapid development of remote sensing technology, our ability to obtain remote sensing data has been improved to an unprecedented level. We have entered an era of big data. Remote sensing data clear showing the characteristics of Big Data such as hyper spectral, high spatial resolution, and high time resolution, thus, resulting in a significant increase in the volume, variety, velocity and veracity of data. This paper proposes a feature supporting, salable, and efficient data cube for time-series analysis application, and used the spatial feature data and remote sensing data for comparative study of the water cover and vegetation change.The spatial-feature remote sensing data cube (SRSDC) is described in this paper. It is a data cube whose goal is to provide a spatial-feature-supported, efficient, and scalable multidimensional data analysis system to handle largescale RS data. It provides a high-level architectural overview of the SRSDC.The SRSDC offers spatial feature repositories for storing and managing vector feature data, as well as feature translation for converting spatial feature information to query operations.The paper describes the design and implementation of a feature data cube and distributed execution engine in the SRSDC. It uses the long time-series remote sensing production process and analysis as examples to evaluate the performance of a feature data cube and distributed execution engine. Big data has become a strategic highland in the knowledge economy as a new strategic resource for humans. The core knowledge discovery methods include supervised learning methods data analysis supervised learning, unsupervised learning methods data analysis unsupervised learning, and their combinations and variants.

2018

In this survey paper, we tend to study land classification from two-dimensional high<br> resolution satellite pictures victimization Hadoop framework. Propelled picture process<br> calculations that need higher system control with monstrous scale inputs is prepared with<br> productivity exploitation the parallel and dispersed procedure of HadoopMapReduce<br> Framework. HadoopMapReduce could be a climbable model that is fit for process petabytes<br> information with enhanced adaptation to non-critical failure and information closeness.<br> During this paper we tend to gift a MapReduce framework for acting parallel remote sensing<br> satellite processing victimization Hadoop and storing the output in HBase. The speed and<br> performance show that by utilizing Hadoop, we are able to distribute our employment across<br> completely different clusters to require advantage of combined process power on goods<br> hardware.

International Journal of Advanced Computer Science and Applications, 2020

Earth data, collected from many sources such as remote sensing imagery, social media, and sensors, are growing tremendously. Among them, satellite imagery which play an important roles for monitoring environment and natural changes are increased exponentially in term of both volume and speed. This paper introduces an approach to managing and analyzing such data sources based on Apache Hadoop and RasterFrames. First, it presents the architecture and the general flow of the proposed distributed framework. Based on this, we can implement and perform efficient computations on a big data in parallel without moving data to the center computer which might lead to network congestion. Finally, the paper presents a case study that analyzes the water surface of a Vietnam region using the proposed platform.

IRJET, 2020

Weather forecasts are made by collecting as much data as possible about the current state of the atmosphere to determine how the atmosphere evolves in the future. To handle such humongous data-"Big Data" is introduced. Big Data has become an imminent part of all industries and business sectors today. we propose a Pre-Processing Framework to address quality of data in weather monitoring. Hence, it is imperative to improve Data quality even it is absorbed and utilized in an industry's Big Data system. In this paper, we propose a Pre-Processing Framework to address quality of data in a weather monitoring and forecasting application that also takes into account global warming parameters and raises alerts/notifications to warn users and scientists in advance. MOTIVATION-We have conceptualized a Weather Monitoring and Forecasting Application to raises alerts/notifications to warn users and scientists in advance.