analysis ready data

This paper proposes a general framework for symbolic reasoning rooted in the construction, preservation, and communication of meaning across layered systems. Drawing from work in theology, linguistics, and logic, it formalizes a two-mode interpretive model (strict and communicative) supported by definitional discipline, diagnostic self-correction, and layered structure. The methodology has broad implications for epistemology, artificial intelligence, language theory, and interpretive philosophy.

The EU-funded DIONE project (grant agreement No. 870378) offers an innovative close-to-market (TRL7) solution seeking to improve the traditional methods of agricultural monitoring. The project introduces a cloud-based Software as a Service (SaaS) system architecture, building on a fusion of novel technologies that will support the forthcoming needs of the modernized Common Agriculture Policy (CAP) and the "Greening" perspectives, with an automated area-based monitoring system. In particular, an interoperable and harmonized system is designed, connecting large volumes of Earth Observation data (Satellite, UAV, and in-situ) and user-generated highly precise geolocated data (geo-tagged photos, soil measurements, etc.). DIONE's system architecture encompasses customized and third-party frameworks, where heterogeneous and multi-source data are stored, processed and managed using Artificial Intelligence (AI) algorithms. These harmonized, curated and open accessed data are then provided as Open Geospatial Consortium (OGC)-compliant, web-service layers (WMS, WFS, and WCS). Furthermore, the proposed solution formulates a scalable, flexible, interoperable, and semantically enriched environment, taking advantage of a Spatial Data Infrastructure (SDI) framework capabilities, whilst allowing an interactive connection among different tools and components through RESTful APIs. Our approach establishes a novel, cloud-based, accurate and inexpensive agriculture monitoring solution, enabling the real-time provision of multi-source data to relevant stakeholders such as Paying Agencies, Policy Officers and Control & Certification Bodies, and other domain experts. The system architecture was formulated exploiting a codesign methodology, aiming to ensure a long-term and sustainable solution. Two large scale demonstration will take place in Lithuania and Cyprus, evaluating the system capabilities in real-life and operational conditions.

Earth observations data cubes (EODCs) are a paradigm transforming the way users interact with large spatio-temporal Earth observation (EO) data. It enhances connections between data, applications and users facilitating management, access and use of analysis ready data (ARD). The ambition is allowing users to harness big EO data at a minimum cost and effort. This significant interest is illustrated by various implementations that exist. The novelty of the approach results in different innovative solutions and the lack of commonly agreed definition of EODC. Consequently, their interoperability has been recognized as a major challenge for the global change and Earth system science domains. The objective of this paper is preventing EODC from becoming silos of information; to present how interoperability can be enabled using widely-adopted geospatial standards; and to contribute to the debate of enhanced interoperability of EODC. We demonstrate how standards can be used, profiled and enr...

We are at the dawn of a new era, where advances in computer power, broadband communication and digital sensor technologies have led to an unprecedented flood of data inundating our surrounding. It is generally believed that means such as Computational Intelligence will help to survive these tough times. However, these hopes are improperly high. Computational Intelligence is a surprising composition of two mutually exclusive and contradicting constituents that could be coupled only if you disregard and neglect their controversies: "Computational" implies reliance on data processing and "Intelligence" implies reliance on information processing. Only those who are indifferent to data-information discrepancy can believe that such a combination can be viable. We do not believe in miracles, so we will try to share with you our reservations.

Earth Observation Data Cubes (EODC) have emerged as a promising solution to efficiently and effectively handle Big Earth Observation (EO) Data generated by satellites and made freely and openly available from different data repositories. The aim of this Special Issue, “Earth Observation Data Cube”, in Data, is to present the latest advances in EODC development and implementation, including innovative approaches for the exploitation of satellite EO data using multi-dimensional (e.g., spatial, temporal, spectral) approaches. This Special Issue contains 14 articles covering a wide range of topics such as Synthetic Aperture Radar (SAR), Analysis Ready Data (ARD), interoperability, thematic applications (e.g., land cover, snow cover mapping), capacity development, semantics, processing techniques, as well as national implementations and best practices. These papers made significant contributions to the advancement of a more Open and Reproducible Earth Observation Science, reducing the ga...

This paper introduces a new capability for performing the vicarious radiometric calibration of high, medium and low spatial resolution sensors. The SPecular Array Radiometric Calibration (SPARC) method employs convex mirrors to create calibration targets for deriving absolute calibration coefficients of Earth remote sensing systems in the solar reflective spectrum. 17 The combination of these mirror arrays with a targeting station is the basis for a new, ondemand commercial calibration network called FLARE. SPARC METHOD-AN INNOVATION IN CALIBRATION Current best practice for verifying absolute radiometric requirements of remote sensing imaging systems is always based on viewing a traceable extended source that uniformly illuminates many if not all detectors on the focal plane. This holds true whether the calibration images are recorded in pre-flight testing or in-flight viewing on-board or vicarious sources. Illuminating a large number of detectors establishes a mean response to the calibration radiance with high precision and averages out the blurring effects induced by the sensor's optical system. The procedure derives radiometric gain coefficients with uncertainties that can be maintained through the image processing chain to higher level products but only for extended uniform scenes for which the response is unaffected by the sensor's image quality performance.

With a rapidly growing small satellite industry deploying constellations of low cost imaging sensors for academic, commercial, civil, and intelligence applications, the quality and exploitability of data products will play a critical role in maintaining that growth. As a result, absolute intersensor calibration is critical in providing reproducible and accurate measurements over time for delivering image data that is affordable, maximizes information content, and optimizes value added potential. A significant part of keeping the cost down has typically been the elimination of onboard calibration and relying on vicarious calibration methods. Robust techniques are required to ensure that observations from different instruments establish a reliable form of traceability and can be normalized to a common absolute radiometric scale for achieving seamless physics based exploitation. Outlined in this paper is a versatile and effective method for achieving calibration equalization and performance characterization of a constellation of imaging sensors in the solar reflective spectrum. The Specular Array Calibration (SPARC) method, is an adaptable ground based system that uses convex mirrors to create small reference targets revealing radiometric, spatial, spectral, geometric and temporal characteristics of individual sensors for transforming them into a unified earth-monitoring system. Overall, the wide range of SPARC capabilities provides the potential to minimize pre-flight requirements (reduce cost and schedule) by being able to shift more calibration and validation analysis on orbit, completed quickly after launch and continued routinely through the life of each imaging system and duration of the constellation.

The H2020 DeepCube project leverages advances in the fields of Artificial Intelligence and Semantic Web to unlock the potential of Copernicus Big Data and contribute to the Digital Twin Earth initiative. DeepCube aims to address problems of high socio-environmental impact and enhance our understanding of Earth’s processes correlated with Climate Change. To achieve this, the project employs novel technologies, such as the Earth System Data Cube, the Semantic Cube, the Hopsworks platform for distributed deep learning, and visual analytics tools, integrating them into an open, cloud-interoperable platform. DeepCube will develop Deep Learning architectures that extend to non-conventional data, apply hybrid modeling for data-driven AI models that respect physical laws, and open up the Deep Learning black box with Explainable Artificial Intelligence and Causality.

The objective of the Ground to Space CALibration Experiment (G-SCALE) is to demonstrate the use of convex mirrors as a radiometric and spatial calibration and validation technology for Earth Observation assets, operating at multiple altitudes and spatial scales. Specifically, point sources with NIST-traceable absolute radiance signal are evaluated for simultaneous vicarious calibration of multi- and hyperspectral sensors in the VNIR/SWIR range, aboard Unmanned Aerial Vehicles (UAVs), manned aircraft, and satellite platforms. We introduce the experimental process, field site, instrumentation, and preliminary results of the G-SCALE, providing context for forthcoming papers that will detail the results of intercomparison between sensor technologies and remote sensing applications utilizing the mirror-based calibration approach, which is scalable across a wide range of pixel sizes with appropriate facilities. The experiment was carried out at the Rochester Institute of Technology’s Tait...

Pressures on natural resources are increasing and a number of challenges need to be overcome to meet the needs of a growing population in a period of environmental variability. Some of these environmental issues can be monitored using remotely sensed Earth Observations (EO) data that are increasingly available from a number of freely and openly accessible repositories. However, the full information potential of EO data has not been yet realized. They remain still underutilized mainly because of their complexity, increasing volume, and the lack of efficient processing capabilities. EO Data Cubes (DC) are a new paradigm aiming to realize the full potential of EO data by lowering the barriers caused by these Big data challenges and providing access to large spatio-temporal data in an analysis ready form. Systematic and regular provision of Analysis Ready Data (ARD) will significantly reduce the burden on EO data users. Nevertheless, ARD are not commonly produced by data providers and therefore getting uniform and consistent ARD remains a challenging task. This paper presents an approach to enable rapid data access and pre-processing to generate ARD using interoperable services chains. The approach has been tested and validated generating Landsat ARD while building the Swiss Data Cube.

Aiming at the convergence between Earth observation (EO) Big Data
and Artificial General Intelligence (AGI), this paper consists of two
parts. In the previous Part 1, existing EO optical sensory imagederived
Level 2/Analysis Ready Data (ARD) products and processes
are critically compared, to overcome their lack of harmonization/
standardization/ interoperability and suitability in a new notion of
Space Economy 4.0. In the present Part 2, original contributions
comprise, at the Marr five levels of system understanding: (1) an
innovative, but realistic EO optical sensory image-derived semantics-
enriched ARD co-product pair requirements specification. First,
in the pursuit of third-level semantic/ontological interoperability, a
novel ARD symbolic (categorical and semantic) co-product, known
as Scene Classification Map (SCM), adopts an augmented Cloud versus
Not-Cloud taxonomy, whose Not-Cloud class legend complies with
the standard fully-nested Land Cover Classification System’s
Dichotomous Phase taxonomy proposed by the United Nations
Food and Agriculture Organization. Second, a novel ARD subsymbolic
numerical co-product, specifically, a panchromatic or multispectral
EO image whose dimensionless digital numbers are radiometrically
calibrated into a physical unit of radiometric measure,
ranging from top-of-atmosphere reflectance to surface reflectance
and surface albedo values, in a five-stage radiometric correction
sequence. (2) An original ARD process requirements specification.
(3) An innovative ARD processing system design (architecture),
where stepwise SCM generation and stepwise SCM-conditional EO
optical image radiometric correction are alternated in sequence. (4)
An original modular hierarchical hybrid (combined deductive and
inductive) computer vision subsystem design, provided with feedback
loops, where software solutions at the Marr two shallowest
levels of system understanding, specifically, algorithm and implementation, are selected from the scientific literature, to benefit from their technology readiness level as proof of feasibility, required in addition to proven suitability. To be implemented in operational mode at the space segment and/or midstream segment by both public and private EO big data providers, the proposed EO optical sensory image-derived semantics-enriched ARD product-pair and process reference standard is highlighted as linchpin for success of a new notion of Space Economy 4.0.

Aiming at the convergence between Earth observation (EO) Big
Data and Artificial General Intelligence (AGI), this two-part paper
identifies an innovative, but realistic EO optical sensory imagederived
semantics-enriched Analysis Ready Data (ARD) productpair
and process gold standard as linchpin for success of a new
notion of Space Economy 4.0. To be implemented in operational
mode at the space segment and/or midstream segment by both
public and private EO big data providers, it is regarded as necessarybut-
not-sufficient “horizontal” (enabling) precondition for: (I)
Transforming existing EO big raster-based data cubes at the midstream
segment, typically affected by the so-called data-rich information-
poor syndrome, into a new generation of semanticsenabled
EO big raster-based numerical data and vector-based categorical
(symbolic, semi-symbolic or subsymbolic) information cube
management systems, eligible for semantic content-based image
retrieval and semantics-enabled information/knowledge discovery.
(II) Boosting the downstream segment in the development of an
ever-increasing ensemble of “vertical” (deep and narrow, user-specific
and domain-dependent) value–adding information products
and services, suitable for a potentially huge worldwide market of
institutional and private end-users of space technology. For the
sake of readability, this paper consists of two parts. In the present
Part 1, first, background notions in the remote sensing metascience
domain are critically revised for harmonization across the multidisciplinary
domain of cognitive science. In short, keyword “information”
is disambiguated into the two complementary notions of
quantitative/unequivocal information-as-thing and qualitative/
equivocal/inherently ill-posed information-as-data-interpretation. Moreover, buzzword “artificial intelligence” is disambiguated into
the two better-constrained notions of Artificial Narrow Intelligence as part-without-inheritance-of AGI. Second, based on a better- defined and better-understood vocabulary of multidisciplinary terms, existing EO optical sensory image-derived Level 2/ARD products and processes are investigated at the Marr five levels of understanding of an information processing system. To overcome their drawbacks, an innovative, but realistic EO optical sensory image-derived semantics-enriched ARD product-pair and process gold standard is proposed in the subsequent Part 2.

Calibration and characterization of radiometric and geospatial performance is necessary for the accurate retrieval of information from Earth Observation platforms. For small satellite constellations, this is especially true to ensure data quality and consistency among multiple craft that lack on-board calibration equipment. The Field Line-of-sight Automate Radiance Exposure (FLARE) Network provides an automated, on-demand calibration solution designed to meet the requirements of both agency and commercial operators by providing NIST traceable data without the need for expensive campaigns o r on-board calibration sources. FLARE radiometric performance has been verified with Landsat 8 – OLI. FLARE was successfully utilized with Planet SkySat and PlanetScope assets for rapid verification of resolution performance, and in commissioning efforts of new satellites following launch

Environmental issues become an increasing global concern because of the continuous pressure on natural resources. Earth observations (EO), which include both satellite/UAV and in-situ data, can provide robust monitoring for various environmental concerns. The realization of the full information potential of EO data requires innovative tools to minimize the time and scientific knowledge needed to access, prepare and analyze a large volume of data. EO Data Cube (DC) is a new paradigm aiming to realize it. The article presents the Swiss-Armenian joint initiative on the deployment of an Armenian DC, which is anchored on the best practices of the Swiss model. The Armenian DC is a complete and up-to-date archive of EO data (e.g., Landsat 5, 7, 8, Sentinel-2) by benefiting from Switzerland’s expertise in implementing the Swiss DC. The use-case of confirm delineation of Lake Sevan using McFeeters band ratio algorithm is discussed. The validation shows that the results are sufficiently relia...

This paper introduces a new capability for performing the vicarious radiometric calibration of high, medium and low spatial resolution sensors. The SPecular Array Radiometric Calibration (SPARC) method employs convex mirrors to create calibration targets for deriving absolute calibration coefficients of Earth remote sensing systems in the solar reflective spectrum. 17 The combination of these mirror arrays with a targeting station is the basis for a new, ondemand commercial calibration network called FLARE. SPARC METHOD-AN INNOVATION IN CALIBRATION Current best practice for verifying absolute radiometric requirements of remote sensing imaging systems is always based on viewing a traceable extended source that uniformly illuminates many if not all detectors on the focal plane. This holds true whether the calibration images are recorded in pre-flight testing or in-flight viewing on-board or vicarious sources. Illuminating a large number of detectors establishes a mean response to the calibration radiance with high precision and averages out the blurring effects induced by the sensor's optical system. The procedure derives radiometric gain coefficients with uncertainties that can be maintained through the image processing chain to higher level products but only for extended uniform scenes for which the response is unaffected by the sensor's image quality performance.

Collections of free and open-access Earth observation (EO) data with global coverage are growing with increasingly higher spatial resolutions and temporal frequency. They are one of the few globally consistent data sources available for generating information in support of international initiatives. However, these data require automated workflows for handling, processing and analysis, including methods to convert data into valid information. A proof-of-concept implementation of a semantic EO data cube is presented using Open Data Cube technology to generate ad-hoc reproducible, scalable, repeatable and spatiallyexplicit information as indicators to support monitoring international environmental initiatives. Sentinel-2 data for the study area in north-western Syria (30,000km2) are incorporated daily, including automatically generated generic semantic enrichment. As of December 2018, this encompasses over 800 scenes (~480GB unprocessed Sentinel-2 data). A semantic query resulting in a...

Abstract length < 1500 words. The dictum “small is the new big”, synonym of “when bigger isn't better”, increasingly applies to the traditional Earth observation (EO) industry whose business model is under pressure by an emerging EO small-scale (< 500 Kg in weight) satellite constellation technology, aiming at sub-meter spatial resolution and sub-daily temporal resolution in a “seamless innovation chain” required by Space 4.0, whose ambitious objectives range from the business applications market to grand societal challenges, e.g., the United Nations Development Programme (UNDP) Sustainable Developments Goals (SDGs), such as migration, food security, climate change, terrorism, urbanisation and poverty. In spite of an increasing interest on EO small satellite constellations, according to some EO experts “the small satellite market is a failed business model in the short term at least. It is hype. It is here to hit established players (just like Uber – an excellent player with an innovative business model, to shake up the existing setup), but making huge losses in the process. It is quite obvious that we have already reached a point with the EO industry that the amount of EO data collected is way beyond than the need. So, the next time you get excited about 80 odd satellites launched by someone, just question, when and how will they make money?” To further investigate pros and cons of existing EO small satellite constellations, our analytic and pragmatic analysis starts from first principles proposed by the intergovernmental Group on Earth Observations (GEO), whose visionary goal of a Global Earth Observation System of Systems (GEOSS) implementation plan for years 2005-2015, unaccomplished to date, is systematic transformation of multi-sensor, multi-angular and multi-temporal EO big data cubes into timely, comprehensive and operational EO value-adding information products and services (VAPS). According to the joint GEO-Committee on Earth Observation Satellites (CEOS) Quality Accuracy Framework for EO (QA4EO) Calibration/Validation (Cal/Val) guidelines, necessary-but-not-sufficient pre-conditions for GEOSS “to allow the access to the Right Information, in the Right Format, at the Right Time, to the Right People, to Make the Right Decisions” are the four key principles of Availability/Accessibility and Suitability/Reliability applied to data and information products. Related to the four GEO-CEOS key principles are the foundational principles of Findability, Accessibility, Interoperability, and Reusability (FAIR) applied to data, products and processes to enhance the ability of machines to automatically find and use the data, in addition to supporting its reuse by individuals. Indeed, existing EO small satellite constellations, such as the PlanetScope constellation of 130+ Dove nano-satellites (standardized 3U CubeSat technology, 5 kg in weight, 3 m multi-spectral (MS) spatial resolution, time resolution > 10X Daily), and Planet’s SkySat constellation of 13 micro-satellites (125 kg in weight, 1 m MS spatial resolution, time resolution 2X Daily), guarantee augmented Suitability (in terms of very high spatial and temporal resolutions) and enhanced Availability/Accessibility (expected to be near real-time). On the other hand, the key principle of Reliability/Interoperability requires the remote sensing (RS) community to enforce mandatory GEO-CEOS QA4EO Cal/Val principles within and across small satellite constellations. Radiometric Cal is the transformation of dimensionless digital numbers (DNs) into a radiometric unit of measure to be community-agreed upon at increasing levels of information quality, specifically, top-of-atmosphere (TOA) radiance (TOARD) in range [0, infty), TOA reflectance (TOARF) in range [0, 1], surface reflectance (SURF) in range [0, 1], corrected for atmospheric topographic and/or adjacency effects, and surface albedo in range [0, 1], corrected for bidirectional reflectance distribution function (BRDF) effects. On theory, EO data Cal is a well-known “prerequisite for physical model-based analysis of airborne and satellite sensor measurements in the optical domain”. For example, EO data Cal is considered mandatory by the GEO-CEOS QA4EO Cal/Val guidelines. It guarantees interoperability (harmonization) of EO data acquired across time, space and sensors, in agreement with the FAIR criteria. On a daily basis, all sensory data we deal with are provided with a community-agreed physical unit of measure , e.g., hour, meter, kg, etc. Whereas physical variables can be investigated by physical model-based, statistical model-based or hybrid (combined deductive/ top-down/ physical model-based and inductive/ bottom-up/ statistical model-based) inference systems, uncalibrated sensory data provided with no physical meaning can be investigated by statistical data models exclusively. Although statistical models do not require physical variables as input, they can benefit from data Cal in terms of augmented robustness to changes in the input data set acquired across time, space and sensors. Irrespective of these unquestionable true-facts, the GEO-CEOS QA4EO Cal requirement remains largely neglected in the RS common practice. For example, in the large majority of papers published in the RS literature the word “radiometric calibration” is absent, which means these works cope with uncalibrated EO data provided with no physical unit of radiometric measure. One consequence is that, to date, statistical model-based and inductive learning-from-data EO image understanding (EO-IU) systems dominate the RS literature as well as commercial EO image processing software toolboxes, consisting of overly complicated collections of inductive learning-from-data algorithms to choose from based on heuristics. In EO small-scale (< 500 Kg in weight) satellite constellations, the lack of an on-board radiometric Cal sub-system, to be compensated by vicarious Cal strategies, is expected to have an impact on intra- and inter-constellation Reliability/Interoperability requirements. For example, a Planet Surface Reflectance (SR) Product is systematically generated by Planet from the PlanetScope constellation. Based on the Planet SR product documentation, the PlanetScope radiometric Cal metadata file misses a per-band offset (bias) Cal coefficient. As a consequence, TOARD, TOARF and SURF values estimated in sequence are affected by an unknown bias (additive error term), either positive, null or negative. Consisting of SURF values corrected for atmospheric effects, the PlanetScope SR Product is expected to ensure consistency across localized atmospheric conditions, minimizing uncertainty in spectral response across time and location. No Planet SR product is available for the Planet SkySat micro-satellite constellation. According to the Planet SkySat Imagery Product specification, SkySat DNs are radiometrically calibrated into absolute (at-sensor, TOA) radiance (TOARD) units, derived using vicarious Cal methods. This means that, first, PlanetScope and SkySat radiometrically calibrated products are not comparable in physical terms. Second, the PlanetScope SR Product is not comparable in physical terms and “difficult” to be compared (e.g., fused) in statistical terms with other EO data-derived products radiometrically calibrated into SURF values corrected for atmospheric, topographic and adjacency effects, such as the Sentinel-2 Correction Prototype Processor (Sen2Cor), developed and distributed free-of-cost by the European Space Agency (ESA) to be run on user side. In compliance with the GEO-CEOS QA4EO Cal/Val guidelines, the goal of the present work is to validate by independent means the absolute radiometric quality (in comparison with a “ground truth” or reference baseline) and robustness to changes in input data, acquired across time, space and sensors within and across constellations, of the PlanetScope SR Product, consisting of SURF values corrected for atmospheric effects for augmented Reliability/Interoperability. To validate the radiometric Cal of the PlanetScope SR Product, an off-the-shelf Satellite Image Automatic Mapper™ (SIAM™) lightweight computer program was adopted.

Data cubes seem to be a promising methodological development to deal with the big EO data challenge. The Sentinel-2 Semantic Data Cube Austria (Sen2Cube.at) goes beyond the current state-of-the-art and aims to build an Austrian data & information cube. The main goal is to show
that semantic content-based image and information retrieval is possible in big EO image databases, allowing users to query and analyse EO data on a higher semantic level (i.e. based on at least basic land cover units and encoded ontologies). This includes: (1) fully automatic semantic enrichment of Sentinel-2 images up to land cover types ready for semantic content-based analysis; (2) the use of suitable database technologies to develop spatio-temporal modelling and querying techniques using encoded ontologies to decrease the complexity of queries for user interaction; (3) a Web interface for human-like queries based on semantic models of the spatio-temporal 4D physical-world domain; and (4) the demonstration of the potential of the generic data &
information cube in future service developments based on different service types.