Valmar: High-bandwidth real-time streaming data management

Studies in Computational Intelligence, 2013

New approaches for data provenance and data management (DPDM) are required for mega science projects like the Square Kilometer Array, characterized by extremely large data volume and intense data rates, therefore demanding innovative and highly efficient computational paradigms. In this context, we explore a streamcomputing approach with the emphasis on the use of accelerators. In particular, we make use of a new generation of high performance stream-based parallelization middleware known as InfoSphere Streams. Its viability for managing and ensuring interoperability and integrity of signal processing data pipelines is demonstrated in radio astronomy.

SpringerBriefs in Computer Science, 2019

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

2007

Abstract Network monitoring systems that support data archiving and after-the-fact (retrospective) queries are useful for a multitude of purposes, such as anomaly detection and network and security forensics. Data archiving for such systems, however, is complicated by (a) data arrival rates, which may be hundreds of thousands of packets per second on a single link, and (b) the need for online indexing of this data to support retrospective queries.

International Journal on Digital Libraries, 1997

We describe a distributed, wide area network based approach to collecting, cataloguing, storing, and providing Web access for large-data-objects that originate as high-speed data streams. Such data streams result from the operation of many types of on-line instruments and imaging systems, and are a "staple" of modern intelligence, scientific, and health care environments. The approach provides for real-time conversion of the data streams and large datasets to "objects" that are manageable, extensible, searchable, browsable, persistent, and available to both "ordinary" and high performance applications through the integration of a high-speed distributed cache and transparent management of tertiary storage. The user interfaces -for both application users and data collection curatorsare provided by the capabilities of the World Wide Web.

2017

Big Data applications are rapidly moving from a batch-oriented execution to a real-time model in order to extract value from the streams of data just as fast as they arrive. Such stream-based applications need to immediately ingest and analyze data and in many use cases combine live (i.e., real-time streams) and archived data in order to extract better insights. Current streaming architectures are designed with distinct components for ingestion (e.g., Kafka) and storage (e.g., HDFS) of stream data. Unfortunately, this separation is becoming an overhead especially when data needs to be archived for later analysis (i.e., near real-time): in such use cases, stream data has to be written twice to disk and may pass twice over high latency networks. Moreover, current ingestion mechanisms offer no support for searching the acquired streams in real time, an important requirement to promptly react to fast data. In this paper we describe the design of Kera: a unified storage and ingestion arc...

2007

Sensors capable of sensing phenomena at high data rates-on the order of tens to hundreds of thousands of samples per second-are useful in many industrial, civil engineering, scientific, networking, and medical applications. In these applications, high-rate streams of data produced by sensors must be processed and analyzed using a combination of both event-stream and signal-processing operations. This paper motivates the need for a data management and continuous query processing architecture that integrates these two different desired classes of functions into a single, unified software system. The key goals of such a system include: the ability to treat a sequence of samples that constitute a "signal segment" as a basic data type; ease of writing arbitrary event-stream and signal-processing functions; the ability to process several million samples per second on conventional PC hardware; and the ability to distribute application code across both PCs and sensor nodes.

19th International Conference on Scientific and Statistical Database Management (SSDBM 2007), 2007

Applications that query data streams in order to identify trends, patterns, or anomalies can often benefit from comparing the live stream data with archived historical stream data. However, searching this historical data in real time has been considered so far to be prohibitively expensive. One of the main bottlenecks is the update costs of the indices over the archived data. In this paper, we address this problem by using our highly-efficient bitmap indexing technology (called FastBit) and demonstrate that the index update operations are sufficiently efficient for this bottleneck to be removed. We describe our prototype system based on the TelegraphCQ streaming query processor and the FastBit bitmap index. We present a detailed performance evaluation of our system using a complex query workload for analyzing real network traffic data. The combined system uses TelegraphCQ to analyze streams of traffic information and FastBit to correlate current behaviors with historical trends. We demonstrate that our system can simultaneously analyze (1) live streams with high data rates and (2) a large repository of historical stream data.

Research in Astronomy and Astrophysics, 2021

Data archiving is one of the most critical issues for modern astronomical observations. With the development of a new generation of radio telescopes, the transfer and archiving of massive remote data have become urgent problems to be solved. Herein, we present a practical and robust file-level flow-control approach, called the Unlimited Sliding-Window (USW), by referring to the classic flow-control method in the TCP protocol. Based on the USW and the Next Generation Archive System (NGAS) developed for the Murchison Widefield Array telescope, we further implemented an enhanced archive system (ENGAS) using ZeroMQ middleware. The ENGAS substantially improves the transfer performance and ensures the integrity of transferred files. In the tests, the ENGAS is approximately three to twelve times faster than the NGAS and can fully utilize the bandwidth of network links. Thus, for archiving radio observation data, the ENGAS reduces the communication time, improves the bandwidth utilization, ...

arXiv (Cornell University), 2017

Sources of multidimensional data are becoming more prevalent, partly due to the rise of the Internet of Things (IoT), and with that the need to ingest and analyze data streams at rates higher than before. Some industrial IoT applications require ingesting millions of records per second, while processing queries on recently ingested and historical data. Unfortunately, existing database systems suited to multidimensional data exhibit low per-node ingestion performance, and even if they can scale horizontally in distributed settings, they require large number of nodes to meet such ingest demands. For this reason, in this paper we evaluate a singlenode multidimensional data store for high-velocity sensor data. Its design centers around a two-level indexing structure, wherein the global index is an in-memory R*-tree and the local indices are serialized kd-trees. This study is confined to records with numerical indexing fields and range queries, and covers ingest throughput, query response time, and storage footprint. We show that the adopted design streamlines data ingestion and offers ingress rates two orders of magnitude higher than those of a selection of open-source database systems, namely Percona Server, SQLite, and Druid. Our prototype also reports query response times comparable to or better than those of Percona Server and Druid, and compares favorably in terms of storage footprint. In addition, we evaluate a kd-tree partitioning based scheme for grouping incoming streamed data records. Compared to a random scheme, this scheme produces less overlap between groups of streamed records, but contrary to what we expected, such reduced overlap does not translate into better query performance. By contrast, the local indices prove much more beneficial to query performance. We believe the experience reported in this paper is valuable to practitioners and researchers alike interested in building database systems for high-velocity multidimensional data.

Sigspatial Special, 2015

Today's observation streams of large numbers of geographically distributed sensors in geosensor networks arrive continuously at powerful servers, and users are interested in analyzing the sensor data streams in real-time. This article investigates challenges in data management that arise when massive real-time data streams become available and discusses different data management technologies for managing and analyzing real-time sensor data streams.