Enabling Real-Time Querying of Live and Historical Stream Data

This paper describes our experiences building a coherent framework for ef- ficient simultaneous querying of live and archived stream data. This work was motivated by the need to analyze the network traffic patterns of research labora- tories funded by the U.S. Department of Energy. We review the requirements of such a system and implement a prototype based on the TelegraphCQ streaming query processor and the FastBit bitmap index. The combined system uses Tele- graphCQ to analyze streams of traffic information and FastBit to correlate current behaviors with historical trends. We present a detailed performance analysis of our system based on a complex query workload and real network traffic collected at Lawrence Berkeley National Laboratory (Berkeley Lab). Our experiments identify key performance bottlenecks for stream query processing systems that incorporate historical data. We also identify strategies for mitigating these bottlenecks. With these strategies in place, we demonstrate ...

With the proliferation of network applications and user groups, the network stream data scale are getting larger and larger. To manage and analyze the massive data to get better service for end users (a.k.a. network optimization) or understanding of impacts of new network applications are attracting many researchers. This paper focuses on the storage and real-time retrieval of network traffic for analysis. We address this problem by integrating Time Machine and FastBit as a real-time querying and storage of massive stream data system, named TIFA. TIFA is designed for operation in Gbps network, accompanied with UTM as a module or an independent device. It leverages multiple large volume storage, and can achieve Ten TB level in volume. It has been proved useful in the performance of querying and storage of massive stream data in the experiments.

2014 International Wireless Communications and Mobile Computing Conference (IWCMC), 2014

Network traffic monitoring systems generate high volumes of heterogeneous data streams which have to be processed and analyzed with different time constraints for daily network management operations. Some monitoring applications such as anomaly detection, performance tracking and alerting require fast processing of specific incoming real-time data. Other applications like fault diagnosis and trend analysis need to process historical data and perform deep analysis on generally heterogeneous sources of data. The Data Stream Warehousing (DSW) paradigm provides the means to handle both types of monitoring applications within a single system, providing fast and rich data analysis capabilities as well as data persistence. In this paper, we introduce DBStream, a novel online traffic monitoring system based on the DSW paradigm, which allows fast and flexible analysis across multiple heterogeneous data sources. DBStream provides a novel stream processing language for implementing data processing modules, as well as aggregation, filtering, and storage capabilities for further data analysis. We show multiple traffic monitoring applications running on DBStream, processing real traffic from operational ISPs.

The 21st International Conference on Computer Communications and Networks (ICCCN 2012)

Archiving Internet traffic is an essential function for retrospective network event analysis. The state-of-the-art approach for network monitoring and analysis is to store and analyze the statistics of network flows. However this approach loses much valuable information inside Internet traffic. With the advancement of commodity hardware, especially the volume of storage device and the speed of interconnect technologies used in network adapter card, now it is practical to capture 10Gbps real-time network traffic with a commodity computer. In this context, this paper presents the design and implementation of a novel system for archiving and querying network flows. This paper propose a bitmap indexing and storage mechanism with flow granularity, and the bitmap index database is utilized to store index information. Based on real network traces, we demonstrate that the system has a higher performance in storing and querying with respect to both time and space metrics compared with traditional methods.

Many traffic analysis tasks are solved with tools that are developed in an ad-hoc, incremental, and cumbersome way instead of seeking systematic solutions that are easy to reuse and understand. The huge amount of data that has to be managed and analyzed together with the fact that many different analysis tasks are performed over a small set of different network trace formats, motivates us to study whether Data Stream Management Systems (DSMSs) might be useful to develop traffic analysis tools. We have performed an experimental study to analyze the advantages and limitations of using DSMS in practice. We study how simple and complex analysis tasks can be solved with TelegraphCQ, a public domain DSMS, and present a preliminary performance analysis.