Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Borealis System Overview
Data Model
Dynamic Revision of Query Results
Overview
Related Work
Discussion and Future Plans
References
All Topics
Computer Science
Distributed Systems

The Design of the Borealis Stream Processing Engine

Profile image of Ahmad HarisAhmad Haris

2004

Abstract

Borealis is a second-generation distributed stream processing engine that is being developed at Brandeis University, Brown University, and MIT. Borealis inherits core stream processing functionality from Aurora and distribution functionality from Medusa . Borealis modifies and extends both systems in non-trivial and critical ways to provide advanced capabilities that are commonly required by newly-emerging stream processing applications.

Related papers

A Scalable Architecture for Heterogeneous Sensor Management
Claudia Roncancio

16th International Workshop on Database and Expert Systems Applications (DEXA'05), 2005

Thanks to their miniature size and high autonomy, sensors have become an integral part of our life. They are being successfully used in many fields (military, house automation, medical, urban environments, etc.). However, a lot of work still needs to be done on querying and system management of large sets of sensors. Scalable solutions supporting continuous data stream management are required. This paper presents a scalable architecture for management of data streams coming from a large set of heterogeneous sensors. Our proposal is at the boundaries of two traditional approaches. The first approach uses centralized or slightly distributed platforms to manage continuous data streams, while the second one allows data processing in a strongly distributed way and is based on networks of "intelligent" sensors. Our hybrid approach proposed in this article offers scalable solutions combining the advantages of these two approaches.

View PDFchevron_right
Resilient Sensor Network Query Processing
Alan Stokes

2014

Sensor networks comprise of a collection of resource-constrained, low cost, sometimes fragile wireless motes which have the capability to gather information about their surroundings through the use of sensors, and can be conceived as a distributed computing platform for applications ranging from event detection to environmental monitoring. A Sensor Network Query Processor (SNQP) is a means of collecting data from sensor networks where the requirements are defined using a declarative query language with a set of Quality of Service (QoS) expectations. As sensor networks are often deployed in hostile environments, there is a high possibility that the motes could break or that the communication links between the motes become unreliable. SNQP Query Execution Plans (QEPs) are often optimised for a specific network deployment and are designed to be as energy efficient as possible whilst ensuring the QEPs meet the QoS expectations, yet little has been done for handling the situation where t...

View PDFchevron_right
Adapting to Node Failure In Sensor Network Query Processing
Alan Stokes

The typical nodes used in mote-level wireless sensor networks (WSNs) are often brittle and severely resource-constrained. In particular, nodes are often battery-powered, thereby making energy depletion a significant risk. When changes to the connectivity graph occur as a result of node failure, the overall computation may collapse unless it is capable of adapting to the new WSN state. Sensor network query processors (SNQPs) construe a WSN as a distributed, continuous query platform where the streams of sensed values constitute the logical extents of interest. Crucially, in the context of this paper, they must make assumptions about the connectivity graph of the WSN at compile time that are likely not to hold for the lifetime of the compiled query evaluation plans (QEPs) the SNQPs generate. This paper address the problem of ensuring that a QEP continues to execute even if some nodes fail. The goal is to extend the lifetime of the QEP, i.e., the period during which it produces results, beyond the point where node failures start to occur. We contribute descriptions of two different approaches that have been implemented in an existing SNQP and present experimental results indicating that each significantly increases the overall lifetime of a query compared with non adaptive approach.

View PDFchevron_right
A Dynamically Scalable Cloud Data Infrastructure for Sensor Networks
Tonglin Li

Proceedings of the 6th Workshop on Scientific Cloud Computing - ScienceCloud '15, 2015

As small, specialized sensor devices become more ubiquitous, reliable, and cheap, increasingly more domain sciences are creating "instruments at large"-dynamic, often selforganizing, groups of sensors whose outputs are capable of being aggregated and correlated to support experiments organized around specific questions. This calls for an infrastructure able to collect, store, query, and process data set from sensor networks. The design and development of such infrastructure faces several challenges. The challenges reflect the need to interact with and administer the sensors remotely. The sensors may be deployed in inaccessible places and have only intermittent network connectivity due to power conservation and other factors. This requires communication protocols that can withstand unreliable networks as well as an administrative interface to sensor controller. Further, the system has to be scalable, i.e., capable of ultimately dealing with potentially large numbers of data producing sensors. It also needs to be able to organize many different data types efficiently. And finally, it also needs to scale in the number of queries and processing requests. In this work we present a set of protocols and a cloud-based data streaming infrastructure called WaggleDB that address those challenges. The system efficiently aggregates and stores data from sensor networks and enables the users to query those data sets. It address the challenges above with a scalable multi-tier architecture, which is designed in such way that each tier can be scaled by adding more independent resources provisioned on-demand in the cloud.

View PDFchevron_right
Multi-query Optimization for Sensor Networks
Alan Demers

Lecture Notes in Computer Science, 2005

The widespread dissemination of small-scale sensor nodes has sparked interest in a powerful new database abstraction for sensor networks: Clients "program" the sensors through queries in a high-level declarative language permitting the system to perform the low-level optimizations necessary for energy-efficient query processing. In this paper we consider multi-query optimization for aggregate queries on sensor networks. We develop a set of distributed algorithms for processing multiple queries that incur minimum communication while observing the computational limitations of the sensor nodes. Our algorithms support incremental changes to the set of active queries and allow for local repairs to routes in response to node failures. A thorough experimental analysis shows that our approach results in significant energy savings, compared to previous work.

View PDFchevron_right
Infrastructure for data processing in large-scale interconnected sensor networks
ali salehi

Mobile Data Management, …, 2007

With the price of wireless sensor technologies diminishing rapidly we can expect large numbers of autonomous sensor networks being deployed in the near future. These sensor networks will typically not remain isolated but the need of interconnecting them on the network level to enable integrated data processing will arise, thus realizing the vision of a global" sensor Internet." This requires a flexible middleware layer which abstracts from the underlying, heterogeneous sensor network technologies and supports fast and simple ...

View PDFchevron_right
A QOS -AWARE REACTIVE APPROACH FOR QUERY PROCESSING IN WIRELESS SENSOR NETWORK
IJRIT Journal

Data aggregation is rather a common but critical operation in many applications of wireless sensor networks. Innovative techniques that improve energy efficiency to prolong the network lifetime are highly required. Wireless Sensor Networks (WSNs) are highly distributed networks consisting of a large number of tiny, low cost, light-weight wireless nodes deployed to monitor an environment or a system. Each node in a WSN consists of three subsystems: the sensor subsystem which senses the environment, the processing subsystem which performs local computations on the sensed data, and the communication subsystem which is responsible for message exchange with neighboring sensor nodes. While an individual sensor node has limited sensing region, processing power, and energy, networking a large number of sensor nodes give rise to a robust, reliable, and accurate sensor network covering a wide region satisfying QoS challenges. Thus, QOS -aware routing protocols in WSNs is a very important issue.

View PDFchevron_right
Power-Aware In-Network Query Processing for Sensor Data
Alexandros Labrinidis

2003

Minimizing energy consumption has been a major objective at all levels in sensor networks. In this paper, we present TiNA, an in-network aggregation scheme that maintains the user-specified quality of data requirement while significantly reducing the overall energy consumption. Specifically, since communication dominates power usage in sensor networks, TiNA exploits end-user temporal coherency tolerances to reduce the amount of information transmitted by individual nodes. Further, we show that TiNa, by using temporal coherency tolerances, can allow for better quality of data when the time given to perform readings is too short for all data to be propagated up through the network. We compare our proposed scheme against an existing in-network aggregation scheme with a local sensor cache. We present experimental results measuring both power savings and also the quality of data for group-by queries. These results show that TiNA can reduce power consumption by up to 60% without any loss in the quality of data and extend the life of the sensor network by up to 300%.

View PDFchevron_right
Query processing in sensor networks
rahul gaikwad

Distributed and Parallel Databases, 2010

Hardware for sensor nodes that combine physical sensors, actuators, embedded processors, and communication components has advanced significantly over the last decade, and made the large-scale deployment of such sensors a reality. Applications range from monitoring applications such as inventory maintenance over health care to military applications. In this paper, we evaluate the design of a query layer for sensor networks. The query layer accepts queries in a declarative language that are then optimized to generate efficient query execution plans with in-network processing which can significantly reduce resource requirements. We examine the main architectural components of such a query layer, concentrating on in-network aggregation, interaction of in-network aggregation with the wireless routing protocol, and distributed query processing. Initial simulation experiments with the ns-2 network simulator show the tradeoffs of our system.

View PDFchevron_right
A Survey on Sensor-Cloud: Architecture, Applications, and Approaches
shadab ansari

International Journal of Distributed Sensor Networks, 2013

Nowadays, wireless sensor network (WSN) applications have been used in several important areas, such as healthcare, military, critical infrastructure monitoring, environment monitoring, and manufacturing. However, due to the limitations of WSNs in terms of memory, energy, computation, communication, and scalability, efficient management of the large number of WSNs data in these areas is an important issue to deal with. There is a need for a powerful and scalable high-performance computing and massive storage infrastructure for real-time processing and storing of the WSN data as well as analysis (online and offline) of the processed information under context using inherently complex models to extract events of interest. In this scenario, cloud computing is becoming a promising technology to provide a flexible stack of massive computing, storage, and software services in a scalable and virtualized manner at low cost. Therefore, in recent years, Sensor-Cloud infrastructure is becoming ...

View PDFchevron_right

References (51)

  1. D. J. Abadi, D. Carney, U. C ¸etintemel, M. Cherniack, C. Con- vey, C. Erwin, E. F. Galvez, M. Hatoun, J.-H. Hwang, A. Maskey, A. Rasin, A. Singer, M. Stonebraker, N. Tatbul, Y. Xing, R. Yan, and S. B. Zdonik. Aurora: A Data Stream Management System. In ACM SIGMOD Conference, June 2003.
  2. D. J. Abadi, D. Carney, U. C ¸etintemel, M. Cherniack, C. Convey, S. Lee, M. Stonebraker, N. Tatbul, and S. Zdonik. Aurora: A New Model and Architecture for Data Stream Management. VLDB Jour- nal, 12(2), August 2003.
  3. P. M. G. Apers. Data allocation in distributed database systems. ACM TODS, 13(3), 1988.
  4. A. Arasu, B. Babcock, S. Babu, M. Datar, K. Ito, I. Nizhizawa, J. Rosenstein, and J. Widom. STREAM: The Stanford Stream Data Manager. In ACM SIGMOD Conference, June 2003.
  5. A. Arasu, S. Babu, and J. Widom. CQL: A Language for Continuous Queries over Streams and Relations. In DBPL Workshop, Sep. 2003.
  6. A. Arasu, M. Cherniack, E. Galvez, D. Maier, A. Maskey, E. Ryvk- ina, M. Stonebraker, and R. Tibbetts. Linear Road: A Stream Data Management Benchmark. In VLDB Conference, Sept. 2004.
  7. R. Avnur and J. M. Hellerstein. Eddies: Continuously Adaptive Query Processing. In ACM SIGMOD Conference, May 2000.
  8. B. Babcock, S. Babu, M. Datar, and R. Motwani. Chain: Operator Scheduling for Memory Minimization in Data Stream Systems. In ACM SIGMOD Conference, June 2003.
  9. B. Babcock, M. Datar, and R. Motwani. Load Shedding for Aggre- gation Queries over Data Streams. In IEEE ICDE Conference, April 2004.
  10. H. Balakrishnan, M. Balazinska, D. Carney, U. C ¸etintemel, M. Cherniack, C. Convey, E. Galvez, J. Salz, M. Stonebraker, N. Tat- bul, R. Tibbetts, and S. Zdonik. Retrospective on Aurora. VLDB Journal, Special Issue on Data Stream Processing, 2004. to appear.
  11. M. Balazinska, H. Balakrishnan, S. Madden, and M. Stonebraker. Availability-consistency trade-offs in a fault-tolerant stream pro- cessing system. Technical Report TR974, MIT, November 2004.
  12. M. Balazinska, H. Balakrishnan, and M. Stonebraker. Contract- Based Load Management in Federated Distributed Systems. In NSDI Symposium, March 2004.
  13. E. A. Brewer. Lessons from giant-scale services. IEEE Internet Computing, 5(4):46-55, 2001.
  14. D. Carney, U. C ¸etintemel, M. Cherniack, C. Convey, S. Lee, G. Sei- dman, M. Stonebraker, N. Tatbul, and S. Zdonik. Monitoring Streams -A New Class of Data Management Applications. In VLDB Conference, Hong Kong, China, August 2002.
  15. D. Carney, U. C ¸etintemel, A. Rasin, S. Zdonik, M. Cherniack, and M. Stonebraker. Operator Scheduling in a Data Stream Manager. In VLDB Conference, Berlin, Germany, September 2003.
  16. S. Chandrasekaran, O. Cooper, A. Deshpande, M. J. Franklin, J. M. Hellerstein, W. Hong, S. Krishnamurthy, S. Madden, F. Reiss, and M. A. Shah. TelegraphCQ: Continuous Dataflow Processing. In ACM SIGMOD Conference, June 2003.
  17. S. Chandrasekaran, A. Deshpande, M. Franklin, J. Hellerstein, W. Hong, S. Krishnamurthy, S. Madden, V. Raman, F. Reiss, and M. Shah. TelegraphCQ: Continuous Dataflow Processing for an Un- certain World. In CIDR Conference, January 2003.
  18. M. Cherniack, H. Balakrishnan, M. Balazinska, D. Carney, U. C ¸etintemel, Y. Xing, and S. Zdonik. Scalable Distributed Stream Processing. In CIDR Conference, Asilomar, CA, January 2003.
  19. C. Collett, P. Habraken, T. Coupaye, and M. Adiba. Active rules for the software engineering platform GOODSTEP. In 2nd Interna- tional Workshop on Database and Software Engineering, 1994.
  20. A. Das, J. Gehrke, and M. Riedewald. Approximate Join Processing Over Data Streams. In ACM SIGMOD Conference, June 2003.
  21. R. Gallager. A minimum delay routing algorithm using distributed computation. IEEE Transactions on Communication, 25(1), 1977.
  22. M. N. Garofalakis and Y. E. Ioannidis. Multi-dimensional resource scheduling for parallel queries. In ACM SIGMOD Conference, 1996.
  23. S. Ghandeharizadeh, R. Hull, and D. Jacobs. Heraclitus: Elevating deltas to be first-class citizens in a database programming language. ACM TODS, 21(3), 1996.
  24. J. Gray, P. Helland, P. O'Neil, and D. Shasha. The dangers of repli- cation and a solution. In ACM SIGMOD Conference, June 1996.
  25. A. Gupta and I. S. Mumick. Maintenance of materialized views: Problems, techniques and applications. IEEE Data Engineering Bul- letin, 18(2), 1995.
  26. J. M. Hellerstein, R. Avnur, A. Chou, C. Olston, V. Raman, T. Roth, C. Hidber, and P. Haas. Interactive Data Analysis: The Control Project. IEEE Computer, August 1999.
  27. J. M. Hellerstein, P. J. Haas, and H. J. Wang. Online Aggregation. In ACM SIGMOD Conference, May 1997.
  28. J.-H. Hwang, M. Balazinska, A. Rasin, U. C ¸etintemel, M. Stone- braker, and S. Zdonik. High-availability algorithms for distributed stream processing. In IEEE ICDE Conference, April 2005.
  29. D. Karger, E. Lehman, T. Leighton, R. Panigrahy, M. Levine, and D. Lewin. Consistent hashing and random trees: distributed caching protocols for relieving hot spots on the world wide web. In ACM STOC Symposium, 1997.
  30. O. Koremien, J. Kramer, and J. Magee. Scalable, adaptive load shar- ing for distributed systems. IEEE parallel and distributed technol- ogy: systems and applications, 1(3), 1993.
  31. S. R. Madden, M. J. Franklin, J. M. Hellerstein, and W. Hong. The Design of an Acquisitional Query Processor for Sensor Networks. In ACM SIGMOD Conference, June 2003.
  32. R. Motwani, J. Widom, A. Arasu, B. Babcock, S. Babu, M. Datar, G. Manku, C. Olston, J. Rosenstein, and R. Varma. Query Process- ing, Approximation, and Resource Management in a Data Stream Management System. In CIDR Conference, January 2003.
  33. P. Radoslavov, R. Govindan, and D. Estrin. Topology-informed in- ternet replica placement. In WCW'01: Web Caching and Content Distribution Workshop, Boston, MA, June 2001.
  34. V. Raman and J. M. Hellerstein. Partial results for online query processing. In ACM SIGMOD Conference, June 2002.
  35. V. Raman, B. Raman, and J. M. Hellerstein. Online Dynamic Re- ordering. VLDB Journal, 9(3), 2000.
  36. R. Reiter. On specifying database updates. Journal of Logic Pro- gramming, 25(1), 1995.
  37. D. S. Santry, M. J. Feeley, N. C. Hutchinson, A. C. Veitch, R. W. Carton, and J. Ofir. Deciding when to forget in the Elephant file system. In ACM SOSP Symposium, December 1999.
  38. K. Schloegel, G. Karypis, and V. Kumar. Graph partitioning for high performance scientific simulations. In CRPC Parallel Computing Handbook. Morgan Kaufmann, 2000.
  39. M. Shah, J. Hellerstein, and E. Brewer. Highly-available, fault- tolerant, parallel dataflows. In ACM SIGMOD Conference, June 2004.
  40. N. G. Shivaratri, P. Krueger, and M. Singhal. Load distributing for locally distributed systems. Computer, 25(12), 1992.
  41. E. Simon and J. Kiernan. The A-RDL System. 1996.
  42. U. Srivastava and J. Widom. Flexible time management in data stream systems. In ACM PODS Symposium, June 2004.
  43. M. Stonebraker. The Design of the POSTGRESS Storage System. In VLDB Conference, Brighton, England, September 1987.
  44. N. Tatbul, U. C ¸etintemel, S. Zdonik, M. Cherniack, and M. Stone- braker. Load Shedding in a Data Stream Manager. In VLDB Con- ference, Berlin, Germany, September 2003.
  45. P. A. Tucker, D. Maier, and T. Sheard. Applying punctuation schemes to queries over continuous data streams. IEEE Data En- gineering Bulletin, 26(1), Mar. 2003.
  46. P. A. Tucker, D. Maier, T. Sheard, and L. Fegaras. Exploiting Punc- tuation Semantics in Continuous Data Streams. TKDE, 15(3), 2003.
  47. J. Widom and S. J. Finkelstein. A syntax and semantics for set- oriented production rules in relational database systems (extended abstract). SIGMOD Record, 18(3), 1989.
  48. M. Willebeek and A. Reeves. Strategies for dynamic load balanc- ing on highly parallel computers. IEEE Trans. on parallel and dis- tributed systems, 4(9), September 1993.
  49. O. Wolfson, S. Jajodia, and Y. Huang. An adaptive data replication algorithm. ACM TODS, 22(2), 1997.
  50. Y. Xing, S. Zdonik, and J.-H. Hwang. Dynamic load distribution in the borealis stream processor. In IEEE ICDE Conference, April 2005.
  51. S. Zdonik, M. Stonebraker, M. Cherniack, U. C ¸etintemel, M. Bal- azinska, and H. Balakrishnan. The Aurora and Medusa Projects. IEEE Data Engineering Bulletin, 26(1), March 2003.

Related papers

Stream engines meet wireless sensor networks: cost-based planning and processing of complex queries in AnduIN
Kai-Uwe Sattler

Distributed and Parallel Databases, 2010

Wireless sensor networks are powerful, distributed, self-organizing systems used for event and environmental monitoring. In-network query processors like TinyDB afford a user friendly SQL-like application development. Due to sensor nodes' resource limitations, monolithic approaches often support only a restricted number of operators. For this reason, complex processing is typically outsourced to the base station as a part of processing tasks. Nevertheless, previous work has shown that complete or partial in-network processing can be more efficient than the base station approach. In this paper, we introduce AnduIN , a system for developing, deploying, and running complex innetwork processing tasks. Particularly, we present the query planning and execution strategies used in An-duIN , which combines sensor-local in-network processing and a data stream engine. Query planning employs a multi-dimensional cost model taking energy consumption into account and decides autonomously which query parts will be processed within the sensor network and which parts will be processed at the central instance. Keywords sensor networks • data streams • power awareness • distributed computation • in-network query processing • query planning This work was in parts supported by the BMBF under grant 03WKBD2B and by the Science Foundation Ireland under Grant No. SFI/08/CE/I1380 (Lion-2) and 08/SRC/I1407 (Clique).

View PDFchevron_right
Query Processing and System-Level Support for Runtime-Adaptive Sensor Networks
Klaus Meyer-Wegener

Informatik aktuell, 2009

We present an integrated approach for supporting in-network sensor data processing in dynamic and heterogeneous sensor networks. The concept relies on data stream processing techniques that define and optimize the distribution of queries and their operators. We anticipate a high degree of dynamics and heterogeneity in the network, which is expected to be the case for wildlife monitoring applications. The distribution of operators to individual nodes demands several system level capabilities not available in current sensor node operating systems. In particular, we developed means for replacing software modules, i.e. small applications, on demand and without loss of status information. In order to facilitate this operation, we added a lightweight module support for the Nut/OS system and implemented a new memory management that uses tags for preserving state across module updates and node reboots.

View PDFchevron_right
MaD-WiSe: a distributed stream management system for wireless sensor networks
Claudio Vairo, Giuseppe Amato

2010

Wireless sensor networks (WSN) are composed of several sensors having limited memory, processing power, communication bandwidth, and energy, which cooperate in performing a given task. The use of the database paradigm has emerged in the last few years as a viable solution to manage data in such a context. In this paper we present the MaD-WiSe system, a distributed query processing framework that moves the processing of the query into the network. MaD-WiSe reconsiders various aspects related to database system design and it reinterprets them according to the WSN constraints and requirements. In particular it considers the aspects related to the definition of a query language to formalize the queries, a stream model to manage data acquired by the sensors, a query algebra to define the operators that actually perform the query, and energy efficiency and query optimization strategies for saving energy.

View PDFchevron_right
Data stream query optimization across system boundaries of server and sensor network
Klaus Meyer-Wegener

2006

with Borealis and TinyDB. Some preliminary results from the ongoing evaluation are given.

View PDFchevron_right
A stream database server for sensor applications
Walid Aref

2002

We present a framework for stream data processing that incorporates a stream database se11Jer as a fundamental component. The server operates as the stream control iflterjace between arrays of distributed data stream sources and end-user clients thaJ access and analyze the streams. The underlying framework provides novel stream managemem and query processing mechanisms to support the online acquisition, management, storage, non-blocking query. and imegration of data streams for distributed muLti-sensor networks. In this paper, we define OUT stream model and stream representation for the stream database, and we describe the functionality alld implementation of key components of the stream processing framework, including the query processing interface for source streams, the stream manager, the stream buffer manager, non-blocking query execution, and a new class ofjoin aLgorithms for joining multipLe data streams constrained by a sliding time window. We conduct experiments using real data streams to evaluate the performance of the new aLgoritluns against traditional stream join aLgorithms. The experiments show significant performance improvements and aLso demonstrate the flexibility of our system ;n handling data streams. A muLti-sensor network appLicatioll for the intelligent detection of lwzardous materials ;s presented to illustrate the capabilities ofourframework.

View PDFchevron_right

Related topics

  • Stream Processing
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved