A One-Pass Space-Efficient Algorithm for Finding Quantiles

Proceedings of the VLDB Endowment, 2016

Modern enterprises generate huge amounts of streaming data, for example, micro-blog feeds, financial data, network monitoring and industrial application monitoring. While Data Stream Management Systems have proven successful in providing support for real-time alerting, many applications, such as network monitoring for intrusion detection and real-time bidding, require complex analytics over historical and real-time data over the data streams. We present a new method to process one of the most fundamental analytical primitives, quantile queries, on the union of historical and streaming data. Our method combines an index on historical data with a memory-efficient sketch on streaming data to answer quantile queries with accuracy-resource tradeoffs that are significantly better than current solutions that are based solely on disk-resident indexes or solely on streaming algorithms.

Evolving Systems, 2010

Quantiles play an important role in data analysis. On-line estimation of quantiles for streaming data-i.e. data arriving step by step over time-especially with devices with limited memory and computation capacity like electronic control units is not as simple as incremental or recursive estimation of characteristics like the mean (expected value) or the variance. In this paper, we propose an algorithm for incremental quantile estimation that overcomes restrictions of previously described techniques. We also develop a statistical test for our algorithm to detect changes, so that the on-line estimation of the quantiles can be carried out in an adaptive or evolving manner. Besides a statistical analysis of our algorithm, we also provide experimental results comparing our algorithm with a recursive quantile estimation technique which is restricted to continuous random variables.

IEEE Transactions on Knowledge and Data Engineering, 2000

In this paper, we present new algorithms for dynamically computing quantiles of a relation subject to insert as well as delete operations. At the core of our algorithms lies a small-space multiresolution representation of the underlying data distribution based on random subset sums or RSSs. These RSSs are updated with every insert and delete operation. When quantiles are demanded, we use these RSSs to estimate quickly, without having to access the data, all the quantiles, each guaranteed to be accurate to within userspecified precision. While quantiles have found many uses in databases, in this paper, our focus is primarily on network management applications that monitor the distribution of active sessions in the network. Our examples are drawn both from the telephony and the IP network, where the goal is to monitor the distribution of the length of active calls and IP flows, respectively, over time. For such applications, we propose a new type of histogram that uses RSSs for summarizing the dynamic parts of the distributions while other parts with small volume of sessions are approximated using simple counters.

arXiv (Cornell University), 2020

We present UDDSketch (Uniform DDSketch), a novel sketch for fast and accurate tracking of quantiles in data streams. This sketch is heavily inspired by the recently introduced DDSketch, and is based on a novel bucket collapsing procedure that allows overcoming the intrinsic limits of the corresponding DDSketch procedures. Indeed, the DDSketch bucket collapsing procedure does not allow the derivation of formal guarantees on the accuracy of quantile estimation for data which does not follow a sub-exponential distribution. On the contrary, UDDSketch is designed so that accuracy guarantees can be given over the full range of quantiles and for arbitrary distribution in input. Moreover, our algorithm fully exploits the budgeted memory adaptively in order to guarantee the best possible accuracy over the full range of quantiles. Extensive experimental results on synthetic datasets confirm the validity of our approach.

1997

The cpquantile of an ordered sequence of data values is the element with rank 'pn, where n is the total number of values. Accurate estimates of quantiles are required for the solution of many practical problems. In this paper, we present a new algorithm for estimating the quantile values for disk-resident data. Our algorithm has the following characteristics: (1) It requires only one pass over the data; (2) It is deterministic; (3) It produces good lower and upper bounds of the true values of the quantiles; (4) It requires no a priori knowledge of the distribution of the data set; (5) It has a scalable parallel formulation; (6) Extra time and memory for computing additional quantiles (beyond the first one) are constant per quantile. We present experimental results on the IBM SP-2. The experimental results show that the algorithm is indeed robust and does not depend on the distribution of the data sets.

Proceedings of the 2006 Winter Simulation Conference, 2006

Managing telecommunication networks involves collecting and analyzing large amounts of statistical data. The standard approach to estimating quantiles involves capturing all the relevant data (what may require significant storage/processing capacities), and performing an off-line analysis (what may delay management actions). It is often essential to estimate quantiles as the data are collected, and to take management actions promptly. Towards this goal, we present a minimalist approach to sequentially estimating constant/changing over time quantiles. We follow prior work and devise a fixed-point algorithm, which does not estimate the unknown probability density function at the quantile. Instead, our algorithm uses the log-odds transformation of the observed fractions, and the exponentially smoothed estimates of the standard deviation to update the quantile estimate. For large data streams, this algorithm can significantly reduce the amount of collected data and the complexity of data analysis.

1998

Abstract Quantiles are convenient measures of the entire range of values of simulation outputs. However, unlike the mean and standard deviation, the observations have to be stored since calculation of quantiles requires several passes through the data. Thus, quantile estimation (QE) requires a large amount of computer storage and computation time.

Encyclopedia of Database Systems, 2009

SYNONYMS Median; histogram; selection; order statistics DEFINITION Quantiles are order statistics of data: the φ-quantile (0 ≤ φ ≤ 1) of a set S is an element x such that φ|S| elements of S are less than or equal to x and the remaining (1 − φ)|S| are greater than x. This article describes data stream (single-pass) algorithms for computing an approximation of such quantiles.

1985

Abstract A heuristic algorithm is proposed for dynamic calculation of the median and other quantiles. The estimates are produced dynamically as the observations are generated. The observations are not stored; therefore, the algorithm has a very small and fixed storage requirement regardless of the number of observations. This makes it ideal for implementing in a quantile chip that can be used in industrial controllers and recorders. The algorithm is further extended to histogram plotting. The accuracy of the algorithm is analyzed.

2017

This paper gives drastically faster gossip algorithms to compute exact and approximate quantiles. Gossip algorithms, which allow each node to contact a uniformly random other node in each round, have been intensely studied and been adopted in many applications due to their fast convergence and their robustness to failures. Kempe et al. [FOCS'03] gave gossip algorithms to compute important aggregate statistics if every node is given a value. In particular, they gave a beautiful O( n + 1/ϵ) round algorithm to ϵ-approximate the sum of all values and an O(^2 n) round algorithm to compute the exact ϕ-quantile, i.e., the the ϕ n smallest value. We give an quadratically faster and in fact optimal gossip algorithm for the exact ϕ-quantile problem which runs in O( n) rounds. We furthermore show that one can achieve an exponential speedup if one allows for an ϵ-approximation. We give an O( n + 1/ϵ) round gossip algorithm which computes a value of rank between ϕ n and (ϕ+ϵ)n at every node....