How to summarize the universe: Dynamic maintenance of quantiles

2007 Winter Simulation Conference, 2007

A useful performance measure on which to compare manufacturing systems is a quantile of the cycle time distribution. Unfortunately, aside from order statistic estimates, which can require significant data storage, the distribution of quantile estimates has not been shown to be normally distributed, violating a common assumption amongst ranking-and-selection (R&S) procedures. To address this, we provide empirical evidence supporting an approach using the mean of a group of quantile estimates as the comparison measure. The approach is detailed and illustrated through experimentation on four M/M/1 queues in which the 0.9 cycle-time quantile is the performance measure. Results in terms of simulation effort and accuracy are reported and compared to results obtained using the macro-replications approach for inducing normality as well as to results obtained by applying R&S procedures to quantile estimates directly. The suggested procedure is shown to provide significant savings in simulation effort while sacrificing very little in accuracy.

IEEE Transactions on Knowledge and Data Engineering, 2000

In this paper, we present new algorithms for dynamically computing quantiles of a relation subject to insert as well as delete operations. At the core of our algorithms lies a small-space multiresolution representation of the underlying data distribution based on random subset sums or RSSs. These RSSs are updated with every insert and delete operation. When quantiles are demanded, we use these RSSs to estimate quickly, without having to access the data, all the quantiles, each guaranteed to be accurate to within userspecified precision. While quantiles have found many uses in databases, in this paper, our focus is primarily on network management applications that monitor the distribution of active sessions in the network. Our examples are drawn both from the telephony and the IP network, where the goal is to monitor the distribution of the length of active calls and IP flows, respectively, over time. For such applications, we propose a new type of histogram that uses RSSs for summarizing the dynamic parts of the distributions while other parts with small volume of sessions are approximated using simple counters.

Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining

Estimating the distribution and quantiles of data is a foundational task in data mining and data science. We study algorithms which provide accurate results for extreme quantile queries using a small amount of space, thus helping to understand the tails of the input distribution. Namely, we focus on two recent state-of-the-art solutions:-digest and ReqSketch. While-digest is a popular compact summary which works well in a variety of settings, ReqSketch comes with formal accuracy guarantees at the cost of its size growing as new observations are inserted. In this work, we provide insight into which conditions make one preferable to the other. Namely, we show how to construct inputs for-digest that induce an almost arbitrarily large error and demonstrate that it fails to provide accurate results even on i.i.d. samples from a highly non-uniform distribution. We propose practical improvements to ReqSketch, making it faster than-digest, while its error stays bounded on any instance. Still, our results confirm that-digest remains more accurate on the "non-adversarial" data encountered in practice. CCS CONCEPTS • Theory of computation → Sketching and sampling; Streaming models; Data structures and algorithms for data management.

Journal of Statistical Planning and Inference, 2000

The properties of the sample quantiles of ranked-set samples are dealt with in this article. For any ÿxed set size in the ranked-set sampling the strong consistency and the asymptotic normality of the ranked-set sample quantiles for large samples are established. Bahadur representation of the ranked-set sample quantiles are also obtained. These properties are used to develop procedures for the inference on population quantiles such as conÿdence intervals and hypotheses testings. The e ciency of these procedures relative to their counterpart in simple random sampling is investigated. The ranked-set sampling is generally more e cient than the simple random sampling in this context. The gain in e ciency by using ranked-set sampling is quite substantial for the inference on middle quantiles and is the largest on the median. However, the relative e ciency damps away as the quantiles move away from the median on both directions. The gain in e ciency becomes negligible for extreme quantiles.

American Statistician, 1996

Journal of Statistical Planning and Inference, 2010

We derive expressions for the probability that an individual order statistic is closest to the target parameter among all order statistics from a complete random sample. We then apply these general results to location-scale parameter families of distributions with specific applications to estimation of percentiles. In this case, simultaneous-closeness probabilities depend upon the parameters through the value of p in the percentile and the sample size, n. The results can be applied to distributions with both bounded and unbounded supports. We illustrate the results with the estimation of percentiles for Normal and exponential.

Cornell University - arXiv, 2010

Very large datasets are often encountered in climatology, either from a multiplicity of observations over time and space or outputs from deterministic models (sometimes in petabytes= 1 million gigabytes). Loading a large data vector and sorting it, is impossible sometimes due to memory limitations or computing power. We show that a proposed algorithm to approximating the median, "the median of the median" performs poorly. Instead we develop an algorithm to approximate quantiles of very large datasets which works by partitioning the data or use existing partitions (possibly of non-equal size). We show the deterministic precision of this algorithm and how it can be adjusted to get customized precisions.

Evolving Systems, 2010

Quantiles play an important role in data analysis. On-line estimation of quantiles for streaming data-i.e. data arriving step by step over time-especially with devices with limited memory and computation capacity like electronic control units is not as simple as incremental or recursive estimation of characteristics like the mean (expected value) or the variance. In this paper, we propose an algorithm for incremental quantile estimation that overcomes restrictions of previously described techniques. We also develop a statistical test for our algorithm to detect changes, so that the on-line estimation of the quantiles can be carried out in an adaptive or evolving manner. Besides a statistical analysis of our algorithm, we also provide experimental results comparing our algorithm with a recursive quantile estimation technique which is restricted to continuous random variables.

New plots are proposed based on minimum and maximum order statistics that is visually appealing, easy to understand, stable at extreme tails and capture all information about the distribution of the data. The minimum and maximum plots give more weights to the data at the extreme tails unlike quantile quantile plot. Therefore, it can be considered these plots as a completeness of the quantile quantile plot. The minimum and maximum plots are used to obtain a nonparametric visualization for the Gumbel and Weibull distributions. Moreover, the minimum and maximum normal plots are introduced and compared with quantile quantile plot. The new plots have advantage to be applied to discrete distributions.

Journal of the Indian Society for Probability and Statistics, 2016

The Quantile based entropy measure own a few precise properties than its distribution function technique. In this article, the idea of quantile based Bilal's and Baig's uncertainty measure is prolonged for order statistics for residual and past lifetimes and have a look at their properties, this two parametric entropy measure characterizes the distribution characteristic uniquely. A few characterization results of generalized residual entropy of order statistics and monotonicity property is likewise mentioned.