Comparing High Dimensional Computer Simulation Output and Data Using Principal Components Analysis
ABSTRACT Complex computer models are often used to simulate natural phenomena. Evaluations of suc... more ABSTRACT Complex computer models are often used to simulate natural phenomena. Evaluations of such models are undertaken by comparing computer output to data. Summary measures of prediction error are commonly calculated to determine how good the computer model is. Such measures of t can be a useful start, but they obscure vital information by over-summarization. The problem is the high dimensional nature of the data and the computer model output. Principal components analysis is often used for multivariate data reduction. When the multivariate data are repeated measures, extensions to standard methodology are possible. We present these extensions and obtain useful assessments of model performance. We analyze output from a computer model of air quality in the Los Angeles basin and compare it to ozone measurements. 1 Key Words: Atmospheric Modeling; Computer Models; Exploratory Data Analysis; Graphical Data Analysis; Pollution; Repeated Measures. 1 Introduction Researchers using ...
Journal of the Royal Statistical Society: Series A (Statistics in Society), 2010
The use of cost information when defining critical values for prediction of rare events by using ... more The use of cost information when defining critical values for prediction of rare events by using logistic regression and similar methods
Criminology <html_ent glyph="@amp;" ascii="&amp;"/> Public Policy, 2005
Police officials across the United States often claimed credit for crime reductions during the 19... more Police officials across the United States often claimed credit for crime reductions during the 1990s. In this article, we examine homicide trends in three cities that mounted widely publicized policing interventions during the 1990s: Boston's Operation Ceasefire, New York's Compstat, and Richmond, Virginia's Project Exile. Applying growth-curve analysis to data from the 95 largest U.S. cities and controlling for conditions known to be associated with violent crime rates, we find that New York's homicide trend during the 1990s did not differ significantly from those of other large cities. We find some indication of a sharper homicide drop in Boston than elsewhere, but the small number of incidents precludes strong conclusions. By contrast, Richmond's homicide reduction was significantly greater than the decline in other large cities after the implementation of Project Exile, which is consistent with claims of an intervention effect, although the effect may have been small. Policy Implications: Criminologists gave police and other public officials something of a free ride as they claimed credit for the 1990s crime drop. We propose that researchers employ comparable data and methods to evaluate such claims-making, with the current analysis intended as a departure point for ongoing research. The use of common evaluation criteria is especially urgent for assessing the effects of the multiple interventions to reduce violent crime launched under the nation's primary domestic crime-control initiative, Project Safe Neighborhoods.
Journal of the American Statistical Association, 2021
We study a regression problem where for some part of the data we observe both the label variable ... more We study a regression problem where for some part of the data we observe both the label variable (Y) and the predictors (X), while for other part of the data only the predictors are given. Such a problem arises, for example, when observations of the label variable are costly and may require a skilled human agent. When the conditional expectation ErY |Xs is not exactly linear, one can consider the best linear approximation to the conditional expectation, which can be estimated consistently by the least squares estimates (LSE). The latter depends only on the labeled data. We suggest improved alternative estimates to the LSE that use also the unlabeled data. Our estimation method can be easily implemented and has simply described asymptotic properties. The new estimates asymptotically dominate the usual standard procedures under certain non-linearity condition of ErY |Xs; otherwise, they are asymptotically equivalent. The performance of the new estimator for small sample size is investigated in an extensive simulation study. A real data example of inferring homeless population is used to illustrate the new methodology.
In the early 1980s Halbert White inaugurated a "model-robust" form of statistical inference based... more In the early 1980s Halbert White inaugurated a "model-robust" form of statistical inference based on the "sandwich estimator" of standard error. This estimator is known to be "heteroskedasticity-consistent", but it is less well-known to be "nonlinearity-consistent" as well. Nonlinearity, however, raises fundamental issues because in its presence regressors are not ancillary, hence can't be treated as fixed. The consequences are deep: (1) population slopes need to be re-interpreted as statistical functionals obtained from OLS fits to largely arbitrary joint x-y distributions; (2) the meaning of slope parameters needs to be rethought; (3) the regressor distribution affects the slope parameters; (4) randomness of the regressors becomes a source of sampling variability in slope estimates; (5) inference needs to be based on model-robust standard errors, including sandwich estimators or the x-y bootstrap. In theory, model-robust and model-trusting standard errors can deviate by arbitrary magnitudes either way. In practice, significant deviations between them can be detected with a diagnostic test.
Objectives: Conventional statistical modeling in criminology assumes proper model specification. ... more Objectives: Conventional statistical modeling in criminology assumes proper model specification. Very strong and unrebutted criticisms have existed for decades. Some respond that although the criticisms are correct, there is for observational data no alternative. In this paper we provide an alternative. Methods: We draw on work in econometrics and statistics from several decades ago, updated with the most recent thinking to provide a way to properly work with misspecified models. Results: We show how asymptotically, unbiased regression estimates can be obtained along with valid standard errors. Conventional statistical inference can follow. Conclusions: If one is prepared to work with explicit approximations of a "true" model, defensible analyses can be obtained. The alternative is working with models about which all of the usual criticisms hold.
Journal of the American Statistical Association, 2003
The precarious state of the educational system in the inner cities of the United States, as well ... more The precarious state of the educational system in the inner cities of the United States, as well as its potential causes and solutions, have been popular topics of debate in recent years. Part of the dif culty in resolving this debate is the lack of solid empirical evidence regarding the true impact of educational initiatives. The ef cacy of so-called "school choice" programs has been a particularly contentious issue. A current multimillion dollar program, the School Choice Scholarship Foundation Program in New York, randomized the distribution of vouchers in an attempt to shed some light on this issue. This is an important time for school choice, because on June 27, 2002 the U.S. Supreme Court upheld the constitutionality of a voucher program in Cleveland that provides scholarships both to secular and religious private schools. Although this study bene ts immensely from a randomized design, it suffers from complications common to such research with human subjects: noncompliance with assigned "treatments" and missing data. Recent work has revealed threats to valid estimates of experimental effects that exist in the presence of noncompliance and missing data, even when the goal is to estimate simple intention-to-treat effects. Our goal was to create a better solution when faced with both noncompliance and missing data. This article presents a model that accommodates these complications that is based on the general framework of "principal strati cation" and thus relies on more plausible assumptions than standard methodology. Our analyses revealed positive effects on math scores for children who applied to the program from certain types of schools-those with average test scores below the citywide median. Among these children, the effects are stronger for children who applied in the rst grade and for African-American children.
This paper investigates the diffusion and institutionalization of change in formal organization s... more This paper investigates the diffusion and institutionalization of change in formal organization structure, using data on the adoption of civil service reform by cities. It is shown that when civil service procedures are required by the state, they diffuse rapidly and directly from the state to each city. When the procedures are not so legitimated, they diffuse gradually and the underlying sources of adoption change overtime. In the latter case, early adoption of civil service by cities is related to internal organizational requirements, with city characteristics predicting adoption, while late adoption is related to institutional definitions of legitimate structural form, so that city characteristics no longer predict the adoption decision. Overall, the findings provide strong support for the argument that the adoption of a policy or program by an organization is importantly determined by the extent to which the measure is institutionalizedwhether by law or by gradual legitimation.*
Heterogeneous treatment effects can be very important in the analysis of randomized clinical tria... more Heterogeneous treatment effects can be very important in the analysis of randomized clinical trials. Heightened risks or enhanced benefits may exist for particular subsets of study subjects. When the heterogeneous treatment effects are specified as the research is being designed, there are proper and readily available analysis techniques. When the heterogeneous treatment effects are inductively obtained as an experiment's data are analyzed, significant complications are introduced. There can be a need for special loss functions designed to find local average treatment effects and for techniques that properly address post selection statistical inference. In this paper, we tackle both while undertaking a recursive partitioning analysis of a randomized clinical trial testing whether individuals on probation, who are low risk, can be minimally supervised with no increase in recidivism. * Arun Kuchibhotla and Weijie Su provided important insights as the inferential procedures used in this paper were developed.
Heterogeneous treatment effects can be very important in the analysis of randomized clinical tria... more Heterogeneous treatment effects can be very important in the analysis of randomized clinical trials. Heightened risks or enhanced benefits may exist for particular subsets of study subjects. When the heterogeneous treatment effects are specified as the research is being designed, there are proper and readily available analysis techniques. When the heterogeneous treatment effects are inductively obtained as an experiment's data are analyzed, significant complications are introduced. There can be a need for special loss functions designed to find local average treatment effects and for techniques that properly address post selection statistical inference. In this paper, we tackle both while undertaking a recursive partitioning analysis of a randomized clinical trial testing whether individuals on probation, who are low risk, can be minimally supervised with no increase in recidivism. * Arun Kuchibhotla and Weijie Su provided important insights as the inferential procedures used in this paper were developed.
A number of papers have recently appeared claiming to show that in the United States executions d... more A number of papers have recently appeared claiming to show that in the United States executions deter serious crime. There are many statistical problems with the data analyses reported. This paper addresses the problem of "influence," which occurs when a very small and atypical fraction of the data dominate the statistical results. The number of executions by state and year is the key explanatory variable, and most states in most years execute no one. A very few states in particular years execute more than 5 individuals. Such values represent about 1% of the available observations. Re-analyses of the existing data are presented showing that claims of deterrence are a statistical artifact of this anomalous 1%.
AbstractBackground: It has become common practice to analyze randomizedexperiments using linear r... more AbstractBackground: It has become common practice to analyze randomizedexperiments using linear regression with covariates. Improved precision oftreatment effect estimates is the usual motivation. In a series of importantarticles, David Freedman showed that this approach can be badly flawed.RecentworkbyWinstonLinofferspartialremedies,butimportantproblemsremain. Results: In this article, we address those problems through a refor-mulation of the Neyman causal model. We provide a practical estimator andvalidstandarderrorsfortheaveragetreatmenteffect.Propergeneralizationsto well-defined populations can follow. Conclusion: In most applications, theuse of covariates to improve precision is not worth the trouble. 1 Department of Statistics, University of Pennsylvania, Philadelphia, PA, USA 2 Department of Criminology, University of Pennsylvania, Philadelphia, PA, USACorresponding Author:Richard Berk, Department of Criminology, Department of Statistics, University of Pennsyl-vania, 400 Jon ...
In medical practice, when more than one treatment option is viable, there is little systematic us... more In medical practice, when more than one treatment option is viable, there is little systematic use of individual patient characteristics to estimate which treatment option is most likely to result in a better outcome for the patient. This is due in part because practitioners do not have any easy way to holistically evaluate whether their treatment allocation procedure does better than the standard of care --- a metric we term "improvement". Herein, we present easy-to-use open-source software that provides inference for improvement in many scenarios, the R package PTE, "Personalized Treatment Evaluator" and in the process introduce methodological advances in personalized medicine. In the software, the practitioner inputs (1) data from a single-stage randomized trial with one continuous, incidence or survival endpoint and (2) a functional form of a model for the endpoint constructed from domain knowledge. The bootstrap is then employed on out-of-sample data to prov...
Uploads
Papers by Richard Berk