Pamela Buckley

University of Colorado, Boulder, Institute of Behavioral Science, Faculty Member

Followers

Following

Co-authors

Public Views

Dr. Buckley is a Senior Research Associate in the Institute of Behavioral Science at the University of Colorado-Boulder. Trained as an educational psychologist, her expertise includes statistics, intervention research, evaluation methodology, and policy analysis and her interest is in improving opportunities for disadvantaged youth. Dr. Buckley has experience in conducting systematic reviews (she is certified as a What Works Clearinghouse reviewer in Group Design, versions 3.0 and 2.1) and expertise in research synthesis and experimental and quasi-experimental research design. As Principal Investigator of Blueprints for Healthy Youth Development, a web-based registry of programs proven to promote academic performance and emotional/physical well-being or to reduce violence, delinquency, and/or substance abuse among youth, she conducts reviews using guidelines that follow rigorous scientific standards and aides in disseminating certified programs to help policymakers and practitioners select effective programs. Dr. Buckley has worked extensively with large data sets, analysis tools, and advanced statistical techniques. As a former school psychologist, she also has considerable experience consulting in classrooms with teachers, students, families and communities.

less

Interests

Uploads

Papers by Pamela Buckley

Does Hot Spots Policing Reduce Crime? An Alternative Interpretation Based on a Meta-Analysis of Randomized Experiments

Hot spots policing is used by many U.S. police departments and is consistently supported by evide... more Hot spots policing is used by many U.S. police departments and is consistently supported by evidence from evaluations of its effectiveness. Hot spot police work requires identifying small areas where crime is especially concentrated and directing police resources to these areas. This paper meta-analyses 22 reports in which random assignment was used to increase police activities in some hot spots. The analysis finds that the police activities reduced crime by .046 to .051 standard deviation units, with the most plausible estimate being .046. All the average effect sizes were statistically different from zero. However, about a third of the study-specific estimates differed reliably from zero and the moderator variables failed to explain variation in impact. Hot spots policing reduces crime, but its effect is small and inconsistent. We discuss implications for policing theory and practice, and provide suggestions to improve the use of meta-analysis for identifying effective programs a...

Improving the Livelihoods of Justice-Involved Youth: Study Protocol for a Replication and Extension of Functional Family Therapy-Gangs

SSRN Electronic Journal

Does dual enrollment improve progression through college and earnings and do outcomes differ by sociodemographic characteristics or achievement levels? A quasi-experimental analysis of Colorado students

The High School Journal

Influence of Classroom-Level Factors on Implementation Fidelity During Scale-up of Evidence-Based Interventions

Prevention Science

As evidence-based interventions (EBIs) become more widely disseminated, fidelity of implementatio... more As evidence-based interventions (EBIs) become more widely disseminated, fidelity of implementation (FOI) often wanes. This study explores the association between FOI and malleable variables within classrooms that could be targeted to optimize resources without compromising FOI as school-based EBIs are disseminated across real-world settings. We utilized process evaluation data from a national dissemination project of the Botvin LifeSkills Training (LST) middle school program, a universal prevention intervention shown to reduce substance use. The sample included 1,626 teachers in 371 schools across 14 states. Hierarchical linear models examined the relationship between observational measures of implementation factors and three domains of fidelity (e.g., adherence, student responsiveness, and quality of delivery). Findings suggest that curriculum modifications, student misbehavior, and shortage of time to implement the LST middle school program were factors most associated with lower ...

Download

Champions of Transparency in Education: What Journal Reviewers Can Do to Encourage Open Science Practices

As the field of education and especially gifted education gradually moves towards open science, o... more As the field of education and especially gifted education gradually moves towards open science, our research community increasingly values transparency and openness brought by open science practices. Yet, individual researchers may be reluctant to adopt open science practices due to low incentives, barriers of extra workload, or lack of support to apply these in certain areas, such as qualitative research. We encourage and give guidelines to reviewers to champion open science practices by warmly influencing authors to consider applying open science practices to quantitative, qualitative, and mixed methods research and providing ample support to produce higher-quality publications. Instead of imposing open science practices on authors, we advocate reviewers suggest small, non-threatening, specific steps to support authors without making them feel overwhelmed, judged, or punished. We believe that these small steps taken by reviewers will make a difference to create a more supportive e...

The influence of evidence-based program registry websites for dissemination of evidence-based interventions in behavioral healthcare

Evaluation and Program Planning

Visitors’ Assessment and Utilization of Evidence-Based Program Resources (EBPR) Websites

Substance Use & Misuse

Racial and Ethnic Representation in Preventive Intervention Research: a Methodological Study

Prevention Science

Postsecondary Outcomes of the Concurrent Enrollment Policy in Colorado

Proceedings of the 2020 AERA Annual Meeting, 2020

Substance Use Prevention During Adolescence: Study Protocol for a Large-Scale Cluster Randomized Trial of Botvin High School Lifeskills Training

SSRN Electronic Journal

Objective. Because racial and ethnic minoritized groups disproportionately represented essential ... more Objective. Because racial and ethnic minoritized groups disproportionately represented essential workers and lacked equitable access to resources that mitigated exposure and mortality risk, the COVID-19 pandemic brought disparities to the forefront of public health, exacerbating existing discrepancies. These inequities highlight a pressing need for the prevention science eld to investigate whether interventions promote equitable well-being, which served as the impetus for this study. We examined 885 programs with evaluations published from 2010-2021 and recorded in the Blueprints for Healthy Youth Development registry of preventive interventions. Methods. Focusing on race and ethnicity but also tracking reporting of gender, location, and economic disadvantage, we conducted a descriptive analysis on the prevalence of culturally tailored interventions (i.e., those developed for speci c populations) and reporting of sample characteristics. In addition, inferential analyses examined reporting time trends, as well as the relationship between study quality (i.e., methodological soundness, bene cial effects) and culturally tailored programs, and racial ethnic enrollment. Results. Most studies were conducted in the U.S. (n=583). Two percent of programs were developed for Black or African American youth and four percent targeted Hispanic or Latino populations. For the 77% of studies that reported race, most enrollees were White (35%) followed by Black or African American (28%), and 31% were collapsed across racial categories or categorized with ethnicity, thus ignoring the intersectionality of race and ethnicity. In the 64% of studies that reported ethnicity, 32% of enrollees identi ed as Hispanic or Latino. Reporting did not improve over time, and there was no relationship between high quality evaluations and programs developed for racial ethnic minority groups, or samples with high proportions of racial ethnic enrollees. Conclusions. Research gaps on racial and ethnic groups identi ed in this study indicate the need for improved representation and clear reporting to reduce disparities and improve the utility of preventive interventions.

Download

Representation of Racial and Ethnic Minoritized Groups in Preventive Intervention Research

Download

Implementing Evidence-Based Preventive Interventions During a Pandemic

Global Implementation Research and Applications

Download

Does Concurrent Enrollment Program Improve Time-To-Degree and Wages? A Study to Inform the Scale-up of Colorado's Concurrent Enrollment Programs Act

This study serves to inform Colorado's understanding of concurrent enrollment as a driver of ... more This study serves to inform Colorado's understanding of concurrent enrollment as a driver of efficiency in getting to an on-time postsecondary credential. High school students that participate in the Colorado's concurrent enrollment program may enroll tuition-free in postsecondary courses and earn college credits that are transferable to any Colorado public university.

The Role of Clearinghouses in Promoting Transparent Research: A Methodological Study of Transparency Practices for Preventive Interventions

Prevention Science, 2021

Objective. Transparency of research methods is vital to science, though incentives are variable, ... more Objective. Transparency of research methods is vital to science, though incentives are variable, with only some journals and funders adopting transparency policies. Clearinghouses are also important stakeholders; however, to date none have implemented formal procedures that facilitate transparent research. Using data from the longest standing clearinghouse, we examine transparency practices for preventive interventions to explore the role of online clearinghouses in incentivizing researchers to make their research more transparent. Methods. We conducted a descriptive analysis of 88 evaluation reports reviewed in 2018-19 by Blueprints for Healthy Youth Development, when the clearinghouse began checking for trial registrations, and expanded on these efforts by applying broader transparency standards to interventions eligible for an endorsement on the Blueprints website during the study period. Reports were recent, with 84% published between 2010 and 2019. Results. We found few reports had data, code, or research materials that were publicly available. Meanwhile, 40% had protocols that were registered, but only 8% were registered prospectively, while one-quarter were registered before conducting analyses. About one-third included details in a registered protocol describing the treatment contrast and planned inclusions, and less than 5% had a registered statistical analysis plan (e.g., planned analytical methods, pre-specified covariates, etc.). Confirmatory research was distinguished from exploratory work in roughly 40% of reports. Reports published more recently (after 2015) had higher rates of transparency. Conclusions. Preventive intervention research needs to be more transparent. Since clearinghouses rely on robust findings to make well-informed decisions and researchers are incentivized to meet clearinghouse standards, clearinghouses should consider policies that encourage transparency to improve the credibility of evidence-based interventions.

Making Evidence-Based Interventions Relevant for Users: A Comparison of Requirements for Dissemination Readiness Across Program Registries

Evaluation Review, 2020

This study compares prevention program registries in current use on their level of support for us... more This study compares prevention program registries in current use on their level of support for users seeking to implement evidence-based programs. Despite the importance of registries as intermediaries between researchers and the public, and although previous studies have examined how registries define their standards for methodological soundness and evidence of efficacy, little research has focused on the degree to which registries consider programs’ dissemination readiness. The result is that registry users are uncertain whether listed programs and their necessary support materials are even available for implementation. This study evaluates 11 publicly and privately funded prevention registries that review the evidence base of programs seeking to improve child health and prosocial outcomes on the degree to which they use dissemination readiness as an evidentiary criterion for rating programs, and the extent and type of information they provide about dissemination readiness to supp...

Download

Common Methodological Problems in Randomized Controlled Trials of Preventive Interventions

Prevention Science, 2021

Objective. Randomized controlled trials (RCTs) are often considered the gold standard in evaluati... more Objective. Randomized controlled trials (RCTs) are often considered the gold standard in evaluating whether intervention results are in line with causal claims of beneficial effects. However, given that poor design and incorrect analysis may lead to biased outcomes, simply employing an RCT is not enough to say an intervention “works.” This paper applies a subset of the Society for Prevention Research (SPR) Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research, with a focus on internal validity (making causal inferences) to determine the degree to which RCTs of preventive interventions are well-designed and analyzed, and whether authors provide a clear description of the methods used to report their study findings. Methods. We conducted a descriptive analysis of 851 RCTs published from 2010-2020 and reviewed by the Blueprints for Healthy Youth Development web-based registry of scientifically-proven and scalable interventions. We used Blueprints’ evaluation criteria that correspond to a subset of SPR’s standards of evidence. Results. Only 22% of the sample satisfied important criteria for minimizing biases that threaten internal validity. Overall, we identified an average of 1-2 methodological weaknesses per RCT. The most frequent sources of bias were problems related to baseline non-equivalence (i.e., differences between conditions at randomization) or differential attrition (i.e., differences between completers versus attritors or differences between study conditions that may compromise the randomization). Additionally, over half the sample (51%) had missing or incomplete tests to rule out these potential sources of bias. Conclusions. Most preventive intervention RCTs need improvement in rigor to permit causal inference claims that an intervention is effective. Researchers also must improve reporting of methods and results to fully assess methodological quality. These advancements will increase the usefulness of preventive interventions by ensuring the credibility and usability of RCT findings.

Evidence‐based juvenile justice programs and practices: A critical review

Criminology & Public Policy, 2020

There is growing critical commentary and debate about the relative effectiveness of individual pr... more There is growing critical commentary and debate about the relative effectiveness of individual program and generic practice approaches to identifying evidencebased interventions and their impact on the operation of the juvenile justice system. The central issue is whether both of these approaches to identifying evidence-based interventions provide a valid and reliable guide to improving juvenile justice programming and, if so, what are the relative advantages and disadvantages of each? From a public policy perspective, should we be investing more heavily in one or the other, or treating them as effective complementary approaches and encourage both? We address each of these questions and offer some suggestions for improving the effectiveness of each approach.

Download

The relationship between mothers' child abuse potential and current Mental Health symptoms

The Journal of Behavioral Health Services & Research, 2005

This analysis examined data from mothers at 2 of the 9 sites participating in Substance Abuse and... more This analysis examined data from mothers at 2 of the 9 sites participating in Substance Abuse and Mental Health Services Administration's (SAMHSA 's) national Women Co-occurring Disorders and Violence Study (WCDVS). According to previous literature, it was hypothesized that women in the WCDVS would be at high risk of perpetrating child abuse. This research examined mothers' potential for physical child abuse and assessed the association between child abuse potential, current mental health symptoms, alcohol and drug use severity, and trauma. Results revealed that participants" had significant potential for child abuse. Hierarchical regression analyses revealed that current mental health symptoms were the strongest predictor of mothers' scores on the Child Abuse Potential (CAP) Inventory. This study highlights" the important relationships between commonly used instruments across the mental health, substance, and child welfare fields and the potential dual use of these instruments. Implications for policy and practice are discussed.

Download

Does Training Modality Predict Fidelity of an Evidence-based Intervention Delivered in Schools?

Prevention Science

Training prior to implementing evidence-based interventions (EBIs) is essential to reach high lev... more Training prior to implementing evidence-based interventions (EBIs) is essential to reach high levels of fidelity. However, the time and cost of in-person training are often barriers to implementation. Online learning offers a potential solution, though few studies examine the relationship between online training and fidelity of implementation. This study explored whether teachers trained online have similar levels of adherence, dosage, quality of delivery, and student responsiveness compared to teachers trained in-person on the Botvin LifeSkills Training (LST) middle school program, a universal prevention intervention proven to reduce substance use and violence, as part of a national dissemination project. This study involved a sample of 989 LST teachers across 114 school districts, representing 296 schools in 14 states. All teachers were first trained in LST implementation between 2016 and 2019. Hierarchical linear models were used to assess relationships between training modality ...

Download

Validating a Fidelity Scale to Understand Intervention Effects in Classroom-Based Studies

American Educational Research Journal

Pamela Buckley

Uploads

Papers by Pamela Buckley

Log In