Analytic Transparency, Radical Honesty, and Strategic Incentives

2024

We worked on a paper assessing research transparency in CHI publications but decided to anonymize the dataset before publication. Behind this counterintuitive choice are several lessons for researchers interested in making their work more transparent, and developers of data repositories. BACKGROUND: WHAT IS RESEARCH TRANSPARENCY? Research transparency refers to disclosing details of processes and materials involved in a research project-from research design to reporting. Examples of transparent practices are preregistration and sharing research materials, data, and code. These practices could increase rigor and credibility and allow the scholarly community to build upon them.

This paper is probably our umpteenth and umptieth in our litany of paper on scientific method, and our advocacy of the need to bring it uptodate to suit the needs of the present time. We begin this paper aptly by defining what research is, what scientific method is, and follow it up by debating and discussing the bare essentials of transparency, objectivity, and objectivity in mindset. We explain why transparency is of paramount important to scientific endeavour, and can by and large eliminate scientific fraud and misconduct too. We also explore and summarize the current state of transparency in research, and review efforts made by "The Center for Open Science" which promotes transparency in research and puts out guidelines, on issues such as citation standards, data transparency, analytic methods, design and analysis transparency. Various types of transparency such as data and method transparency and reproducibility are also probed and investigated in this paper, along with value ethics and publication ethics. There are several reasons we write this paper. The first is that the ideals of this movement need to be spread among all and sundry because awareness is generally lacking, particularly in developing parts of the world, secondly we need to merge this with the globalization of science movement, and promote better and greater cross-cultural research design, third because there are too many ideologies rampant in science, fourthly because the ideals of this movement need to be incorporated into scientific method as required, fifthly, because an integration with our other papers on scientific method is required, sixthly because universities and research organizations need to issue or come out with their standards, and seventhly because suitable cascading changes in pedagogy and curricula as required.

Statistical Science, 2010

Putting data into the public domain is not the same thing as making those data accessible for intelligent analysis. A distinguished group of editors and experts who were already engaged in one way or another with the issues inherent in making research data public came together with statisticians to initiate a dialogue about policies and practicalities of requiring published research to be accompanied by publication of the research data. This dialogue carried beyond the broad issues of the advisability, the intellectual integrity, the scientific exigencies to the relevance of these issues to statistics as a discipline and the relevance of statistics, from inference to modeling to data exploration, to science and social science policies on these issues.

Merrill Series on The Research Mission of Public Universities

Scientific journals have long acted as a stabilizing force in academia, by defining scientific communities, demarcating subfields and showcasing their key insights. Stability derives not least from the structure of a scientific paper, which imposes order on the ever-shifting processes of data collection and analysis by fitting results into a systematic and cohesive narrative, aimed at persuading readers of the validity of a given knowledge claim. This is what philosophers of science have dubbed 'reconstruction' or 'justification' of scientific insights: papers provide an account that excludes research aspects and outputs that are not directly relevant to the arguments at hand – such as experiments or models that failed, data that proved irrelevant or neutral with regards to the hypotheses at stake, and procedural details that do not fit existing formats.1 Journal publications thus provide science with a quantifiable, fixed and spatio-temporally located output, which helps to adjudicate the success, and disseminate the achievements of, a given research effort.

Qualitative and Multi-Method Research, 2020

JCOM-Journal of Science Communication

The problem of accessing data is as old as science itself. Complete popularisation of scientific data (of a theoretical model), and even more so of the methods and materials used during an experimental process and of the empirical data amassed, has always been considered an essential part of the process of authentication, duplication and filing of scientific knowledge. It is also true, however, that this theory has always been a complex riddle with no simple solution. Strangely enough, in today's era of instant communication, the challenge of information access seems to be facing new, daunting obstacles, some of which have the same name and characteristics they had 100 or 300 years ago, but which have been intensified by new dimensions and unexpected corollaries. Others have a new core, an example being, the problem related to disclosure, which implies the (more or less) complete popularisation of the data, procedures, and tools used during research. This is a subject which, although ancient in form, has recently taken on new, far-reaching implications. The scientific community now has to face a problem which originated, first, with the sequencing of the human genome and, later, with that of certain types of rice; a problem which could redefine certain aspects of the epistemological practice and nature of science.

PS: Political Science & Politics, 2011

Dissemination of journal submission data is critical for identifying editorial bias, creating an informed scholarly marketplace, and critically mapping the contours of a discipline’s scholarship. However, our survey and case study investigations indicate that nearly a decade after the Perestroika movement began, political science journals remain reserved in collecting and releasing submission data. We offer several explanations for this lack of transparency and suggest ways that the profession might address this shortcoming.

Academy of Management Learning & Education, 2016

Research articles often give inaccurate information about how researchers developed hypotheses, analyzed data, and drew conclusions. Published articles sometimes report only some of the hypotheses that researchers tested, or some of the statistical analyses that researchers made. Articles often imply that researchers formulated all hypotheses before they examined their data when in fact they added or deleted hypotheses after they made some data analyses. Indeed, such covert practices are so common that new entrants into management research may think they are correct behavior. Yet, these practices create false impressions about the validity of research and they undermine the openness that ought to create trust among researchers. Researchers have tried to halt these practices by labeling them "unethical" but their continued prevalence questions the effectiveness of wholly critical approaches. This article proposes a constructive path toward reform: advocating honesty about actual research processes by adding discussions of inferences drawn after data analyses. Post-hoc data analyses can stimulate important theoretical ideas; running alternative statistical models can deepen understanding of empirical patterns; lack of support for hypotheses can identify incorrect or incomplete theories. The management research culture should encourage these practices. The negative effects result from the lack of explicit reporting about them.

Dados

We provide the first replication study of political science research published in Brazil by attempting to replicate every quantitative article published in five major Brazilian journals between 2012 and 2016. We also tested whether replication rates varied between established fields, more traditional and where the use of quantitative data is more common, and emerging fields. Our results show that transparency and reproduction are still in a development stage in Brazilian Political Science. Of the 650 articles reviewed, we asked for data to 197 quantitative articles. From those, only 28% agreed to share datasets and computed codes. We were able to attempt a replication for only 14%, and successfully reproduce the results of less than 5%. We conclude by suggesting the adoption of transparency and replicability procedures that are standard in other scientific communities.