Chunco, associate professor of environmental studies, was part of a consortium of scientists who studied the question of whether some of the problems with reproducibility might result from decisions scientists make when analyzing data.
Reproducibility in scientific experiments is an increasingly recognized issue in the sciences. Several recent studies have shown that replicating previously published experimental results can result in wildly different outcomes from the original study. To date, most studies of reproducibility have focused on how experiments were designed. However, reproducibility in data analysis has rarely been considered.
Amanda Chunco, associate professor of environmental studies, was part of a consortium of scientists who studied the question of whether some of the problems with reproducibility might result from decisions scientists make when analyzing data. To do so, scientists were given one of two real datasets from studies designed to answer a specific hypothesis in ecology.
One study looked at whether baby birds with more siblings grew more slowly because of competition with siblings. The other study measured how the amount of grass cover affected the growth of Eucalyptus seedlings.
The scientists were then asked to analyze the data they were given, draw conclusions, and write the methods and results section of a paper based on their findings. Importantly, each of the 174 different scientific teams involved did their analyses and statistical tests completely independently from each other. Then, each resulting analysis was reviewed by a different team of scientists acting as peer reviewers. Despite the datasets being identical, each scientist made different choices in how to appropriately analyze the data. For both datasets, results from the independent teams varied widely. This variability was not just in the statistical strength of the hypothesis test, but whether or not a team concluded that the data supported the hypotheses themselves.
This study is the first of its kind in ecology and will shape how ecologists think about statistics and experimental design in the future. The article was published as a preprint on EcoEvoRxiv and was profiled in the journal Nature.