PERSPECTIVE ARTICLE
Front. Ecol. Evol., 19 May 2015 | http://dx.doi.org/10.3389/fevo.2015.00051
Blind trust in unblinded observation in Ecology, Evolution, and Behavior
Melissa R. Kardish1,2*, Ulrich G. Mueller1, Sabrina Amador-Vargas1, Emma I. Dietrich1, Rong Ma1, Brian Barrett1 and Chi-Chun Fang1
1Department of Integrative Biology, University of Texas at Austin, Austin, TX, USA
2Center for Population Biology, University of California, Davis, CA, USA
ABSTRACT
We surveyed 492 recent studies in the fields of ecology, evolution, and behavior (EEB) to evaluate potential for observer bias and the need for blind experimentation in each study. While 248 articles included experiments that could have been influenced by observer bias, only 13.3% of these articles indicated that experiments were blinded. The use of blind observation therefore was either grossly underreported in the surveyed articles, or many EEB studies were not blinded. We hope that a concerted effort of the field of EEB—including researchers, peer-reviewers, and journal editors—will help promote and institute routine, blind observation as an essential standard that should be practiced by all sciences.
Training in the scientific method emphasizes accurate observation, unbiased experimentation, and objective thinking. Despite this training, much research remains infused with unconscious biases (Rosenthal and Rosnow, 2007; Fanelli, 2010), resulting in wasted effort (i.e., the need to rectify wrong conclusions) and even medical harm (Foster et al., 1975; Hróbjartsson et al., 2012; van Wilgenburg and Elgar, 2013; Kozlov et al., 2014; Tuyttens et al., 2014). It is unclear how much published science is affected by bias and might therefore be false (Ioannidis, 2005; Fanelli, 2012).
Unconscious confirmation bias is a common source of inaccurate observation where observers interpret what they see in a way that supports their expectations or preferred hypotheses. To eliminate confirmation bias, experimenters should blind themselves, for example by concealing any information that could bias observations (e.g., awareness of the hypothesis tested or of the treatment condition of a specific sample). While blinding is routine in medical and psychological research (Burghardt et al., 2012; Hróbjartsson et al., 2012; Begley, 2013), blinding is unfortunately not a universal practice in biology. For example, a survey of kin-recognition studies—a cornerstone of animal behavior—found that 71% of studies testing for kin recognition in ants did not report the use of blind observation, and, more disconcerting, studies that did not report blind observation reported significantly greater effect sizes than studies that reported blinding (van Wilgenburg and Elgar, 2013). Likewise, herbivory of woody plants was rated nearly 500% higher with unblinded methods compared to blinded methods (Kozlov et al., 2014). We here expand on such previous surveys by evaluating the methods of 492 research articles published recently in the general area of ecology, evolution, and behavior (EEB), specifically assessing the use of blind observation and the potential for observer bias to affect results.
For our survey, we selected nine prominent journals publishing in the area of EEB (e.g., Ecology, Evolution, American Naturalist, Animal Behavior) and EEB articles in four general-interest journals (Science, Nature, Proceedings of the National Academy of Sciences, Proceedings of the Royal Society B) (see Supplementary Materials, Appendix A). For each study, we first evaluated whether observations could potentially be influenced by observer bias, then asked whether blind observation was reported in the published methods. Because blinding is sometimes logistically difficult, for studies that did not report blind observation, we also evaluated whether blinding would be (i) easy, (ii) easy with a naïve experimenter, or (iii) difficult to implement. We attempted to evaluate relative importance of blind observation between studies, however, criteria for the importance of blind observation differ between experimental designs and between EEB research areas (see Table S1), and exact methodological details were sometimes difficult to evaluate for readers (a key reason why blinding should be routine to assure readers of unbiased observation). Two readers independently read and scored each article; if scores differed, the readers discussed the respective article in detail to reach a consensus (see Supplementary Materials for detailed protocols). Readers also identified experimental steps where blind observation would reduce any bias and evaluated possible methodologies that could have been used to reduce observer bias. Table S1 summarizes the rationale for scores of each article.
Across all 492 EEB articles surveyed, we judged 50.4% (n = 248) to have potential for observer bias, but only 13.3% (n = 33 of 248) of these articles stated use of blind observation. Some articles explicitly stated the use of blind observation in the methods (n = 24), while others indicated indirectly that experiments had been done blind (n = 9; e.g., use of a naïve experimenter; Figure 1). In the remaining articles (n = 244 of 492), it seemed unlikely that unblinded observation could bias observations, though many studies could easily have been blinded (Figure S1) (Balph and Balph, 1983). Therefore, either the use of blind observation was grossly underreported in the surveyed articles, or many studies were not conducted blind.
FREE PDF GRATIS: Frontiers in Ecology and Evolution
