Many scientists avoid consideration of mind/body interactions; scientific recognition of the placebo effect is an exception. This pitfall is familiar to nearly all scientists who use human subjects. It is avoidable through the use of a blind: the experimenter who interacts with the subject does not know whether the subject is receiving medicine or placebo.
• subconscious signaling: We can influence an experimental subject’s response involuntarily, through subconsciously signaling. As with the placebo effect, this pitfall is avoidable through the use of blinds.
Data Interpretation
• confirmation bias in data interpretation: Data interpretation is subjective, and it can be dominated by prior belief. We should separate the interpretation of new data from the comparison of these data to prior results. Most publications do attempt to distinguish data interpretation from reconciliation with previous results. Often, however, the boundary is fuzzy, and we bias the immediate data interpretation in favor of our expectations from previous data.
• hidden control of prior theories on conclusions: Ideally, we should compare old and new data face-to-face, but too often we simply recall the conclusions based on previous experiments. Consequently, we may not realize how little chance we are giving a new result to displace our prior theories and conclusions. This problem is considered in more detail in that part of the next chapter devoted to paradigms.
• biased evaluation of subjective data: Prior theories always influence our evaluation of subjective data, even if we are alert to this bias and try to be objective. We can avoid this pitfall through an experimental method that uses a blind: the person rating the subjective data does not know whether the data are from a control or test group, or what the relationship is of each datum to the variable of interest. However, researchers in most disciplines never even think of using a blind; nor can we use a blind when evaluating published studies by others.
• changing standards of interpretation: Subjectivity permits us to change standards within a dataset or between datasets, to exclude data that are inconsistent with our prior beliefs while including data that are more dubious but consistent with our expectations [Gould, 1981]. A similar phenomenon is the overestimation of correlation quality when one expects a correlation and underestimation of correlation quality when no correlation is expected [Kuhn et al., 1988].
Publication
• language bias: We may use different words to describe the same experimental result, to minimize or maximize its importance (e.g., ‘somewhat larger’ vs. ‘substantially larger’). Sarcasm and ridicule should have no place in a scientific article; they undermine data or interpretations in a manner that obscures the actual strengths and weaknesses of evidence.
• advocacy masquerading as objectivity: We may appear to be objective in our interpretations, while actually letting them be strongly influenced by prior theories. Gould [1981], who invented this expression, both criticizes and falls victim to this pitfall.
