• confirmation bias:
- During the literature review that precedes experiment, we may preferentially seek and find evidence that confirms our beliefs or preferred hypothesis.
- We select the experiment most likely to support our beliefs. This insidiously frequent pitfall allows us to maintain the illusion of objectivity (for us as well as for others) by carrying out a rigorous experiment, while nevertheless obtaining a result that is comfortably consistent with expectations and desires.
This approach can hurt the individual more than the scientific community. When two conflicting schools of thought each generate supporting information, the battling individuals simply grow more polarized, yet the community may weigh the conflicting evidence more objectively. Individuals seeking to confirm their hypothesis may overlook ways of refuting it, but a skeptical scientific community is less likely to make that mistake.
• biased sampling: Subjective sampling that unconsciously favors the desired outcome is easily avoided by randomization. Too often, however, we fail to consider the relevance of this problem during experimental design, when countermeasures are still available.
• wish-fulfilling assumption: In conceiving an experiment, we may realize that it could be valuable and diagnostic if a certain assumption were valid or if a certain variable could be controlled. Strong desire for an obstacle to disappear tempts us to conclude that it is not really an obstacle.
Experiment Execution
• biased abortive measurements: Sometimes a routine measurement may be aborted. Such data are rejected because of our subjective decision that a distraction or an intrusion by an uncontrolled variable has adversely affected that measurement’s reliability. If we are monitoring the measurement results, then our data expectations can influence the decision to abort or continue a measurement. Aborted measurements are seldom mentioned in publications, because they weaken reader confidence in the experiment (and maybe even in the experimenter).
The biasing effect can be reduced in several ways: (1) ‘blind’ measurements, during which we are unaware of the data’s consistency or inconsistency with the tested hypothesis; (2) a priori selection of criteria for aborting a measurement; and (3) completion of all measurements, followed by discussion in the publication of the rationale for rejecting some.
• biased rejection of measurements: Unanticipated factors and uncontrolled variables can intrude on an experiment, potentially affecting the reliability of associated data. Data rejection is one solution. Many data-rejection decisions are influenced by expectations concerning what the data ‘should be’. Rejection may occur as soon as the measurement is completed or in the analysis stage. As with aborted data, rejected measurements should be, but seldom are, mentioned in publication. Datarejection bias is avoidable, with the same precautions as those listed above for reducing bias from aborted measurements.
• biased mistakes: People make mistakes, and elaborate error checking can reduce but not totally eliminate mistaken observations. Particularly in the field of parapsychology where subtle statistical
