The most important childhood development in reasoning skill is obtaining conscious control over the interaction between theory and evidence [Kuhn et al., 1988]. Immature thinking fails to distinguish between theory and evidence. Scientific thinking requires critical evaluation of observations and of their impact on the validity of hypotheses. This skill is polished by practice -- particularly by coping with contradictory evidence and contradictory hypotheses.
In order to relate evidence to hypotheses effectively, the researcher needs three related skills [Kuhn et al., 1988]:
• The evidence must be analyzed independently of the hypothesis, before evaluating the relationship between data and hypothesis.
• One must be able to think about a hypothesis rather than just with it. If one allows the hypothesis to guide interpretation of the evidence, objective evidence evaluation is impossible.
• While considering the impact of evidence on the hypothesis, one must be able to ignore personal opinion of the affected hypothesis. Favorable and unfavorable evidence must be given a chance to affect the final conclusion.
Kuhn et al. [1988] find that these three skills, which are absent in children and are developed gradually during middle adolescence and beyond, are still below optimum even in most adults.
Like most college students, I memorized facts and absorbed concepts, but I was seldom faced -at least in class-work -- with the ‘inefficient’ task of personally evaluating evidence and deciding what to believe. Imagine my surprise when I went to graduate school, began reading the scientific literature, and discovered that even some ridiculous ideas have proponents. Textbook learning does not teach us the necessity of evaluating every conclusion personally -- regardless of how famous the writer is, regardless of how meager one’s own experience is.
Effective evidence evaluation requires active critical thinking, not passive acceptance of someone else’s conclusion. The reader of a publication must become a reviewer who judges the evidence for and against the writer’s conclusion.
Effective evidence evaluation is far more comprehensive than discrimination of whether statements are correct. It also involves assessment of the scope and ambiguities of observations, generalizations, and deductions, as well as the recognition of implicit and explicit assumptions. Have all perspectives been considered? Is any conclusion warranted by the evidence?
The evaluation techniques of this chapter can aid in this wresting of control from subconscious feelings and toward rational decision-making.
Judgment Values
Evidence evaluation, like scientific research in general, involves not only technique but also style. One’s judgment of an hypothesis or evidence set is more a product of subjective values than of objective weighting factors. Those values, like scientific research style, are based on personal taste.
Prediction of observations is perhaps the most compelling type of confirmation or refutation. As discussed later in this chapter, the confirmatory power of evidence depends on how surprising the prediction is. A rule of thumb is:
