“In almost every game of chess there comes a crisis that must be recognized. In one way or another a player risks something -- if he knows what he’s doing, we call it a ‘calculated risk’.
“If you understand the nature of this crisis; if you perceive how you’ve committed yourself to a certain line of play; if you can foresee you’ve committed yourself to a certain line of play; if you can foresee the nature of your coming task and its accompanying difficulties, all’s well. But if this awareness is absent, then the game will be lost for you, and fighting back will do no good.” [Reinfeld, 1959]
“Genius . . . means transcendent capacity of taking trouble.” [Carlyle, 17951881]
Preparation, experimental design, experiment execution, data analysis, and interpretation are all essential aspects of most research projects. Earlier sections of this chapter discussed experimental design with minimal reference to these companion facets of research, but here we will consider experimental design in the context of the overall experiment. Drawing on useful summaries by Wilson [1952], Killeffer [1969], and Open University [1970], I list the main initial steps in a research project (from conception through experiment), along with tips and guidelines on successful execution of each step:
1) state the general problem.
- What is the objective? Focus on a specific hypothesis; don’t undertake a fishing expedition.
- Is the experiment necessary? Can the question that the experiment hopes to address be answered by simply evaluating the hypothesis and its implications critically? Can the problem be solved by finding relevant data that are already published? Is it merely a puzzle that interests you (a perfectly valid reason) or does it affect interpretation of other problems? If the latter, is it a minor or major factor?
- Can the problem be restated in a form that makes it more feasible to solve? Should one test a simplified perspective or include refinements? Does the problem need to be broken down into components that are tested individually?
- What assumptions are implicit in the experimental design? Could the outcome of the hypothesis test be affected by an invalid assumption?
- What is the crux of the problem, the critical unknown aspect? What is the crucial, decisive experiment? Don’t settle for one that is merely incrementally useful, but don’t reject all potential experiments as not decisive enough.
2) thoroughly review existing data on the research topic. Review evidence on the more general problem, to the extent that time permits.
- Methodically summarize assumptions, data, interpretations, and speculations of previous relevant studies. Include your evaluation of reliability and possible weaknesses of each.
- Identify the critical deficiency of previous work.
3) select the most promising experiment.
- Seek a compromise or reconciliation of critical needs with viable techniques.
