EXPERIMENTAL LIMITATIONS

Conclusions about Measurable Hypotheses must be defended before being interpreted.

The Discussion can be divided into specific sections that make particular arguments. Using subheadings can clarify the purpose of each section of the Discussion.

Paragraph 1: SUMMARY.

Particularly for long or complex studies, it can be helpful to begin the Discussion by summarizing the overall reasoning of the paper (Brand and Huiskes, 2001). Leading up to the Discussion, the reasoning includes the overall goal of the study, the General Hypotheses, and the Measurable Hypotheses (explained in the Introduction). If the reasoning in the Results section is strong, the Discussion section can begin with a set of conclusions about Measurable Hypotheses (instead of simply reviewing the Measurable Hypotheses or the data themselves). Summarizing the Results seldom requires more than 1 paragraph (the first paragraph of the Discussion).

Paragraph 2: DEFENDING the Conclusions Despite Study LIMITATIONS.

After the summary of the Results, there may still be unanswered questions that the Discussion must address before moving on to test General Hypotheses (the main focus of the Discussion). Specifically, experimental research involves limitations. Careful scientific readers will reach the Discussion having already compiled a list of questions associated with the  limitations of the study. For example, "How does limitation Y affect the data collected by the study?" "What if the data had been collected or analyzed differently?" Most importantly, readers will ask the question: "Do any of the limitations, or choices affect the conclusions of the Results?"

It is important to DEFEND the conclusions of the Results from questions about potential limitations.

All experiments have limitations. Broadly, money and time can limit many aspects of experiments. Using the newest, most powerful equipment is not be possible for most researchers. Even well-provisioned laboratories may only be able to perform complex, expensive, or time-consuming analysis on subsets of data. Experimental subjects often have limited time that they can volunteer to participate in research. Access to patient populations may be limited. Participants may drop out of longitudinal studies. Animals show individual physiological and behavioral differences (and cannot be given specific instructions). And many, many more limitations can be encountered...

Because all studies have limitations, there is no shame in identifying experimental limitations. Scientists cannot expect all researchers to be able to conduct research with the state-of-the art equipment and infrastructure available to the world's best-supported institutions. Strong reasoning is just as important as technology for scientific progress (Platt, 1964). Moreover, innovative research can be performed with simple, clever techniques (Wigglesworth, 1974). The important criterion that scientists use for evaluating scientific publications is: are the conclusions of the research justified despite the limitations of the study. Therefore, it is important to make forthright arguments that study limitations do not affect the conclusions of the study. 

It is not possible to anticipate and address every conceivable question that potential readers could ask. However, clearly identifying the rationale for making choices in the Methods can help authors identify some of the questions about study limitations that readers are most likely to have. 

A useful framework for defending the conclusions of the study despite limitations is:

1) Identify the limitation, and why the limitation was unavoidable.

2) Explain with a reasoned argument why the limitation does NOT affect the conclusions of the study (e.g. the tests of the Measurable Hypotheses in the Results).

Both steps of the framework are important. Identifying limitations without immediately explaining the reasons that the limitations do not affect the conclusions implies that the limitations DO affect the conclusions. Limitations that are allowed to affect study conclusions degrade trust in all arguments of the paper. Therefore, it is important to explain why the conclusions ARE justified despite limitations. 

Arguing that identified limitations do not affect study conclusions can include references to other studies, alternative analysis of data, or limited additional calculations as necessary. For example, studies that were limited to laboratory data collection could identify evidence that laboratory measurements are equivalent to field measurements. Addressing probable questions and explaining how the limitations, or choices made, in the study are not likely to affect the conclusions of the Results can therefore strengthen the interpretation of the data and discussion of the General Hypotheses.

Arguments that the limitations of the research do not affect the conclusions of the study must be compelling. If a compelling argument is not possible, and a limitation can potentially affect the conclusions of the study, then the study may need substantial changes. Changes may involve revising the hypotheses, re-analyzing data, or even performing additional experiments. 

Do NOT try to ignore or conceal study limitations. Scientific readers are not likely to be deceived (sooner or later). Whereas limitations are an inevitable aspect of science, deception is a serious breach of scientific trust. Honor and integrity are essential to scientific progress, and deception invites severe consequences.