Research is the process of creating and communicating novel, valuable, valid and generalizable knowledge.
Therefore, when you read and evaluate a research paper, you want to first ask yourself what conceptual claims the authors are making. Once you have identified these claims (some authors will cleanly list them for you while others will bury them deep in their prose), for each claim you want to decide whether it is:
- Novel -- this is very hard to do unless you know the field well so for now assume that whatever the authors claim is novel.
- Valuable -- do we care that this new piece of knowledge has been shared with the world? Does it enable further knowledge production? Does it enable practical applications?
- Valid -- here you want to evaluate the internal validity of the work, that is: does the chain of the reasoning presented in the paper hold together? does the design of the evaluation really address the questions it is meant to answer? do the results provide compelling evidence in support of the claims?
- Generalizable -- if you determine that the claims are valid, how far do they generalize? Given the evidence presented, do you believe that they will hold in a broad variety of settings or will they only work in only in the exact settings in which the authors conducted their evaluation?
So, just to be painfully explicit, when you write your paper responses you should include the following elements:
- Start with a synthesis of the paper's contributions. In some cases (especially when you feel there are substantial flaws in the paper), you may wish to make a distinction between claimed contributions and actual contributions.
- Next, in any order you wish, evaluate whether the contributions are valuable, valid and generalizable to the extent that the authors claim.
- Optionally, feel free to speculate how you might build on the results of the paper (follow up research ideas, practical applications).