Conducting a literature review

Because you've been reading and writing all along, it's a bit of a misnomer to say that you need to "conduct a literature review". That phrase suggests that you find and read a lot of papers all at once, then write up a synthesis of them.

A more appropriate way to interact with the literature is in small batches as needed. You do a bit of analysis; it suggests that you should read some papers on a topic. You want to know how to deal with a problem in your data; the problem suggests you should learn more about methodology. 

In the beginning stages of your project when you don't know much about what you're doing or how to do it, interacting with a lot of literature very quickly can give you a rough idea of the shape of the scholarly conversation around your topic.  You'll be able to refine your topic and ideas, which will both help focus your research project and direct your further forays into the literature.  In the later stages of your project, you'll do a very focused search to help you figure out very specific questions and their answers. 

There are more papers published every year than anyone can read thoroughly, even if all they do is read.  To get started on reading papers, first you have to identify them and acquire their PDFs.  

Searching for papers

There are two basic strategies for getting and reading paper: depth-first, and breadth-first.  In a depth first search, you read a paper in depth, and use that paper to suggest the next paper you should read in depth.  You keep reading papers thoroughly, one at a time, until you know enough or until your brain shuts down.  I do not recommend this process.  As you read more papers, you'll get more out of each one.  If you do a depth-first search, you're learning more from the 10th paper than you are from the first, because you don't have enough background to really understand the scholarly conversation in the first paper.  You also run the risk of starting with papers that are only tangentially relevant, simply because you found them first.  That's not efficient.  

In a breadth-first search, you will identify a large number of papers (via searching), then iteratively skim and discard them to identify ever-more-relevant papers to read in depth.  In this search, because you will have skimmed a lot of papers before you read your first one, you're much better prepared to understand the scholarly conversation in your first paper.  You also have identified the most valuable papers to read first, which means you're not going to get distracted reading lots of papers that are only tangentially relevant.

The basic strategy for a breadth-first search goes like this.

1. Skim titles and abstracts.  

   • Goal: Is this relevant at all?

   • Scale: 100 papers, 3 mins per paper

   • Do this in your browser window, selecting the most promising papers to download.

2. Skim intro and figures

   • Goal: Is this helpful?

   • Scale: 20 papers, 5 mins per paper

   • These papers will help you understand the shape of the scholarly conversation and prioritize the most promising papers to read in more depth. 

3. Read relevant sections

   • Goal: Learn things

   • Scale: 5 papers, 25 mins per paper

   • These are the papers that you will summarize and synthesize. 

After you've gone through this cycle once, you have a much better grounding in the literature.  It's ok to go back to some of the papers that you found in part 1 or 2 and see if you'd like to prioritize them for reading now that you have more understanding.   

You should plan to do this process several times for each kind of literature that you want to look at: theories, methods, prior work on your topic, issues within your population, etc.  I do an abbreviated version of it (maybe only 20 papers to start with) whenever I get curious about a new-to-me idea. 

Tips for finding papers

Do you know one paper already? Read some of the papers that it cites.  This kind of backwards search is great to discover the seminal papers in a field, and to enter into the scholarly conversation about those topics.  The authors of the paper you've already read have done a lot of work to position their paper within a scholarly conversation, so go ahead and use their paper as a guide to reading older ones.  You can also do a "forward search": look for papers that cite the paper you've already found. Search for that paper on Google scholar or Web of Science, and then click on "cited by" to find papers that cite that paper (forwards search).  This works really well when the paper you have is old enough that other papers have already cited it. 

You can also use forward and backward search in tandem to skitter around in the research literature, backward searching to find seminal papers then forward searching to find recent ones. 

What are your keywords?  Do a search on Google scholar or Web of Science or ERIC using your keywords.  You should use a scholar search engine, not just a regular search engine, because you're looking for scholarly papers.  If your keywords are popular, this will give you a lot of papers to look for.  However, different fields tend to use different jargon to mean the same general idea.  Keywords that come from one field won't necessarily pick up the papers from another field.  You can mitigate this problem by searching for different kinds of keywords, and by being flexible about which papers will contribute to your research in different ways.  For example, if you're looking for papers about clinical interviewing and you come across papers that are related to challenges in your population, you can read those too.

Do you know a live human who knows about your topic?  You can ask them what to read.  Don't ask someone else to do your lit review for you, but it's ok to ask someone for a suggestion to get started, either as a quick introduction to a topic or as a good citation for a specific claim.  This is also helpful when you're looking for good keywords: other people might be able to suggest useful keywords for you to search on. 

Look nearby.  When you find a paper, consider looking at the other papers in that issue of the journal, or other papers by that author.  Some journals have a feature which suggests other papers in their journal based on what you've already found; other journals will tell you what their most downloaded papers are.  Go ahead and grab any papers that look like they might be interesting.  

What's a good way to tell if a paper is any good?

There are a lot of papers out there.  Some of them are great, and a lot of them are sub-great.  To evaluate the quality of a new paper you've found, here are some helpful questions:

• • Is it aligned with other papers you've skimmed?  The claims in this paper should be of similar scale and believability to other papers you've read, and this paper should cite (or be cited by) other papers that feel reasonable to you.  

  • Does it have a lot of citations?  People tend to cite good papers, so if this paper is old enough to have citations, then it should have some.  Different fields and journals have different norms for how many citations are a lot, so don't worry too much about strict numbers here.  

  • Did someone you respect recommend it?  If no, consider asking someone who knows about the field what they think of this paper, or whether they can recommend other papers to help you contextualize this one. 

  • Is it published by a reputable journal?  As a reader, it's sometimes hard to identify which journals are predatory and bad.  That said, journals published by professional societies tend to be good, and often journal papers are better than conference papers.   If you look around at the other papers in the journal, if they look good, then the journal is probably good. 

These heuristics are not perfect.  There are great papers in weird places, and there are bad papers in otherwise good places.  As you read more papers and develop a better sense for the scholarly conversations in your field, you'll become much better at identifying which papers are high quality for the work you do. 

Tips for acquiring papers

The most common way that I use for finding papers is via google scholar. Either via keyword search or by searching for the paper directly, I use google scholar to find me a link to the paper.  Sometimes it's a direct PDF download (hooray!), but more often it's a link to the journal.  The first step in a breadth-first search doesn't require me to download a paper -- I can read the title and abstract from the journal -- but if I select that paper for further reading, I probably need to download it.

If your university library has a subscription to the journal, you can download it from the journal site (you might need to search for it again via your institutional library).  If not, you can request it over inter-library loan (ILL).  Your library will have a form to fill out, then the librarians will comb through their secret librarian network and send you a pdf.  It sometimes takes a few days, but it's really magical. You can also use ILL to get specific chapters of books delivered to you as PDFs, so that's a great way to get the useful parts of books in a portable format that you don't have to return to the library.

If your university (including the librarians) can't help you, it's ok to email the authors directly and ask for the paper.  Academics really love it when someone wants to read their work, and they're usually gracious about sending it to you (or sending a link where you can download it).  You might get the "author submitted version" instead of the published version, but that's ok -- they're only trivially different.  You can also ask your friends: different universities have different access to papers, and what's hard for you might be easy for them. 

Never pay for papers.  There are tons of legal and free ways to get papers.  Journals might want to charge you to read their stuff, but that is never worth it.  

Managing all those papers

You will acquire a very large number of papers, sometimes as PDFs, for possible future citation.  Use a reference manager, like Zotero, to organize your PDFs and their bibliographic information. Not only will this help you cite papers in your writing, but it will also help you remember what papers you have and what they are about.  Once your personal library gets big, that's a great place to start searching for future lit review cycles.  

Do not keep your PDFs in random folders on your hard drive. That method does not scale well.

Reading papers

If you came to this paper via the breadth-first search (above), then before you actually read it,  you've already read the title and abstract, and skimmed the introduction and figures. If someone gave you this paper directly, you might not have done that yet.  

No matter how this paper came to you, you'll get a lot more out of it if you read like this:

1. Read the title and abstract.  Ask yourself: what is the scope of this paper? why is it interesting or relevant to my work?

2. Skim the introduction and figures / tables.  Ask yourself: what are the central claims of this paper? why is it important?  what's confusing?  Do this quickly: you're just getting a sense for the paper, not actually reading it yet.

3. Skim the most relevant sections to you.  Depending on your research question and why this paper is relevant, that might be the theory or method or results.  Ask yourself: is there enough information here to understand what's going on? what are the authors trying to do? Do I agree with this framing or interpretation?

4. Read the paper as a whole.  Often you don't need to do this part, because you got all that you needed from a prior step.  If you do need to read the paper as a whole, all of your prior engagement  will help you get more out of it.  

Writing a summary

Some of the papers you find will be valuable enough to read.  When you read each paper, write a small summary of it. The summaries can be as long or as short as you like, but somewhat less than a page is good. The act of summarizing the paper will help you glean the ideas from it, and the archive of the summary will be useful later when you put together your paper.

I like to ask three questions of each paper I read:

• What's the point of this paper? Look for their central claims. 

• What's interesting or novel? This may or may not be the point of the paper: maybe you liked the methodology, or the way they describe professional development, or the delicate way they refuted counterclaims. 

• What's confusing or annoying? Maybe their visualizations were ugly, or they were too acronym-happy, or they got the physics wrong.

Synthesizing papers

Once you have a bunch of summaries, you are ready to start synthesizing them to make part of your argument. Array your summaries before you and group them thematically. Give each group a descriptor (e.g. "Professional development for preservice teachers"; "Methods of video-based research"; "Ampere's law is hard"). It's common to have the same paper in multiple groups, but if two groups are the same as each other except for one paper, think about combining them into one group. Practically speaking, I like to have 3-10 papers per group.

Within each group, write a paragraph about why these papers are together: they cover the same methods, they ask the same kinds of questions, they study the same kinds of populations, whatever.

From the summary of each paper in the group, select a few sentences of information about what makes that paper special within the theme. You might want to return to the papers to find more about what makes them special. Finally, for each paper, write a clause about why this paper is relevant to your argument.

Putting your lit review in your paper

Some papers have a section, usually after the introduction, called "Literature Review", where they concentrate a lot of their citations. It's often more appropriate to think about putting several smaller reviews of the literature in your paper: 

• in the introduction, where you argue that your paper fills a hole in the literature; 

• in the theory section, where you combine elements of theories to describe why your work is meaningful; 

• in the methods section, where you describe what you did and why that was appropriate.

Your thematic clusters of papers -- and the generative writing you did to synthesize them -- form the nucleus of these reviews.  As you collect your generative writing into a paper, go ahead and insert your syntheses into the appropriate parts of the paper.  You might notice that there are some places where your argument needs to be better connected to existing literature.  This is normal.  Everyone needs to go back to the literature again to find more papers, synthesize the papers, and refine their own arguments.   The more you do that, the more you'll be able to include a synthesis of a paper you've already read, rather than having to read a new paper. 

No one is ever "done" with a literature search. There are always more papers to read, if not about your content area, then about methodologies or theoretical frameworks. Maybe there are comparable results in cognate fields to compare in your discussion section. You will never be done.

But, you can be "done enough". How much is "enough" is collaboratively determined between you, your co-authors, and your reviewers. You know you're not done yet when someone asks "hey, what else has been done here?" or "hey, how have these problems been posed/solved before?" and someone else says "good question!". You are done when nobody asks those questions any more, or when the best response to them is "answering that question is outside the scope of this paper".

On a purely practical level, you know you've found a quorum of relevant papers when:

• The new papers you read only cite (or are cited by) papers you've already read. This means you've found a local maximum of information. 

• Google scholar or Eric or other database searches on your topic (possibly assisted by a trained librarian) don't turn up new relevant material in the first two pages of results. This means you've found a global maximum of information. 

• Your beta-readers tell you that you're ok.