Oompa-Loompas (AI) for Systematic Literature Review
An introduction to ASreview
This seminar introduces participants to ASReview software which significantly speeds up screening of abstracts during the literature review process. We also show how to extract relevant knowledge from papers into tables.
This workshop has been recorded and the link for watching will be sent on request.
Install ASreview
ASreview for now only runs withing Python environment
Install Anaconda Python Distribution
After installation, there will be new app on your computer named "Anaconda Prompt"
Start anaconda prompt and install asreview by typing "pip install asreview"
Then verify that the app is correctly installed by typing "asreview lab"
Run/Close ASreview
To run ASreview, simply start anaconda prompt and type "asreview lab"
The web window will open, this is your work environment; while you are working keep the Anaconda Prompt running
Everything in asreview is being saved automatically on-the-go, you don't have to worry
When you are finished with working for the day (or hour), you can just close both the web window and Anaconda Prompt
Get acquainted with PRISMA
PRISMA guides you through 4 stages of your LR
1. Identification = identifying records through database searching
2. Screening = reading abstracts and deciding which papers will survive for reading stage
3. Eligibility assesment = reading the full papers that survived the screening phase and deciding whether they are really relevant or not, put notes on the papers that were rejected (this will go into appendix of your paper)
4. Final inclusion = creating a table of final set of included papers, ideally extract the knowledge from papers into a table (when possible)
Here is a PRISMA checklist for your LR paper. Both can also be filled online here
Complete 2020 paper with guidelines is openly available here
Step 1: Identification - Search for all relevant literature in Scopus and Web of Science
Be broad! Use the least ammount of limitations
But note that for now we have to limit our search to max. 2000 results in Scopus and 50 results in WoS
Login with your uni account to Scopus Search. To access Scopus Search through Prague University of Economics and Business, use this link.
Use advanced queries, see the search tips here . Search mainly withing TITLE-ABS-KEY (means the searchword must be present either in title, abstract or keywords). On the other side ALL looks for the searchword everywhere in the paper (usually not very helpful).
When you are done, export your search (RIS or BibTex format, so it also includes abstracts)
Fill the appropriate part of PRISMA checklist and workflow after this step
Step 2: Screening - review abstracts in ASreview OR use Rayyan to help with your screening
Start ASreview (Anaconda Prompt: "asreview lab")
Import the RIS/BibTex file from Scopus into your ASreview
Do the preliminary screening - it is always very helpful if you already know some influential article you like. Otherwise just use the random function
Note: you decide when you stop reviewing, we usually stop after we get 20-30 irrelevant abstracts
After you stop reviewing, export the results in csv/excel. Also export your project (for safety reason and for future attachment to the submission of the paper)
Fill the appropriate part of PRISMA checklist and workflow after this step
Rayyan
If you do not wish to use ASReview, you can also use Rayyan which also prioritizes articles (after you make your first 50 decisions)
Step 3: Assess papers for eligibility, provide reasons for rejection
Work in an Excel file that was exported from ASReview. Skimm through the papers, put a column with notes on why the papers were rejected at this stage - this very much depends on your research question. Simple table with accepted and rejected papers and notes on rejection will go to the appendix of your paper.
For example in our venture capital literature review, we were only interested in studies that used counterfactual methodology. For each reject paper, we then wrote "no counterfactuals". In our gender studies, we only included studies that specifically dealth with a concept of "belief".
Fill the appropriate part of PRISMA checklist and workflow after this step
Step 4: Final inclusion & knowledge extraction
Create a list of final included papers. Start extracting the knowledge into columns
The columns very much depend on your research scope, but usually more is better then less
We provide an example of table for empirical and experimental literature review
In many domains, it might be beneficial to have a column titled "Open questions?" where you can write what the paper outlines as future avenues for research. This might be beneficial for your future research
The table with extracted knowledge should ideally be in body of your text, after you have it finished, you can start writing the paper. The table is in my humble opinion the most valuable part of the paper.
Using chatGPT to aid in knowledge extraction
You can use chatGPT to assist you in extracting the knowledge but then carefully check by skimming through the paper!
Example:
chatGPT first prompt:
I am writing a systematic literature review on the Venture Capital and Growth of Young High-Tech Firms in the EU: A Systematic Review of the Empirical Evidence . I am giving you one of the papers that came positive from screening phase in PRISMA framework. Now I need to extract following information into Excel table which will be then provided in the paper. First answer this question: is the study empirical - does it use dataset and statistical methods? Please extract following information from the table: Title, authors, journal, which countries are covered by the dataset used (use two-character country codes), period that dataset covers (use only years in format yyyy - yyyy), treatment condition (whether receiving Venture Capital is operationalized as a treatment effect and if yes which kind of venture capital), sample characteristics (sample size and what the units in sample are - firms, institutions, startups?), underlying dataset (if used, such as ZEW, VICO, RITA, etc.), outcome variable in the models used (if more outcomes, list all of them in alphabetical order), control variables (list all of them in alphabetical order), empirical methods used (such as matching, PSM, OLS, 2-step IV, list all the methods used), findings - what the analysis found (does VC financing has effect on growth of startups and if yes provide more characteristics but limit this to max 4 sentences)
Problem - chatGPT was looking only for two-letter country codes, here are revised instructions I sent afterwards and then it worked:
read the countries also normally, not just alphabetical codes - only return extracted countries in two letter code. same for years - looking for mentions of period covered in different format
Last step - start writing the paper :-)
The table is a central part of your paper, it gives it extra value because it is a very easy and fast-to-read knowledge. Many people will only read your table ( ! ). You can see the included table e.g. in our article here.
Tips for writing: cluster papers into topics they deal with and/or the methods they use (depending on the context). Write one paragraph per cluster. Discuss the evidence brought by the papers overall, don't forget to discuss limitations of (groups of) papers. The most important part of the paper (after summarizing the knowledge in a table) is a section "where do we go from here?" that should directly provide inspiration to researcher that will be reading your paper to what to do next.
Extra-bits - Meta-analysis
Here you can watch Christian Elbaek on discussing the meta-analysis techniques on the example of his Material Scarcity and Unethical Economic Behavior paper (it starts at after first hour of the workshop).