Using ChatGPT for Systematic Review Searching - Problem or Panacea?

Figure 1: Prompts used for ChatGPT for systematic review searching
Abbreviations: IL-1, interleukin-1; MeSH, Medical Subject Headings.

What were your key findings?
Writing a Boolean search and limiting it to randomised controlled trials
In answer to Prompt 1 (Figure 1), ChatGPT produced a search strategy that looked good but had some flaws (Figure 2). It did include indexing terms, keyword searching using synonyms for "postpartum depression", Boolean logic, and truncation. However, the number of synonyms or variant terms for postpartum depression were very limited; for example, it did not explore concepts such as depression in new mothers or new parents. Terms like these should be added to capture all the available evidence. Truncation was inconsistently applied. It included "depress*" as a term, which would also pick up depressed, depression, depressions etc. but "anxiety" was not truncated to pick up anxieties, and anxious did not feature at all. 

Figure 2: The search strategy produced by ChatGPT

For Prompt 2 (Figure 1), ChatGPT did successfully limit the search to randomised controlled trials (Figure 3), using a recognised set of validated search terms (a search filter) from the Cochrane Handbook for Systematic Reviews of Interventions [4]. This was impressive, but there are two versions of this filter, and the version used by ChatGPT was more focused and precise than the sensitivity-maximising version. This may be acceptable in the context of this example, but some knowledge of searching is needed to recognise this and to decide which version of the filter is most appropriate. At this stage, ChatGPT also introduced a limit to English language records only. It was not prompted to do this, and the language limit doesn't form part of the Cochrane search filter. Limiting to English language studies only would not generally be acceptable for a systematic review. It's also worth noting that the syntax used by ChatGPT to limit the search by language is syntax that won't run in the database. 

Figure 3: The search strategy produced by ChatGPT limited the search to randomised controlled trials

Our research showed that taking a search strategy written by ChatGPT and trying to run it uncritically in Ovid MEDLINE would result in a suboptimal search in the systematic review context. However, the rapid generation of a full strategy by ChatGPT – even if flawed – clearly has potential value to the experienced, critical searcher; the strategy can help inform early thinking on search challenges, strategy structure, and potential indexing and textword search terms.

Converting the search to a different database
In answer to Prompt 3 (Figure 1), the search strategy produced for CINAHL via Ebsco looked satisfactory on the surface but, on further analysis, there were serious problems with the indexing terms and search syntax. In this case, ChatGPT misinterpreted a search of title, abstract and keywords and instead created a strategy designed to search the full text of the documents. This would exponentially increase the number of records to screen and lead to less efficiency, not more! The key indexing term "postpartum depression" was also incorrectly entered in the strategy and retrieved zero results. The Boolean "AND" and "OR" searches were incorrectly rendered and would have searched for the wrong concept. The search that was produced by ChatGPT was unusable, although it looked as if it would work. It is possible that a novice searcher would not recognise or understand the flaws in the ChatGPT search and would attempt to run it, leading to the retrieval of many irrelevant studies, which would take additional time to screen. The search could also miss potential eligible studies, possibly introducing bias into the systematic review as a result.

Using ChatGPT to create the building blocks of a search strategy
For Prompt 4 (Figure 1), ChatGPT did retrieve a list of indexing terms for IL-1 inhibitors from MEDLINE, terms that we might potentially add to a search strategy (Figure 4). However, although most of the terms were correct, some of the terms presented were not indexing terms. Cytokine inhibitors was a term in the list, but this is not a MeSH. Anakinra is a type of IL-1 inhibitor but doesn't exist as a MeSH either. Hallucinations are a well-known issue with ChatGPT, and this seems to be another example of an area where, if ChatGPT doesn't "know", it makes a best guess [5]. 

Figure 4: The list of indexing terms from MEDLINE retrived by ChatGPT
Abbreviations: IL-1, interleukin-1; MeSH, Medical Subject Headings.