Last updated: 28 October 2021

Introduction

Qualitative research can make an important contribution to health technology assessment (1). This can involve the identification and review of qualitative evidence (qualitative evidence synthesis) designed to explain differences in intervention effect, to explore aspects of implementation or to capture evidence of patient experience or preference (2-4). Whilst the integration of qualitative evidence in health technology assessment is not common, its value is increasingly recognised as useful (2, 5).

The process of identifying qualitative research is less clear than for identifying studies reporting randomised controlled trials. Poor reporting of qualitative research in studies (3, 6, 7), limited indexing of studies (3, 8, 9), apparent confusion in the reporting of methods of data collection (interviews, focus groups) and synthesis (e.g. thematic synthesis, meta-ethnography) in studies (10, 11), and a need to search beyond primary biomedical databases (12, 13), are some of the reasons that qualitative research is more challenging to identify (2). Moreover, methodological guidance on ‘how to’ literature search for qualitative research is limited where it does exist (c.f. (3, 14)) (2).

Research supporting the process of searching for qualitative research was reviewed and summarised in a methodological review by Booth (2016) (2). This review forms the basis for the original version of this chapter.

At the outset of this chapter, we thoroughly recommend contacting an information professional or researcher with experience in literature searching for qualitative research.

Sources to search

Bibliographic

A consensus has not been reached on the number of databases, or which databases, to be searched when conducting a literature search for qualitative research. Wright et al. (2014), in a retrospective case-study to consider the contribution of CINAHL when identifying qualitative research, indicated review teams searched between 3 and 20 databases to identify qualitative research (15).

The CRD handbook (3) currently recommends that searches should include the following databases for reviews and primary studies:

• MEDLINE

• Embase

• CINAHL - Cumulative Index to Nursing and Allied Health

• PsycINFO - psychological literature database

MEDLINE and CINAHL are the most frequently searched sources to identify qualitative research (16, 17) and their value has been demonstrated empirically (18). A study comparing search strategies on MEDLINE, EMBASE, CINAHL, and PsycINFO demonstrated the value of CINAHL in providing differentiated subject headings to retrieve qualitative studies in dementia (19). However, the extent to which this finding relates only to dementia-specific search strategies is unclear, thus indicating a need for further investigation into the optimal use and potential benefit of CINAHL and PsycINFO for qualitative evidence synthesis. Seventy-one qualitative systematic reviews (927 qualitative studies), from the Cochrane Database of Systematic Reviews and JBI database of Systematic Reviews and Implementation Reports, were used to analyse qualitative literature across nine bibliographic databases (20). 94.4% of the qualitative studies were indexed in at least one database. Maximum recall with two databases approached 90% (89.1%), with three databases recall increases to 92% and maximum recall with four databases was 93.1%. A long tail of the remaining 6.9% of the publications consisted of 1.3% scattered across five databases and 5.6% not indexed in any of the nine databases used by the study authors (20). Analysis of 23 systematic reviews on diabetes found that searching a combination of PubMed, Embase and CINAHL resulted in an overall recall rate of 99.3%, while adding PsycINFO increased overall recall to 99.8% (21). As a consequence, the authors recommend a combination of CINAHL, PubMed, Embase and PsycINFO. However it should be noted that retrospective studies of yields from reviews may optimise the yield of database searching in the absence of extensive non-database strategies that increase the denominator. The authors also recommend involvement of an information specialist to ensure high recall rates when conducting SRs on qualitative research within their topic area (21). .

The Social Science Citation index (SSCI, Clarivate Analytics) is also commended as a valuable resource for study identification (22). Reviewers are, however, strongly encouraged to identify specialist databases that relate to their review topic, together with databases that host particular types of publications or content e.g. dissertation abstracts, book chapters, theses and unpublished reports. Searching MEDLINE/PubMed was undertaken for 100% of a sample (21) and yet searching MEDLINE alone is considered inadequate. Where resources are limited the searcher should aim to access as diverse a sample of relevant databases as possible, for example capturing different disciplinary or professional perspectives and different literature types, as this will best further the objectives of the subsequent synthesis.

Supplementary search methods

Given the issues with identifying qualitative research (summarised above), supplementary search methods are considered particularly useful for identifying qualitative research (23). Supplementary search methods such as internet searching, personal contact with topic experts, forward citation searching and reference checking, may prove valuable since they may identify content not indexed in bibliographic databases and they do not necessarily rely on identifying precise search terms to identify studies and data (21). In a sample of 23 diabetes systematic reviews, checking reference lists of included articles resulted in a median (IQR) of 2.5 (1 - 6) more references being included per review than by database searches alone (21). A multipronged approach of supplementary searching has been recommended (11). A worked example demonstrates supplementary searching for a qualitative evidence synthesis within a Cochrane public health systematic review (25). Supplementary search methods are summarised by Cooper et al (2017) (24).

Designing search strategies

Differences between conducting quantitative and qualitative research are mirrored in differences when designing literature searches for quantitative and qualitative literature (26). Whilst quantitative research and searches take a linear and structured route, qualitative research searches often use emerging results to inform subsequent strategies in an iterative manner (26). As a consequence, qualitative searches may take a long time to plan, be labour intensive to undertake and challenging to replicate (2).

Search strategy design

Designing the search strategy may necessitate a different question formulation and search strategy structure from the Population, Intervention, Comparator, and Outcome (PICO) structure used by systematic reviews of intervention effectiveness (27). Booth (2) summarises alternative structures to PICO (see table 1).

Search filters

Qualitative search strategies seek to optimise a balance between the sensitivity of not missing relevant items against the specificity of only retrieving the most relevant items whilst, at the same time, keeping the yield within the resources available to complete the review. Search filters may help researchers to achieve this balance.

Search filters for commonly used databases (e.g. MEDLINE, Embase, PsycINFO and CINAHL) are reported on the ISSG search filters resource. Whilst researchers approach the use of search filters with caution in qualitative evidence synthesis (6), the performance of search filters is acceptable, albeit slightly inferior to search filters studies to identify trials (2). An overview of thirteen search filters to identify qualitative research papers in MEDLINE (39) reported a sensitivity of 93.63% for a search filter developed at McMaster University (40). For precision, the Medical Subject Heading “qualitative research” performed best at 2.15% but at the expense of sensitivity, retrieving less than a quarter of relevant studies (22.56%) (39). A University of Texas filter offered the optimal balance with a sensitivity of 81.96% and a precision of 0.80% (UTHealth) (41). An associated study extended evaluation to twelve search filters for PsycINFO and fifteen for CINAHL (42). Filters demonstrating the best sensitivity and precision ratio (among filters with a sensitivity of >80%) were developed at McMaster University for PsycINFO (43) and CINAHL (44) as subsequently evaluated by Rosumeck and colleagues (42).

Interestingly, short filters (e.g. qualitative OR interview* OR Interviews/ OR findings) have, in a limited number of case studies (7, 45), been demonstrated to perform comparably with more extensive filters. Reasons for this observation remain unclear but may relate to less common qualitative terms retrieving records already retrieved by more common terms, impairing their added value and unique yield. Unlike trials there has been no retrospective effort to index qualitative research and the MeSH term for "qualitative research" was only introduced in the year 2003.

It is important to note that there is no dominant search filter used or recommended in practice to identify qualitative research. Unlike reviews of intervention effectiveness, where the Cochrane HSSS is rightly dominant and widely used, a researcher using a qualitative literature search filter will need to match judiciously their choice of filter to the purpose of the literature search which will, in turn, vary by review (2).

Comprehensive literature searching?

For reviews of intervention effectiveness, a comprehensive, exhaustive literature search for studies is commonly perceived as the default requirement. Researchers have demonstrated that even a single missing study may impact negatively on the precision of an estimate of intervention effectiveness.

Qualitative evidence synthesis does not offer such a definitive demonstration of the impact of a missing study. A complete Interpretation of a phenomenon of interest may still be possible in the absence of missing studies, particularly if missing studies share the characteristics ("more of the same") of studies already represented within the sample. The value of comprehensive literature searches is less clear. Further research on how to approach these issues is indicated (46). We note that it may not be possible to undertake ‘exhaustive’ literature searches for qualitative research and the value of doing so continues to be challenged. It is important to understand that when the intent of a qualitative evidence synthesis is interpretive, rather than descriptive, literature searches for qualitative research may not need to be as comprehensive in their coverage as those undertaken to identify quantitative studies (47). On the other hand, unintentional omission of studies capturing particular perspectives, professions or geographical contexts may miss Important nuances (48) reinforcing the point that diversity within the relevant sample may be the most Important driver for the literature search strategy.

Reporting

Reporting the process of literature searching in qualitative evidence synthesis is no less important than for reviews of intervention effectiveness; more so if alternatives to comprehensive sampling are to be used and justified. With careful and disciplined documentation of the search processes by the searcher, producing a transparent account of literature searching is possible even for a more Iterative process (2). Reporting standards for qualitative evidence synthesis, include eMERGe for meta-ethnography (39), RAMESES for meta-narrative approaches (50) (alongside its higher profile guidance for realist syntheses (51)) and ENTREQ for generic approaches to QES (52). All these reporting standards include items relating to documentation of the search process. Most of these reporting standards are derivative from the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) reporting standard and so the number of studies identified by various literature search methods should be reported within a flowchart (53). Reporting of search strategies should also follow generic reporting guidance. While not a reporting standard per se, the EQUATOR Inventory does include a prototypic structure for reporting search strategies, STARLITE (54), compiled from almost 50 published QES:

Sampling strategy

Type of study

Approaches

Range of years

Limits

Inclusion and exclusions

Terms used

Electronic sources

The use of a search narrative has been recommended to improve the transparency of literature searching (55, 56). Whilst not explicitly examined for qualitative literature searching, providing detail on the conceptual and contextual details of the literature searching process is likely to hold value in explaining the conception and execution of the search strategy in a clear and transparent way (46).

Reference list

(1) Carroll C. Qualitative evidence synthesis to improve implementation of clinical guidelines. BMJ 2017, 356. [Publication appraisal]

(2) Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev 2016, 5(1):74. [Publication appraisal]

(3) Centre for Reviews & Dissemination. Systematic reviews: CRD’s guidance for undertaking reviews in healthcare. York: Centre for Reviews and Dissemination, University of York; 2009.][Publication appraisal]

(4) Booth A. Qualitative Evidence Synthesis. In: Facey K. PHH, Single A. (editors). Patient Involvement in Health Technology Assessment. Singapore: Adis; 2017: 187-199. [Publication appraisal]

(5) Murphy E, Dingwall R, Greatbatch D, Parker S, Watson P. Qualitative research methods in health technology assessment: a review of the literature. Health technology assessment (Winchester, England) 1998, 2(16):iii-ix, 1-274. [Publication appraisal]

(6) Grant, MJ. How does your searching grow? A survey of search preferences and the use of optimal search strategies in the identification of qualitative research. Health Info Libr J 2004, 21. [Publication appraisal]

(7) Gorecki C, Brown JM, Briggs M, Nixon J. Evaluation of five search strategies in retrieving qualitative patient-reported electronic data on the impact of pressure ulcers on quality of life. J Adv Nurs 2010, 66. [Publication appraisal]

(8) Toye F, Seers K, Allcock N, Briggs M, Carr E, Barker K. Meta-ethnography 25 years on: challenges and insights for synthesising a large number of qualitative studies. BMC Med Res Methodol 2014, 14. [Publication appraisal]

(9) Shaw RL BA, Sutton A, Miller T, Smith J, Young B. Finding qualitative research: an evaluation of search strategies. BMC Med Res Methodol. 2004;4:5. BMC Med Res Methodol 2004, 4(5). [Publication appraisal]

(10) Evans D. Database searches for qualitative research. J Med Libr Assoc 2002, 90. [Publication appraisal]

(11) Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M. Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Soc Sci Med 2003, 56. [Publication appraisal]

(12) Dixon-Woods M, Fitzpatrick R, Roberts K. Including qualitative research in systematic reviews: opportunities and problems. J Eval Clin Pract 2000, 7. [Publication appraisal]

(13) Atkins S, Lewin S, Smith H, Engel M, Fretheim A, Volmink J. Conducting a meta-ethnography of qualitative literature: lessons learnt. BMC Med Res Methodol 2008, 8. [Publication appraisal]

(14) Noyes J, Booth A, Cargo M, Flemming K, Harden A, Harris J, Garside R, Hannes K, Pantoja T, Thomas J. Chapter 21: Qualitative evidence. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. Available from www.training.cochrane.org/handbook. [Publication appraisal]

(15) Wright K, Golder S, Lewis-Light K. What value is the CINAHL database when searching for systematic reviews of qualitative studies? Syst Rev 2015, 4. [Publication appraisal]

(16) Hannes K, Macaitis K. A move to more systematic and transparent approaches in qualitative evidence synthesis: update on a review of published papers. Qual Res 2012, 12(4):402-442. [Publication appraisal]

(17) Dixon-Woods M, Booth A, Sutton AJ. Synthesizing qualitative research: a review of published reports. Qual Res 2007, 7(3):375-422. [Publication appraisal]

(18) Subirana M, Sola I, Garcia JM, Gich I, Urrutia G. A nursing qualitative systematic review required MEDLINE and CINAHL for study identification. J Clin Epidemiol 2005, 58(1):20-25. [Publication appraisal]

(19) Rogers M, Bethel A, Abbott R. Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: A comparison of search strategies. Res Synth Methods. 2018 Dec;9(4):579-586. doi: 10.1002/jrsm.1280. Epub 2017 Nov 27. [Publication appraisal].

(20) Frandsen TF, Gildberg FA, Tingleff EB. Searching for qualitative health research required several databases and alternative search strategies: a study of coverage in bibliographic databases. J Clin Epidemiol. 2019 Oct;114:118-124. doi: 10.1016/j.jclinepi.2019.06.013. Epub 2019 Jun 25. [Publication appraisal]

(21) Justesen T, Freyberg J, Schultz ANØ. Database selection and data gathering methods in systematic reviews of qualitative research regarding diabetes mellitus - an explorative study. BMC Med Res Methodol. 2021 Apr 30;21(1):94. doi: 10.1186/s12874-021-01281-2. [Publication appraisal]

(22) DeJean D, Giacomini M, Simeonov D, Smith A. Finding Qualitative Research Evidence for Health Technology Assessment. Qual Health Res 2016, 26(10):1307-1317. [Publication appraisal]

(23) Booth A: Chapter 3: Searching for Studies. In: Noyes J BA, Hannes K, Harden A, Harris J, Lewin S, Lockwood C. (editors). Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Cochrane Collaboration Qualitative Methods Group; 2011. [Publication appraisal]

(24) Cooper C, Booth A, Britten N, Garside R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Syst Rev 2017, 6:234. [Publication appraisal]

(25) Cooper C, Lovell R, Husk K, Booth A, Garside R. Supplementary search methods were more effective and offered better value than bibliographic database searching: A case study from public health and environmental enhancement. Res Synth Methods 2018, 9(2):195-223. [Publication appraisal]

(26) Mackay G. Is there a need to differentiate between qualitative and quantitative searching strategies for literature reviewing? Qual Soc Work 2007, 6(2):231-241. [Publication appraisal]

(27) Cooper C, Booth A, Varley-Campbell J, Britten N, Garside R. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol 2018, 18(1):85. [Publication appraisal]

(28) Sandelowski M, Barroso J. Searching for and retrieving qualitative research reports. In: Sandelowski M, Barroso J. (editors). Handbook for synthesizing qualitative research. New York: Springer; 2007. [Publication appraisal]

(29) Stern C, Jordan Z, McArthur A. Developing the review question and inclusion criteria. AJN: Am J Nurs 2014 Apr;114(4):53-6. [Publication appraisal]

(30) Petticrew M, Roberts H. How to find the studies: the literature search. In: Petticrew M, Roberts H. (editors). Systematic reviews in the social sciences: A practical guide. Oxford: Blackwell Publishing; 2006. [Publication appraisal]

(31) Booth A. Australian supermodel? A practical example of evidence-based library and information practice. (EBLIP). Health Info Libr J 2006 Mar;23(1):69-72. [Publication appraisal]

(32) Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well-built clinical question: a key to evidence-based decisions. ACP J Club 1995, 123. [Publication appraisal]

(33) Howell Major C S-BM. Designing the synthesis In: Major CH, Savin-Baden, M. (editors). An introduction to qualitative research synthesis: managing the information explosion in social science research. London: Routledge; 2010: 43–55. [Publication appraisal]

(34) Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res 2014, 14. [Publication appraisal]

(35) Booth A, Carroll C. Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable? Health Info Libr J 2015, 32. [Publication appraisal] [Free full text]

(36) Denyer D, Tranfield D. Producing a systematic review. In: Buchanan DA, Bryman A. (editors). The SAGE handbook of organizational research methods. London: SAGE Publications Ltd; 2009. [Publication appraisal]

(37) Wildridge V, Bell L. How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/management information. Health Info Libr J 2002, 19. [Publication appraisal]

(38) Cooke A, Smith D, Booth A. The SPIDER tool for qualitative evidence synthesis. Qual Health Res 2012, 22. [Publication appraisal]

(39) Wagner M, Rosumeck S, Küffmeier C, Döring K, Euler U. A validation study revealed differences in design and performance of MEDLINE search filters for qualitative research. J Clin Epidemiol. 2020 Apr;120:17-24. doi: 10.1016/j.jclinepi.2019.12.008. Epub 2019 Dec 18. [Publication appraisal]

(40) Wong SSL, Wilczynski NL, Haynes RB. Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE. Stud Health Technol Inform, 107 (2004), 311-316. [Publication appraisal]

(41) UTHealth [University of Texas Health Science Center at Houston] Search Filters for Various Databases: Ovid Medline. Last Updated: 28.11.2018. Houston, US-TX: UTHealth. Available at http://libguides.sph.uth.tmc.edu/search_filters/ovid_medline_filters (Accessed 27/10/2021). [Publication appraisal]

(42) Rosumeck S, Wagner M, Wallraf S, Euler U. A validation study revealed differences in design and performance of search filters for qualitative research in PsycINFO and CINAHL. J Clin Epidemiol. 2020 Dec;128:101-108. doi: 10.1016/j.jclinepi.2020.09.031. Epub 2020 Sep 26. [Publication appraisal]

(43) McKibbon KA, Wilczynski NL, Haynes RB. Developing optimal search strategies for retrieving qualitative studies in PsycINFO. Eval Health Prof, 29 (4) (2006), pp. 440-454. [Publication appraisal]

(44) Wilczynski NL, Marks S, Haynes RB. Search strategies for identifying qualitative studies in CINAHL. Qual Health Res, 17 (2007), 705-710. [Publication appraisal]

(45) Flemming K, Briggs M. Electronic searching to locate qualitative research: evaluation of three strategies. J Adv Nurs 2007, 57. [Publication appraisal]

(46) Benoot C, Hannes K, Bilsen J. The use of purposeful sampling in a qualitative evidence synthesis: A worked example on sexual adjustment to a cancer trajectory. BMC Med Res Methodol 2016, 16:21. [Publication appraisal]

(47) Booth A. Cochrane or cock-eyed? How should we conduct systematic reviews of qualitative research? In: Qualitative Evidence-based Practice Conference, Taking a Critical Stance. Coventry University; 2001. [Publication appraisal]

(48) Carroll C, Booth A, Lloyd-Jones M. Should we exclude inadequately reported studies from qualitative systematic reviews? An evaluation of sensitivity analyses in two case study reviews. Qual Health Res. 2012 Oct;22(10):1425-34. [Publication appraisal]

(49) France, E.F., Cunningham, M., Ring, N. et al. Improving reporting of meta-ethnography: the eMERGe reporting guidance. BMC Med Res Methodol 19, 25 (2019). 10.1186/s12874-018-0600-0 [Publication appraisal]

(40) Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: meta-narrative reviews. BMC Medicine 2013, 11(1):20. [Publication appraisal]

(51) Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: realist syntheses. BMC Medicine 2013, 11(1):21. [Publication appraisal]

(52) Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol 2012, 12(1):181. [Publication appraisal]

(53) Page, M.J., McKenzie, J.E., Bossuyt, P.M. et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Syst Rev 10, 89 (2021). 10.1186/s13643-021-01626-4 [Publication appraisal]

(54) Booth A. "Brimful of STARLITE": toward standards for reporting literature searches. J Med Libr Assoc 2006, 94. [Publication appraisal]

(55) Craven J, Levay P. Recording database searches for systematic reviews - What is the value of adding a narrative to peer-review checklists? A case study of NICE interventional procedures guidance. Evidence Based Library and Information Practice 2011, 6(4):72-87. [Publication appraisal]

(56) Cooper C, Dawson S, Peters J, Varley-Campbell J, Cockcroft E, Hendon J, Churchill R. Revisiting the need for a literature search narrative: A brief methodological note. Res Synth Methods 2018 Sep;9(3):361-365. [Publication appraisal]