DeJean 2016

Appraisal of: DeJean D, Giacomini M, Simeonov D, Smith A. Finding Qualitative Research Evidence for Health Technology Assessment. Qualitative Health Research, 2016; 26(10):1307-1317. 

Reviewer(s): 

Andrew Booth

Full Reference:
DeJean D, Giacomini M, Simeonov D, Smith A. Finding Qualitative Research Evidence for Health Technology Assessment. Qualitative Health Research, 2016; 26(10):1307-1317

Short description: 

This methodological study evaluates the performance of published search filters designed to identify qualitative research in bibliographic databases for health technology assessment purposes. The authors systematically searched three databases (MEDLINE, CINAHL, and Social Science Citation Index) using published search filters and their newly developed hybrid filter to identify qualitative research on two health topics: chronic obstructive pulmonary disease (COPD) and early breast cancer, published between 2005-2010. They identified 13,553 references for COPD and 13,556 for early breast cancer, of which only approximately 1% were truly qualitative research. The study assessed each filter's sensitivity, specificity, and precision by comparing retrieved references against a gold standard set determined through manual title and abstract screening by two independent reviewers. 

The authors found that published filters performed inconsistently across topics and databases, with particularly poor performance in MEDLINE searches. Their hybrid filter, which combined search terms from all published filters and excluded non-specific terms, demonstrated consistently high sensitivity across databases and topics while reducing false positives by approximately 50% compared to using all published filters separately. The study concludes that filter performance varies substantially by topic and database, and recommends their hybrid filter as a more efficient alternative. Additionally, the authors advocate for improved indexing and standardized reporting in qualitative health research to enhance discoverability in evidence syntheses.

Limitations stated by the author(s): 

The authors acknowledge that their evaluation of the hybrid filter was limited to only two topic areas (COPD and early breast cancer), which restricts generalizability. They note that further evaluation across additional health topics would be beneficial for understanding the properties and performance variations of their hybrid filter. The authors also mention that existing filters, including their hybrid filter, are designed to capture primary qualitative studies, and that a separate project is needed to develop search filters specifically for identifying evidence syntheses and secondary analyses of qualitative research. Additionally, the study was conducted within the context of health technology assessment with its specific time constraints (approximately 16 weeks for review completion), which may not reflect the needs or resources of other systematic review contexts.

Limitations stated by the reviewer(s): 

1. Limited scope of validation [External Validity]: The hybrid filter was tested on only two health topics, both in relatively well-researched clinical areas. The performance may differ substantially in emerging health topics, health services research, implementation science, or topics where qualitative research uses different terminology. Testing across a broader range of topics, including those in public health, health policy, and health equity, would strengthen claims of consistent performance.

2. Single time period [Temporal Validity]: The search was restricted to publications from 2005-2010. Qualitative research methodology, terminology, and indexing practices have evolved considerably over the past decade. The filter's performance may differ with more recent literature, particularly as newer qualitative approaches and mixed methods designs have become more prevalent and as database indexing has potentially improved.

3. Gold standard determination bias [Methodological Limitation]: While two reviewers independently screened titles and abstracts, the determination of what constituted "qualitative research" relied on their judgment and what could be discerned from titles and abstracts alone. Some studies may have been misclassified, particularly mixed methods studies where the qualitative component may not be prominent in the abstract. Full-text review of all potentially relevant articles would have been more rigorous but was acknowledged as impractical.

4. Database interface and version specificity [Reproducibility Concern]: The filters were tested using specific database interfaces (OVID for MEDLINE, EBSCOhost for CINAHL, Web of Science for SSCI) at a particular point in time. Database interfaces, indexing, and search algorithms change over time, which may affect filter performance and reproducibility. The study does not account for how interface updates or changes in database architecture might impact results.

5. Limited statistical analysis [Analytical Limitation]: The study reports descriptive statistics (sensitivity, specificity, precision) but does not provide confidence intervals or statistical tests to determine whether observed differences in filter performance are statistically significant. This makes it difficult to assess whether performance differences across filters, topics, or databases are meaningful or within the range of chance variation.

6. Exclusion of grey literature and non-English sources [Comprehensiveness]: The study focused on published literature indexed in three major databases but did not address strategies for finding qualitative research in grey literature, dissertations, or non-English language publications, which are important sources in comprehensive systematic reviews.

7. Resource implications not fully explored [Practical Application]: While the authors note that the hybrid filter reduced the number of false positives compared to using all published filters, they do not provide detailed time or cost analyses comparing different search strategies. The practical trade-offs between sensitivity and screening burden for time-limited health technology assessments could have been more thoroughly examined.

8. Lack of end-user validation [Usability]: The study does not report on whether information specialists or systematic reviewers outside the research team tested the hybrid filter or provided feedback on its usability, feasibility, or performance in real-world systematic review contexts. Prospective validation by independent teams would strengthen confidence in the filter's generalizability.

Study Type: 

Methodological study / Search filter validation study

Related Chapters: 

• Qualitative research

• Search filters

Tags: 

• Search filters

• Search strategies

• Databases

• MEDLINE

• CINAHL

• Social Science Citation Index

• Qualitative research synthesis

• Health technology assessment

• Sensitivity and specificity

• Systematic review methods

• Information retrieval

• Filter validation

• Hybrid filter development

• Evidence synthesis methods

• B. Designing strategies - general