Gorecki 2010

Appraisal of: Gorecki CA, Brown JM, Briggs M, Nixon J. Evaluation of five search strategies in retrieving qualitative patient-reported electronic data on the impact of pressure ulcers on quality of life. Journal of Advanced Nursing, 66(3), 645-652. doi:10.1111/j.1365-2648.2009.05192.x

Reviewer(s): 

Andrew Booth

Full Reference:
Gorecki CA, Brown JM, Briggs M, Nixon J. Evaluation of five search strategies in retrieving qualitative patient-reported electronic data on the impact of pressure ulcers on quality of life. Journal of Advanced Nursing, 66(3), 645-652

Short description: 

This study compared the effectiveness of qualitative methodology-based search strategies with subject-specific health-related quality of life search strategies for retrieving qualitative patient-reported data on the impact of pressure ulcers. The researchers evaluated five search strategies: three research methodology (qualitative) strategies developed by Shaw et al. (2004) including complex free-text terms, complex thesaurus terms, and simple broad-based terms; one complex subject-specific quality of life strategy combining existing validated strategies; and one refined patient-reported health-related quality of life strategy developed by the authors specifically for this systematic review. Each strategy was combined with pressure ulcer MeSH terms and searched across seven bibliographic databases including AMED, BNI, MEDLINE, EMBASE, PsycINFO, CINAHL, and the Cochrane Controlled Trials Register from inception to October 2007. The total yield across all strategies was 4,620 records after removing duplicates, from which 18 relevant studies were identified that reported qualitative patient-reported health-related quality of life data. Performance was assessed using sensitivity, specificity, precision, and accuracy measures. Results showed the subject-specific health-related quality of life strategy had 100% sensitivity, successfully identifying all 18 relevant studies, but relatively low specificity of 46%, meaning many irrelevant records were also retrieved. The methodology-based qualitative strategies had lower sensitivity ranging from 72-83% but higher specificity of 79-83%. Critically, the methodology-based strategies failed to identify studies reporting qualitative data within mixed methods designs. CINAHL demonstrated the best overall database coverage, identifying 15 of 18 relevant studies, while MEDLINE also identified 15 studies but retrieved different unique studies. The authors concluded that subject-specific search strategies are more effective for comprehensive retrieval of qualitative patient-reported health-related quality of life data regardless of study methodology, though this comes at the cost of reduced specificity and increased screening burden.

Limitations stated by the author(s): 

The authors acknowledge that the study included a small sample of only 18 relevant studies despite comprehensive searching methods, which limits the generalizability of findings. The patient-reported health-related quality of life search strategy was developed by comparing performance of search term combinations specifically for pressure ulcer research, meaning its validity needs to be established by replication in other medical areas. However, the authors note the strategy should be generalizable for use in other medical fields when the aim is to identify qualitative patient-reported health-related quality of life data, as it was adapted from existing validated strategies currently used in systematic searching. The authors suggest the strategy could be strengthened by reviewing non-relevant records retrieved to identify additional "but not" search terms that would reduce yield and improve specificity and precision. The study also relied on hand-searching relevant journals and reference lists to identify studies that might not have been retrieved electronically, which while strengthening the gold standard reduced bias but added considerable manual effort.

Limitations stated by the reviewer(s): 

Strengths:

The study demonstrates several notable methodological strengths. First, it addresses a practically important and under-researched question about how to effectively locate qualitative patient-reported quality of life data, which is increasingly recognized as valuable evidence but lacks established search methods. The comparison of five different search strategies provides valuable comparative data that can inform future systematic review practice. The use of multiple performance metrics including sensitivity, specificity, precision, and accuracy allows for a nuanced understanding of trade-offs between different approaches. The inclusion of seven databases provides comprehensive coverage across health sciences disciplines and allows for database-specific performance assessment. The development and testing of a novel patient-reported health-related quality of life search strategy represents an original contribution that addresses the specific challenge of locating qualitative data regardless of study methodology. The hand-searching of relevant journals and reference lists to create a gold standard set of relevant studies strengthens the validity of the evaluation by ensuring the reference set is as complete as possible. The authors demonstrate transparency in acknowledging the limitations of their approach and the need for further validation. The recognition that qualitative data appears in both qualitative and mixed methods studies is an important insight that previous methodology-based filters had not adequately addressed. The clear reporting of results by both search strategy and individual database allows readers to understand where coverage exists and where gaps remain.

Weaknesses:

Several methodological limitations warrant consideration. The very small sample size of only 18 relevant studies severely limits statistical power and the reliability of performance estimates, particularly for measures like specificity and precision where confidence intervals would likely be very wide. Sensitivity and specificity calculations based on such small numbers may not be stable or generalizable to other topics. The restriction to a single clinical topic (pressure ulcers) means findings may not transfer to other disease areas where indexing practices, terminology, or publication patterns differ. The gold standard was defined based on studies the researchers could identify through comprehensive methods, but there is no way to verify that all truly relevant studies were captured, meaning sensitivity calculations may be inflated if relevant studies exist that were missed by all methods. The study focuses exclusively on database coverage and does not examine actual retrieval efficiency in practice, including time costs of screening large numbers of irrelevant records when using high-sensitivity strategies. No assessment of inter-rater reliability was conducted for determining study relevance, introducing potential for inconsistent classification. The patient-reported health-related quality of life strategy was developed iteratively using the same dataset on which it was tested, which represents a form of circular reasoning and overfitting that likely inflates its apparent performance. True validation would require testing on an independent dataset from a different topic area. The authors searched databases only up to October 2007, and indexing practices and database coverage may have changed substantially since then, potentially limiting current applicability. The study does not examine whether the additional studies retrieved by more sensitive strategies actually contributed to changing systematic review conclusions or evidence strength, which is the ultimate measure of value. The complexity and number of search terms in different strategies are not clearly compared, making it difficult to assess the practical burden of implementing each approach. Publication bias may affect results since studies reporting qualitative patient data may be less likely to be published in indexed journals compared to quantitative outcome studies.

Study Type: 

Methodological study; Search strategy evaluation; Comparative study; Database coverage analysis

Related Chapters: 

• Qualitative research

• Search filters

Tags: 

• Search strategies

• Qualitative research

• Patient-reported outcomes

• Health-related quality of life

• Database searching

• Information retrieval

• Sensitivity and specificity

• CINAHL

• MEDLINE

• EMBASE

• PsycINFO

• Pressure ulcers

• Mixed methods research

• Search filters

• Evidence synthesis

• Systematic reviews

• Database coverage

• Methodology filters

• Subject-specific searching