This appraisal is for Wong SS, Wilczynski NL, Haynes RB. Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE. Medinfo 2004;11(1):311-6.
This appraisal was prepared by Julie Glanville
A. Information
A.1 State the author's objective
To develop optimal search strategies to identify clinically relevant qualitative studies in MEDLINE.
A.2 State the focus of the search
[ ] Sensitivity-maximising
[ ] Precision-maximising
[ ] Specificity-maximising
[x] Balance of sensitivity and specificity / precision
[ ] Other
A.3. Database(s) and search interface(s).
MEDLINE (Ovid)
A.4.Describe the methodological focus of the filter (e.g. RCTs).
Qualitative research
A.5 Describe any other topic that forms an additional focus of the filter (e.g. clinical topics such as breast cancer, geographic location such as Asia or population grouping such as paediatrics).
All topics
A.6 Other obervations
B. Identification of a gold standard (GS) of known relevant records
B. 1 Did the authors identify one or more gold standards (GSs)?nown relevant records
One
B.2 How did the authors identify the records in each GS? wn relevant records
6 people handsearched 161 core health journals published in one year.
B.3 Report the dates of the records in each GS. wn relevant records
2000
B.4 What are the inclusion criteria for each GS? relevant records
Original studies or reviews of interest to human health care with content relating to people’s experiences, and where data were collected or analysed in ways appropriate to qualitative data.
B.5 Describe the size of each GS and the authors’ justification, if provided (for example the size of the gold standard may have been determined by a power calculation) antcords
366 records.
B.6 Are there limitations to the gold standard(s)? ntcords
Yes
It contains records published in only one year. The authors describe the journal selection, but don’t justify their selection or discuss generalisability.
B.7 How was each gold standard used? cords
[ ] to identify potential search terms
[ ] to derive potential strategies (groups of terms)
[x] to test internal validity
[x] to test external validity
[ ] other, please specify
B.8 Other observations. cords
C. How did the researchers identify the search terms in their filter(s) (select all that apply)?
C.1 Adapted a published search strategy.
No
C.2 Asked experts for suggestions of relevant terms.
Yes
Suggestions were sought from clinicians and librarians in the USA and Canada.
C.3 Used a database thesaurus.
No
C.4 Statistical analysis of terms in a gold standard set of records (see B above).
No
C.5 Extracted terms from the gold standard set of records (see B above).
No
C.6 Extracted terms from some relevant records (but not a gold standard).
Yes
C.7 Tick all types of search terms tested.
[x] subject headings
[x] text words (e.g. in title, abstract)
[ ] publication types
[ ] subheadings
[ ] check tags
[ ] other, please specify
C.8 Include the citation of any adapted strategies.
C.9 How were the (final) combination(s) of search terms selected?
The performance of search terms was tested against a random 60% of the gold standard. Terms received sensitivity and specificity scores and single terms with best sensitivity, best specificity and best optimization of sensitivity and specificity were identified. 2 and 3 term search strategies are presented but the exact method of derivation is not stated.
C.10 Were the search terms combined (using Boolean logic) in a way that is likely to retrieve the studies of interest?
The terms were combined using the OR operator and this approach means the strategies are likely to retrieve relevant studies.
C.11 Other observations.
The full list of candidate search terms is not supplied.
D. Internal validity testing (This type of testing is possible when the search filter terms were developed from a known gold standard set of records).
D.1 How many filters were tested for internal validity? cords).
Five
5 filter results are presented
A -Best sensitivity
B -Best sensitivity with more specificity
C -Best specificity
D -Best specificity with more sensitivity
E -Best optimization of sensitivity and specificity.
D.2 Was the performance of the search filter tested on the gold standard from which it was derived?ds).
The filters were derived and tested on 60% of the gold standard.
D.3 Report sensitivity data (a single value, a range, ‘Unclear’* or ‘not reported’, as appropriate). *Please describe. ds).
A 95.05%
B 93.96%
C 60.99%
D 84.07%
E 92.31%
D.4 Report precision data (a single value, a range, ‘Unclear’* or ‘not reported’ as appropriate). *Please describe. ).
A 1.93%
B 4.96%
C 37.37%
D 19.42%
E 6.69%
D.5 Report specificity data (a single value, a range, ‘Unclear’* or ‘not reported’ as appropriate). *Please describe. ).
A 69.98%
B 88.78%
C 99.36%
D 97.83%
E 91.98%
D.6 Other performance measures reported.
Accuracy
A 70.13%
B 88.81%
C 99.13%
D 97.74%
E 91.98%
D.7 Other observations.
E. External validity testing (This section relates to testing the search filter on records that are different from the records used to identify the search terms).
E.1 How many filters were tested for external validity on records different from those used to identify the search terms?
5 filter results are presented
A -Best sensitivity
B -Best sensitivity with more specificity
C -Best specificity
D -Best specificity with more sensitivity
E -Best optimization of sensitivity and specificity
E.2 Describe the validation set(s) of records, including the interface.
The remaining 40% of the gold standard.
For each filter report the following information.
E.3 On which validation set(s) was the filter tested?
All 5 filters were tested on the validation set.
E.4 Report sensitivity data for each validation set (a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
A 92.47%
B 91.10%
C 53.42%
D 78.77%
E 86.99%
E.5 Report precision data for each validation set (report a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
A 2.25%
B 5.93%
C 39.59%
D 21.62%
E 7.53%
E.6 Report specificity data for each validation set (a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
A 69.94%
B 89.17%
C 99.39%
D 97.86%
E 92.00%
E.7 Other performance measures reported.
Accuracy
A 70.10%
B 89.18%
C 99.05%
D 97.72%
E 91.96%
E.8 Other observations
The difference and confidence intervals around the difference between the performance in test and validation sets are presented. None of the differences were statistically significant.
F. Limitations and Comparisons
F.1 Did the authors discuss any limitations to their research?
The filters do not filter higher quality from lower quality studies.
F.2 Are there other potential limitations to this research that you have noticed?
F.3 Report any comparisons of the performance of the filter against other relevant published filters (sensitivity, precision, specificity or other measures).
F.4 Include the citations of any compared filters.
F.5 Other observations and / or comments.
G. Other comments. This section can be used to provide any other comments. Selected prompts for issues to bear in mind are given below.
G.1 Have you noticed any errors in the document that might impact on the usability of the filter?
No
G.2 Are there any published errata or comments (for example in the MEDLINE record)?
No
G.3 Is there public access to pre-publication history and / or correspondence?
No
G.4 Are further data available on a linked site or from the authors?
G.5 Include references to related papers and/or other relevant material.
Wilczynski NL et al. Enhancing retrieval of best evidence for health care from bibliographic databases: calibration of the hand search literature. Medinfo 2001;10(1):390-3.
G.6. Other comments