Holland: appraisal
This appraisal is for Holland JL, Wilczynski NL, Haynes RB. Optimal search strategies for identifying sound clinical prediction studies in EMBASE. BMC Medical Informatics and Decision Making 2005;5:11.
This appraisal was prepared by Anne Fry-Smith
Information and Methodological Issues
Categorisation Issues
Detailed information, as appropriate
A. Information
A.1 State the author's objective
To describe the information retrieval properties of single terms and combinations of terms for identifying methodologically sound studies of clinical prediction guides in EMBASE.
A.2 State the focus of the search
[] Sensitivity-maximising
[] Precision-maximising
[] Specificity-maximising
[x] Balance of sensitivity and specificity / precision
[] Other
A.3. Database(s) and search interface(s).
Embase (Ovid)
A.4.Describe the methodological focus of the filter (e.g. RCTs).
Clinical prediction guide studies.
A.5 Describe any other topic that forms an additional focus of the filter (e.g. clinical topics such as breast cancer, geographic location such as Asia or population grouping such as paediatrics).
All topics
A.6 Other obervations
B. Identification of a gold standard (GS) of known relevant records
B. 1 Did the authors identify one or more gold standards (GSs)?nown relevant records
One
B.2 How did the authors identify the records in each GS? wn relevant records
6 research staff hand searched 55 journal titles for the year 2000.
B.3 Report the dates of the records in each GS. wn relevant records
2000
B.4 What are the inclusion criteria for each GS? relevant records
Having content that pertains directly to the prediction of some aspect of a disease or condition and the following methodologic criteria were applied: 1) the guide is generated in one or more sets of real patients (training set) and 2) the guide is validated in another set of real patients (test set).
B.5 Describe the size of each GS and the authors’ justification, if provided (for example the size of the gold standard may have been determined by a power calculation) antcords
163 records of which 69 were methodologically sound.
B.6 Are there limitations to the gold standard(s)? ntcords
Yes
It contains records only published in 1 year.
B.7 How was each gold standard used? cords
[x] to identify potential search terms
[] to derive potential strategies (groups of terms)
[x] to test internal validity
[] to test external validity
[ ] other, please specify
B.8 Other observations. cords
C. How did the researchers identify the search terms in their filter(s) (select all that apply)?
C.1 Adapted a published search strategy.
No
C.2 Asked experts for suggestions of relevant terms.
Yes
C.3 Used a database thesaurus.
No
C.4 Statistical analysis of terms in a gold standard set of records (see B above).
No
C.5 Extracted terms from the gold standard set of records (see B above).
No
C.6 Extracted terms from some relevant records (but not a gold standard).
No
C.7 Tick all types of search terms tested.
[x] subject headings
[x] text words (e.g. in title, abstract)
[ ] publication types
[ ] subheadings
[ ] check tags
[ ] other, please specify
C.8 Include the citation of any adapted strategies.
C.9 How were the (final) combination(s) of search terms selected?
The sensitivity precision and accuracy of each term was determined against the gold standard. The best single term, 2-term and multiple term strategies that would optimise sensitivity, specificity or both were identified.
Sensitivity optimizing strategies were determined by testing all 2-term search strategies with sensitivity of 75% or more and specificity of 50% or greater.
Accuracy optimising 2-term strategies were identified by testing all 2-term strategies with accuracy greater than 7%. 36,232
strategies were tested.
C.10 Were the search terms combined (using Boolean logic) in a way that is likely to retrieve the studies of interest?
Search terms were combined with OR.
C.11 Other observations.
An initial list of index terms and textwords relating to the studies was compiled in house. No further details are given.
D. Internal validity testing (This type of testing is possible when the search filter terms were developed from a known gold standard set of records).
D.1 How many filters were tested for internal validity? cords).
Six
A: Single term best sensitivity.
B: Single term best specificity.
C: Single term best optimization of sensitivity & specificity.
D: Combination of terms best sensitivity.
E: Combination of terms best specificity.
F: Combination of terms best optimization of sensitivity & specificity.
D.2 Was the performance of the search filter tested on the gold standard from which it was derived?ds).
Yes
D.3 Report sensitivity data (a single value, a range, ‘Unclear’* or ‘not reported’, as appropriate). *Please describe. ds).
A: 78.3% (95% CI: 68.5 to 88.0)
B: 60.9% (95% CI: 49.4 to 72.4)
C: 78.3% (95% CI: 68.5 to 88.0)
D: 97.1% (95% CI: 93.1 to 100.0)
E: 60.9% (95% CI: 49.4 to 72.4)
F: 91.3% (95% CI: 84.7 to 98.0)
D.4 Report precision data (a single value, a range, ‘Unclear’* or ‘not reported’ as appropriate). *Please describe. ).
A: 2.3% (95% CI:1.7 to 2.9)
B: 10.7% (95% CI: 7.6 to 13.7)
C: 2.3% (95% CI: 1.7 to 2.9)
D: 0.9% (95% CI: 0.7 to 1.2)
E: 10.8% (95% CI: 7.7 to 13.9)
F: 2.3% (95% CI: 1.7 to 2.8)
D.5 Report specificity data (a single value, a range, ‘Unclear’* or ‘not reported’ as appropriate). *Please describe
A: 91.6% (95% CI: 91.2 to 91.9)
B: 98.7% (95% CI: 98.6 to 98.9)
C: 91.6% (95% CI: 91.2 to 91.9)
D: 74.2% (95% CI: 73.7 to 74.7)
E: 98.8% (95% CI: 98.6 to 98.9)
F: 90.2% (95% CI: 89.9 to 90.6)
D.6 Other performance measures reported.
Accuracy
A: 91.5% (95% CI: 91.2 to 91.7)
B: 98.6% (95% CI: 98.5 to 98.8)
C: 91.5% (95% CI: 91.2 to 91.7)
D: 74.2% (95% CI: 73.7 to 74.7)
E: 98.7% (95% CI: 98.5 to 98.8)
F: 90.2% (95% CI: 89.9 to 90.6)
D.7 Other observations.
Strategies were derived from a small subset of gold standard.
E. External validity testing (This section relates to testing the search filter on records that are different from the records used to identify the search terms).
E.1 How many filters were tested for external validity on records different from those used to identify the search terms?
Since previous studies on developing strategies for other categories of articles found that the comparison between development and validation database results was not statistically significant and also because there were too few pass articles in the gold standard, the search strategies were developed using the entire gold standard database.
E.2 Describe the validation set(s) of records, including the interface.
For each filter report the following information.
E.3 On which validation set(s) was the filter tested?
E.4 Report sensitivity data for each validation set (a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
E.5 Report precision data for each validation set (report a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
E.6 Report specificity data for each validation set (a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
E.7 Other performance measures reported.
E.8 Other observations
F. Limitations and Comparisons
F.1 Did the authors discuss any limitations to their research?
Yes
F.2 Are there other potential limitations to this research that you have noticed?
No
F.3 Report any comparisons of the performance of the filter against other relevant published filters (sensitivity, precision, specificity or other measures).
F.4 Include the citations of any compared filters.
F.5 Other observations and / or comments.
G. Other comments. This section can be used to provide any other comments. Selected prompts for issues to bear in mind are given below.
G.1 Have you noticed any errors in the document that might impact on the usability of the filter?
No
G.2 Are there any published errata or comments (for example in the MEDLINE record)?
No
G.3 Is there public access to pre-publication history and / or correspondence?
See:http://www.biomedcentral.com/1472-6947/5/11/prepub
G.4 Are further data available on a linked site or from the authors?
No
G.5 Include references to related papers and/or other relevant material.
G.6. Other comments