Montori: appraisal
This appraisal is for Montori VM, Wilczynski NL, Morgan D, Haynes RB. Optimal search strategies for retrieving systematic reviews from MEDLINE: analytical survey. BMJ 2005;330(7482):68.
This appraisal was prepared by Sue Bayliss
Information and Methodological Issues
Categorisation Issues
Detailed information, as appropriate
A. Information
A.1 State the author's objective
To develop optimal search strategies for retrieving systematic reviews from
MEDLINE.
A.2 State the focus of the search
[x] Sensitivity-maximising
[x] Precision-maximising
[x] Specificity-maximising
[x] Balance of sensitivity and specificity / precision
[] Other
A.3. Database(s) and search interface(s).
MEDLINE (Ovid)
A.4.Describe the methodological focus of the filter (e.g. RCTs).
Systematic reviews
A.5 Describe any other topic that forms an additional focus of the filter (e.g. clinical topics such as breast cancer, geographic location such as Asia or population grouping such as paediatrics).
All topics
A.6 Other obervations
B. Identification of a gold standard (GS) of known relevant records
B. 1 Did the authors identify one or more gold standards (GSs)?nown relevant records
One
B.2 How did the authors identify the records in each GS? wn relevant records
Handsearch of 161 selected journals indexed in MEDLINE (Ovid).
B.3 Report the dates of the records in each GS. wn relevant records
2000
B.4 What are the inclusion criteria for each GS? relevant records
Systematic reviews using explicit search strategy, inclusion and exclusion criteria and looking at at least one primary study.
B.5 Describe the size of each GS and the authors’ justification, if provided (for example the size of the gold standard may have been determined by a power calculation) antcords
133 systematic reviews, from 10446 records.
B.6 Are there limitations to the gold standard(s)? ntcords
Yes
Limited to year 2000 only.
B.7 How was each gold standard used? cords
[x] to identify potential search terms
[] to derive potential strategies (groups of terms)
[x] to test internal validity
[x] to test external validity
[ ] other, please specify
B.8 Other observations. cords
C. How did the researchers identify the search terms in their filter(s) (select all that apply)?
C.1 Adapted a published search strategy.
Yes
Used published strategies from other groups.
C.2 Asked experts for suggestions of relevant terms.
Yes
Librarians and clinicians.
C.3 Used a database thesaurus.
Unclear
Indexing terms and text words (4862 terms).
C.4 Statistical analysis of terms in a gold standard set of records (see B above).
No
C.5 Extracted terms from the gold standard set of records (see B above).
Unclear
C.6 Extracted terms from some relevant records (but not a gold standard).
Unclear
C.7 Tick all types of search terms tested.
[x] subject headings
[x] text words (e.g. in title, abstract)
[x] publication types
[x] subheadings
[ ] check tags
[ ] other, please specify
C.8 Include the citation of any adapted strategies.
C.9 How were the (final) combination(s) of search terms selected?
Search terms with a sensitivity of more than 50% were selected to develop strategies that optimised sensitivity and with a specificity of more than 75% to develop strategies that optimised specificity.
C.10 Were the search terms combined (using Boolean logic) in a way that is likely to retrieve the studies of interest?
All combinations of terms used the Boolean OR in strings of one to five terms.
C.11 Other observations.
Search terms were tested using Ovid Technologies searching system.
D. Internal validity testing (This type of testing is possible when the search filter terms were developed from a known gold standard set of records).
D.1 How many filters were tested for internal validity? cords).
Four
A - Top strategy maximising sensitivity.
B - Top strategy minimising difference between sensitivity and specificity.
C - Top strategy maximising precision.
D - Top strategy combining most precise term with most sensitive terms.
D.2 Was the performance of the search filter tested on the gold standard from which it was derived?ds).
No
D.3 Report sensitivity data (a single value, a range, ‘Unclear’* or ‘not reported’, as appropriate). *Please describe. ds).
A - 100%
B - 92.5%
C - 75.2%
D - 90.2%
D.4 Report precision data (a single value, a range, ‘Unclear’* or ‘not reported’ as appropriate). *Please describe. ).
A - 3.41%
B - 14.6%
C - 60.2%
D - 46.5%
D.5 Report specificity data (a single value, a range, ‘Unclear’* or ‘not reported’ as appropriate). *Please describe
Not reported
A - 63.5%
B - 93.0%
C - 99.4%
D - 98.4%
D.6 Other performance measures reported.
D.7 Other observations.
Strategies were derived from a small subset of gold standard.
E. External validity testing (This section relates to testing the search filter on records that are different from the records used to identify the search terms).
E.1 How many filters were tested for external validity on records different from those used to identify the search terms?
Four
A - Top strategy maximising sensitivity.
B - Top strategy minimising difference between sensitivity and specificity.
C - Top strategy maximising precision.
D - Top strategy combining most precise term with most sensitive terms.
E.2 Describe the validation set(s) of records, including the interface.
2 databases of records from 161 high yield journals:
Validation set including CDSR (Cochrane Database of Systematic Reviews) and a validation set without CDSR
For each filter report the following information.
E.3 On which validation set(s) was the filter tested?
Set 1: Validation set including CDSR (Cochrane Database of Systematic Reviews)
Set 2: Validation set without CDSR
E.4 Report sensitivity data for each validation set (a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
Set 1:
A - 99.9%
B - 98.0%
C - 71.2%
Set 2:
A - 99.7%
B - 95.5%
C - 74.4%
Complete validation set: D – 90.2%
E.5 Report precision data for each validation set (report a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
Set 1:
A - 3.14%
B - 14.2%
C - 57.1%
Set 2:
A - 1.4%
B - 6.1%
C - 26.3%
Complete validation set: D - 46.5%
E.6 Report specificity data for each validation set (a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
Set 1:
A - 52.0%
B - 90.8%
C - 99.2%
Set 2:
A - 51.1%
B - 89.9%
C - 98.6%
Complete validation set: D - 98.4%
E.7 Other performance measures reported.
E.8 Other observations
F. Limitations and Comparisons
F.1 Did the authors discuss any limitations to their research?
Yes
Strategies were a derived from small subset of the database. Strategy generation was limited by using Boolean OR to add terms, whereas AND and NOT operators may have resulted in more restricted strategies with better performance.
F.2 Are there other potential limitations to this research that you have noticed?
F.3 Report any comparisons of the performance of the filter against other relevant published filters (sensitivity, precision, specificity or other measures).
Figures given in order of Sensitivity, Specificity, Precision
CRD High Sensitivity filter 97.6%(sens) 69.6%(spec) 4.77%(prec)
CRD intermediate sensitivity/precision filter 96.7% 79.7% 6.91%
CRD high sensitivity and precision filter 95.8% 89.7% 12.7%
Hunt and McKibbon simple query filter 68.8% 99.2% 56.7%
Hunt and McKibbon sensitive query filter 73.4% 99.1% 55.1%
Shojania and Bero PubMed based query 90.0% 97.2% 33.2%
Hedges (this report) Sensitive 5 term filter 99.9% 52.0% 3.14%
Balanced sensitivity specificity 3 term filter 98.0% 90.8% 14.2%
Balanced specificity sensitivity 5 term filter 90.2% 98.4% 46.5%
Specific 3 term filter 71.2% 99.2% 57.1%
F.4 Include the citations of any compared filters.
See references 4, 5 and 6
F.5 Other observations and / or comments.
G. Other comments. This section can be used to provide any other comments. Selected prompts for issues to bear in mind are given below.
G.1 Have you noticed any errors in the document that might impact on the usability of the filter?
G.2 Are there any published errata or comments (for example in the MEDLINE record)?
No
G.3 Is there public access to pre-publication history and / or correspondence?
G.4 Are further data available on a linked site or from the authors?
Yes
A table showing PubMed translations of the Ovid strategy is available at: http://bmj.com/cgi/content/full/bmj.38336.804167.47/DC1
G.5 Include references to related papers and/or other relevant material.
G.6. Other comments