Shojania appraisal
This appraisal is for Shojania KG, Bero LA. Taking advantage of the explosion of systematic reviews: an efficient MEDLINE search strategy. Effective Clinical Practice 2001;4(4):157-62.
This appraisal was prepared by Sue Bayliss
Information and Methodological Issues
Categorisation Issues
Detailed information, as appropriate
A. Information
A.1 State the author's objective
To develop and evaluate a search strategy for identifying systematic reviews by using a publicly available MEDLINE interface (PubMed).
A.2 State the focus of the search
[x] Sensitivity-maximising
[x] Precision-maximising
[ ] Specificity-maximising
[x] Balance of sensitivity and specificity / precision
[ ] Other
A.3. Database(s) and search interface(s).
MEDLINE (PubMed).
A.4.Describe the methodological focus of the filter (e.g. RCTs).
Systematic reviews
A.5 Describe any other topic that forms an additional focus of the filter (e.g. clinical topics such as breast cancer, geographic location such as Asia or population grouping such as paediatrics).
A.6 Other obervations
B. Identification of a gold standard (GS) of known relevant records
B. 1 Did the authors identify one or more gold standards (GSs)?nown relevant records
2
B.2 How did the authors identify the records in each GS? wn relevant records
Gold standard 1 (GS1): First 100 records in DARE (all systematic reviews) also indexed in MEDLINE.
Gold standard 2 (GS2): Handsearch of ACP Journal Club for systematic reviews back to first issue (103) also indexed in MEDLINE.
B.3 Report the dates of the records in each GS. wn relevant records
GS1: not specified. By “first 100” do the authors mean from earliest date or most recent date?
GS2: 1999-2000
B.4 What are the inclusion criteria for each GS? relevant records
GS1: The authors are relying on the inclusion exclusion criteria of DARE (quality assessed systematic reviews only).
GS2: The authors are relying on the inclusion exclusion criteria of ACP Journal Club (“must contain an identifiable description of the methods indicating sources and methods for searching for articles and state the clinical topic and the inclusion and exclusion criteria for selecting articles for detailed review”).
B.5 Describe the size of each GS and the authors’ justification, if provided (for example the size of the gold standard may have been determined by a power calculation) antcords
GS1: 100
GS2: 104
B.6 Are there limitations to the gold standard(s)? ntcords
Yes
Only 2 sources were used and quite small numbers of records are involved.
B.7 How was each gold standard used? cords
[ ] to identify potential search terms
[ ] to derive potential strategies (groups of terms)
[x] to test internal validity
[ ] to test external validity
[ ] other, please specify
Unclear whether terms/strategies were derived from the gold standard(s).
B.8 Other observations. cords
C. How did the researchers identify the search terms in their filter(s) (select all that apply)?
C.1 Adapted a published search strategy.
Unclear
Not clear how the authors did this. Terms may have been derived from the GS but the paper doesn’t say this explicitly.
C.2 Asked experts for suggestions of relevant terms.
Unclear
C.3 Used a database thesaurus.
Unclear
C.4 Statistical analysis of terms in a gold standard set of records (see B above).
Unclear
C.5 Extracted terms from the gold standard set of records (see B above).
Unclear
C.6 Extracted terms from some relevant records (but not a gold standard).
Unclear
C.7 Tick all types of search terms tested.
[ ] subject headings
[x] text words (e.g. in title, abstract)
[x] publication types
[ ] subheadings
[ ] check tags
[ ] other, please specify
Terms specifically in the title were included.
C.8 Include the citation of any adapted strategies.
C.9 How were the (final) combination(s) of search terms selected?
The paper is not very explicit. It suggests that the authors used relevant publication types as well as title and text words “typically found in systematic reviews”, but not exactly how they decided which were most appropriate.
C.10 Were the search terms combined (using Boolean logic) in a way that is likely to retrieve the studies of interest?
Yes
This is not fully explained but can be deduced from the string itself.
C.11 Other observations.
Uses “BUT NOT” to exclude some publication types, but there is no mention of other operators in the methods description.
D. Internal validity testing (This type of testing is possible when the search filter terms were developed from a known gold standard set of records).
D.1 How many filters were tested for internal validity? cords).
1
D.2 Was the performance of the search filter tested on the gold standard from which it was derived?ds).
No
D.3 Report sensitivity data (a single value, a range, ‘Unclear’* or ‘not reported’, as appropriate). *Please describe. ds).
D.4 Report precision data (a single value, a range, ‘Unclear’* or ‘not reported’ as appropriate). *Please describe. ).
D.5 Report specificity data (a single value, a range, ‘Unclear’* or ‘not reported’ as appropriate). *Please describe. ).
D.6 Other performance measures reported.
D.7 Other observations.
E. External validity testing (This section relates to testing the search filter on records that are different from the records used to identify the search terms).
E.1 How many filters were tested for external validity on records different from those used to identify the search terms?
1
The filter’s sensitivity was then tested on its ability to retrieve the GS records from the MEDLINE database and its positive predictive value (precision) in terms of its ability to retrieve systematic reviews when used in combination with terms for three clinical topics.
E.2 Describe the validation set(s) of records, including the interface.
MEDLINE
For each filter report the following information.
E.3 On which validation set(s) was the filter tested?
Both
GS1
GS2
E.4 Report sensitivity data for each validation set (a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
GS1: 93% (95% CI 86% to 97%)
GS2: 97% (95% CI 91% to 99%)
E.5 Report precision data for each validation set (report a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
The precision (positive predictive value) of the strategy (50%) was calculated by applying the strategy to three clinical topics and then evaluating the results in terms of how many systematic reviews were located i.e. precision was not calculated for the gold standards.
E.6 Report specificity data for each validation set (a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
Not reported
E.7 Other performance measures reported.
E.8 Other observations
F. Limitations and Comparisons
F.1 Did the authors discuss any limitations to their research?
The broad definition of a systematic review that was used.
F.2 Are there other potential limitations to this research that you have noticed?
The relatively small size of the gold standards is a potential limitation along with the fact that the sources that the GSs are derived from focus on high-quality reviews. There is inconsistency in the way precision was measured outside of the gold standard, whereas sensitivity was measured against the gold standard.
F.3 Report any comparisons of the performance of the filter against other relevant published filters (sensitivity, precision, specificity or other measures).
Refers to the Boynton et al filter which had a sensitivity of 98% and a precision of 20%.
F.4 Include the citations of any compared filters.
Boynton J, Glanville JG, McDaid D, Lefebvre C Identifying systematic reviews in MEDLINE: developing and objective approach to search strategy design. Journal of Information Science 1998;24: 137-54.
F.5 Other observations and / or comments.
G. Other comments. This section can be used to provide any other comments. Selected prompts for issues to bear in mind are given below.
G.1 Have you noticed any errors in the document that might impact on the usability of the filter?
G.2 Are there any published errata or comments (for example in the MEDLINE record)?
G.3 Is there public access to pre-publication history and / or correspondence?
G.4 Are further data available on a linked site or from the authors?
G.5 Include references to related papers and/or other relevant material.
G.6. Other comments