Bachmann JAMIA filter: appraisal
This appraisal is for Bachmann LM, Coray R, Estermann P, Ter Riet G. Identifying diagnostic studies in MEDLINE: reducing the number needed to read. JAMIA 2002;9(6):653-8.
This appraisal was prepared by Cynthia Fraser
Information and Methodological Issues
Categorisation Issues
Detailed information, as appropriate
A. Information
A.1 State the author's objective
To construct and validate search strategies to identify diagnostic articles recorded on MEDLINE with special emphasis on precision (without loss of sensitivity).
A.2 State the focus of the search
[] Sensitivity-maximising
[ ] Precision-maximising
[ ] Specificity-maximising
[ ] Balance of sensitivity and specificity / precision
[ x] Other
Precision without losing sensitivity.
A.3. Database(s) and search interface(s).
MEDLINE (Datastar) – to identify gold standard set in MEDLINE.
Unclear which interface used in the rest of analysis due to inconsistent use of truncation symbol (“*” for search terms but “$” for Datastar strategy in Table 3).
A.4.Describe the methodological focus of the filter (e.g. RCTs).
Diagnostic studies
A.5 Describe any other topic that forms an additional focus of the filter (e.g. clinical topics such as breast cancer, geographic location such as Asia or population grouping such as paediatrics).
None
A.6 Other obervations
B. Identification of a gold standard (GS) of known relevant records
B. 1 Did the authors identify one or more gold standards (GSs)?nown relevant records
None
GS for construction of filter (identified for this appraisal: GScon89). Two GSs for validation (identified for this appraisal: GSval94 GSval99).
B.2 How did the authors identify the records in each GS? wn relevant records
GScon89: Contents of four medical journals for year 1989 hand-searched independently by 2 researchers.
GSval94 and GSval99: Contents of same four medical journal for 1994 and four different journals for 1999 handsearched.
B.3 Report the dates of the records in each GS. wn relevant records
GS for construction 1989.
GSs for validation 1994 and 1999.
B.4 What are the inclusion criteria for each GS? relevant records
Diagnostic studies - “evaluation of disease process usually through comparing methods of arriving at a diagnosis”.
B.5 Describe the size of each GS and the authors’ justification, if provided (for example the size of the gold standard may have been determined by a power calculation) antcords
Gscon89: 83
GSval94: 53
GSval99: 61
B.6 Are there limitations to the gold standard(s)? ntcords
GS sets are very small.
B.7 How was each gold standard used? cords
[x ] to identify potential search terms (Gsecon89)
[ ] to derive potential strategies (groups of terms)
[x ] to test internal validity (GScon89)
[x ] to test external validity (GSval94 and GSval99)
[ ] other, please specify
B.8 Other observations. cords
C. How did the researchers identify the search terms in their filter(s) (select all that apply)?
C.1 Adapted a published search strategy.
No
C.2 Asked experts for suggestions of relevant terms.
No
C.3 Used a database thesaurus.
No
C.4 Statistical analysis of terms in a gold standard set of records (see B above).
Yes
Word frequency analysis using Idealist software of words in titles, abstracts and subject indexes of MEDLINE records of Gscon89.
Terms were examined by two researchers and excluded if not semantically associated with diagnosis. Truncated terms were used when more than one variation in a word identified.
C.5 Extracted terms from the gold standard set of records (see B above).
No
C.6 Extracted terms from some relevant records (but not a gold standard).
No
C.7 Tick all types of search terms tested.
[x] subject headings
[x] text words (e.g. in title, abstract)
[] publication types
[ ] subheadings
[ ] check tags
[ ] other, please specify
C.8 Include the citation of any adapted strategies.
C.9 How were the (final) combination(s) of search terms selected?
Sensitivity and precision of 20 most prevalent terms derived against GScon89. Sensitivity x Precision for each terms calculated and highest combined (using OR) in a stepwise fashion with term Exp “Sensitivity and Specificity”/. Optimal strategies had highest sensitivity in combination with precision calculated against GScon89.
C.10 Were the search terms combined (using Boolean logic) in a way that is likely to retrieve the studies of interest?
Yes
C.11 Other observations.
D. Internal validity testing (This type of testing is possible when the search filter terms were developed from a known gold standard set of records).
D.1 How many filters were tested for internal validity? cords).
Two
D.2 Was the performance of the search filter tested on the gold standard from which it was derived?ds).
Yes (for both)
D.3 Report sensitivity data (a single value, a range, ‘Unclear’* or ‘not reported’, as appropriate). *Please describe. ds).
1 .92.8
2. 95.2
D.4 Report precision data (a single value, a range, ‘Unclear’* or ‘not reported’ as appropriate). *Please describe. ).
1. 15.6
2. 16.9
D.6 Other performance measures reported.
Number needed to read (NRR) (1/precision):
1. 6.4
2. 5.9
D.7 Other observations.
E. External validity testing (This section relates to testing the search filter on records that are different from the records used to identify the search terms).
E.1 How many filters were tested for external validity on records different from those used to identify the search terms?
Two
E.2 Describe the validation set(s) of records, including the interface.
GSval94 and GSval99 see B2 for further details.
For each filter report the following information.
E.3 On which validation set(s) was the filter tested?
Both
E.4 Report sensitivity data for each validation set (a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
GSval 94: GSval 99:
1.98.1 1. 91.8
2.98.1 2. 95.1
E.5 Report precision data for each validation set (report a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
GSval 94: GSval 99:
1.10.9 1. 4.7
2.12.0 2. 5.0
E.6 Report specificity data for each validation set (a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
E.7 Other performance measures reported.
NRR:
GSval 94: GSval 99:
1.9.2 1. 21.3
2.8.3 2. 20.0
E.8 Other observations
F. Limitations and Comparisons
F.1 Did the authors discuss any limitations to their research?
No
Filter performance is likely to be dependent on prevalence of diagnostic records in a search set as shown in these two gold standards.
F.2 Are there other potential limitations to this research that you have noticed?
F.3 Report any comparisons of the performance of the filter against other relevant published filters (sensitivity, precision, specificity or other measures).
Haynes 1994 filter:
GSval94 GSval99
Sensitivity 96.2 88.5
Precision 8.2 4.3
F.4 Include the citations of any compared filters.
Haynes RB et al. Developing optimal search strategies for detecting clinically sound studies in Medline. JAMIA 1994;1:447-58.
F.5 Other observations and / or comments.
Haynes filter had poorer performance although differences were not statistically significant. Haynes filter was developed with exclusions of commentaries, correspondence and editorials, which did feature in Bachman GSs.
G. Other comments. This section can be used to provide any other comments. Selected prompts for issues to bear in mind are given below.
G.1 Have you noticed any errors in the document that might impact on the usability of the filter?
No
G.2 Are there any published errata or comments (for example in the MEDLINE record)?
No
G.3 Is there public access to pre-publication history and / or correspondence?
No
G.4 Are further data available on a linked site or from the authors?
No
G.5 Include references to related papers and/or other relevant material.
None
G.6. Other comments
The paper offers four different versions of the filter for Datastar, Ovid, PubMed and Silverplatter, but does not explain how the versions were arrived at.
The GS was only four journals in two years plus another four journals in another year.