Ayiku, 2021 appraisal
This appraisal is for Ayiku L, Hudson T, Williams C, Levay P, Jacob C. The NICE OECD countries' geographic search filters: Part 2-validation of the MEDLINE and Embase (Ovid) filters. J Med Libr Assoc. 2021;109(4):583-589.
This appraisal was prepared by Helen Fulbright and Claire Stansfield in December 2022.
Information and Methodological Issues
Categorisation Issues
Detailed information, as appropriate
A. Information
A.1 State the author's objective
To create a validated search filter for the 37 Organisation for Economic Co-operation and Development (OECD) countries and improve the effectiveness of literature searches.
A.2 State the focus of the search
[x] Sensitivity-maximising
[ ] Precision-maximising
[ ] Specificity-maximising
[ ] Balance of sensitivity and specificity / precision
[ ] Other
The authors report that the target for recall was 90%.
A.3. Database(s) and search interface(s).
Ovid MEDLINE and Ovid Embase.
A.4.Describe the methodological focus of the filter (e.g. RCTs).
To retrieve research about OECD countries as a geographic location.
A.5 Describe any other topic that forms an additional focus of the filter (e.g. clinical topics such as breast cancer, geographic location such as Asia or population grouping such as paediatrics).
Not applicable
A.6 Other obervations
None
B. Identification of a gold standard (GS) of known relevant records
B. 1 Did the authors identify one or more gold standards (GSs)?nown relevant records
Two
One gold standard set was used to validate the MEDLINE filter and one for the Embase filter.
The filter performance was also tested on three guidelines containing known relevant references to assess number needed to read (NNR) as well as recall.
B.2 How did the authors identify the records in each GS? wn relevant records
Known, relevant references (thirty National Institute for Health and Care Excellence (NICE) guidelines) were used to generate validation gold standard sets. NICE guidelines have evidence description sections which summarise the included publications and note the geographic setting of each publication. This allowed the authors to identify the records for OECD countries.
B.3 Report the dates of the records in each GS. wn relevant records
The thirty NICE guidelines that were used to generate the validation GS sets were published between December 2018 and December 2019. However, the literature cited in each of these guidelines will cover various date ranges which were not specified in the papers written by the authors on the search filters.
B.4 What are the inclusion criteria for each GS? relevant records
The only inclusion criteria applied to each GS was that records had to be relevant to OECD countries.
B.5 Describe the size of each GS and the authors’ justification, if provided (for example the size of the gold standard may have been determined by a power calculation) antcords
There was no test GS.
Validation GS:
• MEDLINE = 2065
• Embase = 2023
The size of each gold standard was not determined by a power calculation. The numbers used were justified from previous research stating that a set of at least 100 publications is sufficient for search filter development (Sampson, Margaret et al. An alternative to the hand searching gold standard: validating methodological search filters using relative recall. BMC Medical Research Methodology. 2006;6:33. doi:10.1186/1471-2288-6-33)
B.6 Are there limitations to the gold standard(s)? ntcords
Unclear
B.7 How was each gold standard used? cords
[ ] to identify potential search terms
[] to derive potential strategies (groups of terms)
[] to test internal validity
[x] to test external validity
[ ] other, please specify
For the authors' earlier work that tests internal validity and describes development of the filter, see: Ayiku L, Levay P, Hudson T. The NICE OECD countries' geographic search filters: Part 1-methodology for developing the draft MEDLINE and Embase (Ovid) filters. Journal of the Medical Library Association. 2021;109:258-66
B.8 Other observations. cords
None
C. How did the researchers identify the search terms in their filter(s) (select all that apply)?
C.1 Adapted a published search strategy.
No
C.2 Asked experts for suggestions of relevant terms.
No
C.3 Used a database thesaurus.
Yes
The strategy uses National Library of Medicine (NLM) Medical Subject Heading (MeSH) terms.
C.4 Statistical analysis of terms in a gold standard set of records (see B above).
No
C.5 Extracted terms from the gold standard set of records (see B above).
No
C.6 Extracted terms from some relevant records (but not a gold standard).
No
C.7 Tick all types of search terms tested.
[x] subject headings
[ ] text words (e.g. in title, abstract)
[ ] publication types
[ ] subheadings
[ ] check tags
[ ] other, please specify
C.8 Include the citation of any adapted strategies.
Not applicable
C.9 How were the (final) combination(s) of search terms selected?
The filter finds evidence about OECD countries by excluding evidence about non-OECD countries (unless these occur with OECD countries).
C.10 Were the search terms combined (using Boolean logic) in a way that is likely to retrieve the studies of interest?
Yes, by excluding studies with subject headings that are unlikely to be of interest.
C.11 Other observations.
None
D. Internal validity testing (This type of testing is possible when the search filter terms were developed from a known gold standard set of records).
D.1 How many filters were tested for internal validity? cords).
N/A – internal validity cannot be tested as the filters were not derived from a set of relevant records, but were instead derived from the list of OECD countries and available MeSH headings.
D.2 Was the performance of the search filter tested on the gold standard from which it was derived?ds).
No
N/A – internal validity cannot be tested as the filters were not derived from a set of relevant records, but were instead derived from the list of OECD counties and available MeSH headings.
D.3 Report sensitivity data (a single value, a range, ‘Unclear’* or ‘not reported’, as appropriate). *Please describe. ds).
Not applicable
D.4 Report precision data (a single value, a range, ‘Unclear’* or ‘not reported’ as appropriate). *Please describe. ).
Not applicable
D.5 Report specificity data (a single value, a range, ‘Unclear’* or ‘not reported’ as appropriate). *Please describe. ).
Not applicable
D.6 Other performance measures reported.
None
D.7 Other observations.
None
E. External validity testing (This section relates to testing the search filter on records that are different from the records used to identify the search terms).
E.1 How many filters were tested for external validity on records different from those used to identify the search terms?
Two filters were tested for external validity – the Medline filter and the Embase filter. The filters were not derived from a set of relevant records, but were each tested on a validation set.
The filter performance was also tested on three guidelines containing known relevant references to assess number needed to read (NNR) as well as recall.
E.2 Describe the validation set(s) of records, including the interface.
2,065 MEDLINE references and 2,023 Embase references (see Appendix 5 in the paper) formed the GS validation sets.
For each filter report the following information.
E.3 On which validation set(s) was the filter tested?
The MEDLINE and Embase filters were individually tested against their own validation gold standard sets.
E.4 Report sensitivity data for each validation set (a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
Both the MEDLINE and Embase filters achieved 99.95% recall against their respective gold standard sets.
The filter performance was also tested on three guidelines containing known relevant references to assess number needed to read (NNR) as well as recall. Both filters achieved 100% recall for the three NICE guideline reviews.
E.5 Report precision data for each validation set (report a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
None of the validation sets were used to test precision.
E.6 Report specificity data for each validation set (a single value, a range or ‘Unclear’ or ‘not reported’, as appropriate).
Not reported
E.7 Other performance measures reported.
Number Needed to Read (NNR) is reported in Table 2 in the paper.
E.8 Other observations
None
F. Limitations and Comparisons
F.1 Did the authors discuss any limitations to their research?
Yes
• The filters will not remove non-OECD countries that do not have geographic subject headings and therefore some irrelevant non-OECD country evidence will be retained.
• The filter will remove papers that have non-OECD subject headings only, regardless of whether they also have OECD content (that is not reflected by the presence of a subject heading). So for international topics that are likely to be covered in papers that include OECD and non-OECD countries (e.g the authors found this in the case of research about looked-after children from a non-OECD country who moved to live in an OECD country), the filter should be used with caution. This was found during testing as reported in Ayiku L, Levay P, Hudson T. The NICE OECD countries' geographic search filters: Part 1-methodology for developing the draft MEDLINE and Embase (Ovid) filters. Journal of the Medical Library Association. 2021;109:258-66.
F.2 Are there other potential limitations to this research that you have noticed?
No
F.3 Report any comparisons of the performance of the filter against other relevant published filters (sensitivity, precision, specificity or other measures).
Not applicable.
F.4 Include the citations of any compared filters.
Not applicable.
F.5 Other observations and / or comments.
None
G. Other comments. This section can be used to provide any other comments. Selected prompts for issues to bear in mind are given below.
G.1 Have you noticed any errors in the document that might impact on the usability of the filter?
Yes
When copying and pasting the filter from https://osf.io/2cxap, there are errors with the spacing that cannot be seen in the presentation of the document. The spacing errors require the following corrections.
MEDLINE errors:
orcote d'ivoire/ (needs a space after ‘or’)
or indian ocean islands/ (‘or’ needs a space before it)
Embase errors:
orbotswana/ (needs a space after ‘or’)
orlatvia/ (needs a space after ‘or’)
G.2 Are there any published errata or comments (for example in the MEDLINE record)?
No
Checked 21 Dec 2022
G.3 Is there public access to pre-publication history and / or correspondence?
Yes
G.5 Include references to related papers and/or other relevant material.
Ayiku L, Levay P, Hudson T. The NICE OECD countries' geographic search filters: Part 1-methodology for developing the draft MEDLINE and Embase (Ovid) filters. Journal of the Medical Library Association. 2021;109:258-66.
See https://www.oecd.org/ for a list of all current OECD countries.
G.6. Other comments
None