Wilczynski 2005

Appraisal of: "Wilczynski NL, Morgan D, Haynes RB; Hedges Team. An overview of the design and methods for retrieving high-quality studies for clinical care. BMC Med Inform Decis Mak. 2005 Jun 21;5:20."

Reviewer(s):
Julie Glanville
Kath Wright

Full Reference:
Wilczynski NL, Morgan D, Haynes RB; Hedges Team. An overview of the design and methods for retrieving high-quality studies for clinical care. BMC Med Inform Decis Mak. 2005 Jun 21;5:20.

Short description:

The authors describe the methods they use to develop optimal search filters (hedges) to identify a range of study types: causation, prognosis, diagnosis, treatment, economics, clinical prediction guides, reviews, costs, and qualitative: the filters were designed for MEDLINE and Embase. The authors built a large gold standard (reference set) by hand searching 170 journals for one year: 2000. Relevant records were defined and were selected to represent best research methods. The gold standard records were then downloaded from MEDLINE, Embase, CINAHL, and PsycINFO with the subject indexing assigned by each database. Candidate search terms were identified from the gold standard records and consulting experts. The sensitivity, specificity, precision, and accuracy of unique search terms and combinations of search terms were calculated. Once the performance parameters of individual search terms were computed, the authors selected individual terms for the construction of search strategies by choosing search terms with specific levels of sensitivity and specificity (which varied by database). The authors also used logistic regression to explore ways to improve filter performance. Strategies were developed in a random selection of 60% of the gold standard and validated in the remaining 40%. No statistical differences in performance were found between the two strategy development methods or between the test and validation results, so the majority of filter development used the Boolean approach and search strategies were developed using all records in the database.

The gold standard database numbered 60,330 records, each with up to 11 data fields. Filters were developed for studies of causation, prognosis, diagnosis, treatment, economics, clinical prediction guides, reviews, costs, and those of a qualitative nature.

Limitations stated by the author(s):

The authors do not state any limitations to their methods.

Limitations stated by the reviewer(s):

The journals contributing to the gold standard were largely general (internal medicine, general medical practice, mental health and general nursing practice) rather than specialty focused. There may be issues about generalizability to the latter journals. Records were selected for the gold standard according to the authors’ definition of the specific topics and best study designs for those topics. These definitions need to be compared to the definitions of any user. The authors decided, following testing, to use all of the database records to both develop and test the filters. Ideally validation data on a different reference standard should be provided.

Study Type:
Single study

Tags: