Frazier 2015

Appraisal of: "Frazier JJ, Stein CD, Tseytlin E, Bekhuis T. Building a gold standard to construct search filters: a case study with biomarkers for oral cancer. J Med Libr Assoc. 2015;103(1):22-30."

Reviewer(s):
Kath Wright
Sonia Garcia

Full Reference:
Frazier JJ, Stein CD, Tseytlin E, Bekhuis T. Building a gold standard to construct search filters: a case study with biomarkers for oral cancer. J Med Libr Assoc. 2015;103(1):22-30.

Short description:

This paper describes the process involved in the development of a gold standard dataset in oral squamous carcinoma (OSCC) against which future filters for prognostic biomarkers could be tested. Given the lack of filters for biomarkers in OSCC, the authors set out to produce a set of key studies that a filter developed for use with other databases should retrieve. They designed a MEDLINE search strategy to retrieve prognostic studies using the prognostic filter developed by the Haynes group. An annotating system was used to review the references retrieved by the search. Two reviewers independently assessed the studies and a calibration process for the results is described. The agreement about relevance across reviewers was measured by Cohen’s kappa statistic, achieving a high level of agreement. This system for developing gold standards is likely to be generalizable to other subject domains.

Limitations stated by the author(s):

Studies were screened using only the title and abstract, and a small percentage of retrieved studies didn’t include an abstract.

Searches were only conducted in MEDLINE.

Limitations stated by the reviewer(s):

The search strategy used to identify studies of prognostic biomarkers for OSCC incorporated an existing search filter for prognostic studies without any discussion or reference to if, and how, this had been validated. The authors limited to studies indexed with the MeSH Humans tag. A better approach would have been to carefully exclude animal studies. The searches were conducted on a small 5 year segment of MEDLINE which may not be appropriate for other topic areas

Study Type:
Single study

Related Chapters:
Search filters

Tags:

• Search filters