Petrova 2012

Appraisal of: Search terms and a validated brief search filter to retrieve publications on health-related values in Medline: A word frequency analysis study


Reviewer(s): 

Sigrid Droste

Kristin Bakke Lysdahl


Full Reference: 

Petrova M, Sutcliffe P, Fulford KW, Dale J. Search terms and a validated brief search filter to retrieve publications on health-related values in Medline: A word frequency analysis study. J Am Med Inform Assoc 2012; 19(3): 479–488.

Short description: 

The aim of the article is to present search terms (keywords) and a “brief” search filter to identify information on health-related values in Medline developed by using word frequency analysis. The purpose of the project is to support “broad-scoping searches by “a brief search filter (≤20 lines) of high precision (≥67%) and acceptable sensitivity (≥67%) which can be used […] in generic medical and health databases (e.g., Medline, Embase, or Cinahl) and across a range of topics (e.g., health conditions, health settings, health interactions, etc.).

The article provides extensive information on the background as on health-related values defined by the authors and on word frequency analyses. The application of this analysis in the conditions diabetes, dementia, schizophrenia and obesity resulted in a small overlap of relevant MeSH and free-text search terms (9 of 124 different MeSH terms (7.3%) and 4 of 144 different words (2.8%) were identified in all 4 conditions among the 50 “best” MeSH resp. words in each condition). Applying the developed search filter in hypersensitivity and dentistry resulted in (authors defined) sensitivity of 70.1% (hypersensitivity) and 47.1% (dentistry), and precision of 63.6% (hypersensitivity) and 82.6% (dentistry). 


Limitations stated by the author(s): 

- A number of ‘subjective’ additions to the objectively derived filter improved sensitivity and precision.

- The brief values filter can be applied productively to other topics in Medline, but there is instability in its performance.

- The study covers a limited period, limited number of topics, a single database, and a single approach to compiling its corpus of citations. 

- The study lacked well-articulated criteria for distinguishing cases (of values publications) from non-cases. The criteria used may have been overly inclusive. 

- This study did not have the capacity for both theoretical and empirical work of adequate depth.

- The reported frequencies for text words did not necessarily reflect the number of abstracts in which a word appears.

- Data cleaning was performed on text words that had a 100% precision, but not for the remainder.

- Limit to the very field of designing objective search strategies on the basis of word frequency analysis: The field may be lagging behind advances in medical informatics. Some of the methods applied in this study may need to be superseded, including methods that have improved on standard practices of objective search strategy design.

Limitations stated by the reviewer(s): 

The theoretical concept of values defined by the authors is not fully clear and different from those used in ethical analysis in HTA. Besides the many limitations already stated by the authors there are some significant methodological shortcomings, and the results don’t fulfill the needs of systematic reviews in HTA regarding sensitivity in a sufficient manner. The proposed search filter is clearly not this universal solution as claimed in the authors’ objectives. As well the results indicate that the method of word frequency analysis itself fails in the field of values. 

The shortcomings in detail:

- The glossary definitions are not always identical with the common ones in Information Retrieval.

- Authors’ objective: No reasoning for the line number (and no meaning on the length of each line) and sensitivity and precision.

- The validation, in particular the external validation, applied by the authors does not fulfill criteria of an evidence-based validation.

- Building-up the derivation dataset is not reported in a transparent and reproducible manner. As the conditions were just defined by one MeSH term each the sensitivity of the results may be low. Calculating sensitivity of the search filter by using these datasets seems to be questionable (and a self-fulfilling prophecy).

- Internal validity: calculation is not correct as the authors wrote “precision = true positives picked by filter / (true positives picked by filter + false positives picked by filter). 


Study Type: 

Single study

Related Chapters: 

Ethical analysis

Tags: