Prognosis filters for MEDLINE: performance data

This figure shows that the most precise general prognosis filter as reported by authors (blue bars) was Teljeur-Murphy 22 (5%), followed by the Hedges specific strategy (3%). Four filters reported precision of 2%. The Teljeur-Murphy filters 22 and 26 showed a much better precision when used in validation exercises (both achieved 6%). Other filters (which had not been validated by their authors or for which the authors did not report precision data ) showed high precision in validation studies: Geersing (10%), Ingui (6%), Yale-2 (6%) and Yale -1 (5%).

Prognostic factors filters for MEDLINE

This figure shows that the most sensitive filter as reported by authors (blue bars) was Ingui (98.2%), followed by the Haynes broad strategy (96%) and Stallings (95%). The filters that have been validated in other articles, all show a drop-off in performance in the validation studies. Amongst validation studies, the best performing filters (orange bars) are Irvin (90.4%) and the Parker Hedges+natural history filter (90.1%). None of the filters that were validated reached 95% sensitivity.

This figure shows that the most precise filter as reported by authors (blue bars) was the Wilczynski optimised filter (2%). This filter, when validated in other sets of studies (orange bars), showed a large increase in precision to 20.6%. The Wilczynski optimised filter experienced only a small drop in sensitivity in the validation set (see above), which does not explain the large increase in precision.

Among filters that have only have precision data reported in validation studies, Irvin (18.3%) and the two Parker filters were the most precise (17.7% and 20.2%).

The next figure shows the highest sensitivity and precision data for filters that provided authors’ data on both measures. The data are only shown for filters with sensitivity of 94% or greater.

Key:

(1: Only sensitivity data reported)

2: Ebell “modified Haynes 26”

3: Haynes broad

4. Teljeur-Murphy 26

5: Ebell RCSI filter

6: Ingui OR Geersing

Cohort Studies/

incidence.tw.

Mortality/

Follow-Up Studies/

prognos*.tw.

predict*.tw.

course.tw.

Survival Analysis/

or/1-8

Reported in:

Irvin E, Hayden J. Developing and testing an optimal search strategy for identifying studies of prognosis. Presented at: 14th Cochrane Colloquium; Dublin, Ireland; 23–26 Oct 2006

Validated in:

Boulos L, Ogilvie R, Hayden J A. Search methods for prognostic factor systematic reviews: a methodologic investigation. J Med Lib Assoc [Online]. 2021; 109(1): 23–32.

incidence [MeSH: noexp] OR mortality [MeSH Terms] OR follow up studies[MeSH: noexp] OR prognos*[Text Word] OR predict*[Text Word] OR course*[Text Word]

Reported in:

Wilczynski NL, Haynes RB. Developing optimal search strategies for detecting clinically sound prognostic studies in MEDLINE: an analytic survey. BMC Med. 2004;2:23. [PubMed].

and

Clinical queries filters. National Library of Medicine; 2 Nov 2021. https://pubmed.ncbi.nlm.nih.gov/help/#clinical-study-category-filters [Last accessed: 9 Dec 2022]

Validated in:

Kavanagh PL, Frater F, Navarro T, LaVita P, Parrish R, Iorio A, Optimizing a literature surveillance strategy to retrieve sound overall prognosis and risk assessment model papers, J Am Med Inform Assoc 2021; ocaa232.

Frazier JJ, Stein CD, Tseytlin E, and Bekhuis T. Building a gold standard to construct search filters: a case study with biomarkers for oral cancer. J Med Libr Assoc. 2015; 103(1): 22–30.

Stallings, E., Gaetano-Gil, A., Alvarez-Diaz, N. et al. Development and evaluation of a search filter to identify prognostic factor studies in Ovid MEDLINE. BMC Med Res Methodol 2022;22:107.

(prognos*[Title/Abstract] OR (first[Title/Abstract] AND episode[Title/Abstract]) OR cohort[Title/Abstract])

Reported in:

Wilczynski NL, Haynes RB. Developing optimal search strategies for detecting clinically sound prognostic studies in MEDLINE: an analytic survey. BMC Med. 2004;2:23. [PubMed].

and

Clinical queries filters. National Library of Medicine; 2 Nov 2021. https://pubmed.ncbi.nlm.nih.gov/help/#clinical-study-category-filters [Last accessed: 9 Dec 2022]

Valdated in: no validation identified.

prognosis.sh.

diagnosed.tw.

cohort:.mp.

predictor:.tw.

death.tw.

exp models, statistical/

or/1-6

Reported in:

Wilczynski NL, Haynes RB. Developing optimal search strategies for detecting clinically sound prognostic studies in MEDLINE: an analytic survey. BMC Med. 2004;2:23. [PubMed].

Validated in:

Boulos L, Ogilvie R, Hayden J A. Search methods for prognostic factor systematic reviews: a methodologic investigation. J Med Lib Assoc [Online]. 2021; 109(1): 23–32.

(Predict*[tiab) OR Predictive value Of tests[mh] OR Scor*[tiab] OR Observ*[tiab] OR Observer variation[mh]

Reported in:

Clinical queries filters. National Library of Medicine; 2 Nov 2021. https://pubmed.ncbi.nlm.nih.gov/help/#clinical-study-category-filters [Last accessed: 9 Dec 2022]

Validated in:

Keogh C, Wallace E, O'Brien KK, Murphy PJ, Teljeur C, McGrath B, Smith SM, Doherty N, Dimitrov BD, Fahey T. Optimized retrieval of primary care clinical prediction rules from MEDLINE to establish aWeb-based register. J Clin Epi. 2011;64(8):848-60.

Geersing GJ, Bouwmeester W, Zuithoff P, Spijker R, Leeflang M, Moons K. Search filters for finding prognostic and diagnostic prediction studies in Medline to enhance systematic reviews. PLoS One. 2012;7(2):e32844

validation [tiab] OR validate [tiab]

Reported in:

Clinical queries filters. National Library of Medicine; 2 Nov 2021. https://pubmed.ncbi.nlm.nih.gov/help/#clinical-study-category-filters [Last accessed: 9 Dec 2022]

Validated in:

Keogh C, Wallace E, O'Brien KK, Murphy PJ, Teljeur C, McGrath B, Smith SM, Doherty N, Dimitrov BD, Fahey T. Optimized retrieval of primary care clinical prediction rules from MEDLINE to establish aWeb-based register. J Clin Epi. 2011;64(8):848-60.

Ebell MH, Fahey T, Murphy ME, Barry A, Barry H, Hickner J. An updated and more efficient search strategy to identify primary care relevant clinical prediction rules. J Clin Epidemiol. 2020; 125: 26-29 doi:10.1016/j.jclinepi.2020.05.013

cohort.ti,ab.

incidence.ti,ab.

mortality.ti,ab.

follow-up study.ti,ab.

follow-up studies.ti,ab.

prognos*.ti,ab.

predict*.ti,ab.

course.ti,ab.

natural history.ti,ab.

or/1-9

Reported in:

Parker R, Tougas ME, Hayden JA. Validating prognosis search filters using relative recall based on prognosis systematic reviews. Poster presented at: 21st Cochrane Colloquium; Quebec City, QC, Canada; 19–23 Sep 2013

Validated in:

Boulos L, Ogilvie R, Hayden J A. Search methods for prognostic factor systematic reviews: a methodologic investigation. J Med Lib Assoc [Online]. 2021; 109(1): 23–32.

Incidence/

exp Mortality/

Follow-Up Studies/

prognos*.tw.

predict*.tw.

course*.tw.

(first and episode).ti,ab.

cohort.ti,ab.

natural history.tw.

or/1-9

Reported in:

Parker R, Tougas ME, Hayden JA. Validating prognosis search filters using relative recall based on prognosis systematic reviews. Poster presented at: 21st Cochrane Colloquium; Quebec City, QC, Canada; 19–23 Sep 2013

Validated in:

Boulos L, Ogilvie R, Hayden J A. Search methods for prognostic factor systematic reviews: a methodologic investigation. J Med Lib Assoc [Online]. 2021; 109(1): 23–32.

Geersing GJ, Bouwmeester W, Zuithoff P, Spijker R, Leeflang M, Moons K. Search filters for finding prognostic and diagnostic prediction studies in Medline to enhance systematic reviews. PLoS One. 2012;7(2):e32844

(Validat$ OR Predicts.ti. OR Rules OR (PredictS AND (Outcomes OR Risk$ OR ModelS)) OR (History OR Variable$ OR Criteria OR Scors OR Characteristic$ OR Finding$ OR Factors) AND (Predicts OR Models OR Decisions OR IdentifS OR Prognoss» OR (Decisions AND (ModelS OR Clinical$ OR Logistic Models')) OR (Prognostic AND (History OR VariableS OR Criteria OR Scor$ OR CharacteristicS OR FindingS OR FactorS OR Model$))

Reported in:

Ingui BJ, Rogers MA. Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc. 2001;8(4):391-7.

Validated in:

Geersing GJ, Bouwmeester W, Zuithoff P, Spijker R, Leeflang M, Moons K. Search filters for finding prognostic and diagnostic prediction studies in Medline to enhance systematic reviews. PLoS One. 2012;7(2):e32844

Kavanagh PL, Frater F, Navarro T, LaVita P, Parrish R, Iorio A, Optimizing a literature surveillance strategy to retrieve sound overall prognosis and risk assessment model papers, J Am Med Inform Assoc 2021; ocaa232.

Predict$ OR Risk$

Reported in:

Ingui BJ, Rogers MA. Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc. 2001;8(4):391-7.

Validated in:

Wong SS, Wilczynski NL, Haynes RB, Ramkissoonsingh R, Hedges Team. Developing optimal search strategies for detecting sound clinical prediction studies in MEDLINE. AMIA Annual Symposium Proceedings 2003;728-32.

Decision Support Techniques/ AND Predictive Value of Tests

Reported in:

Ingui BJ, Rogers MA. Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc. 2001;8(4):391-7.

Validated in:

Wong SS, Wilczynski NL, Haynes RB, Ramkissoonsingh R, Hedges Team. Developing optimal search strategies for detecting sound clinical prediction studies in MEDLINE. AMIA Annual Symposium Proceedings 2003;728-32.

"stratification" OR "ROC curve" [mesh] Or "discrimination" OR "discriminate" OR "c-statistic" OR "area under the curve" OR "auc" OR "calibration" OR "indices" OR "algorithm" OR "multivariable"

This is reported combined with Ingui or Haynes broad filters using the OR operator

Reported and validated in:

Geersing GJ, Bouwmeester W, Zuithoff P, Spijker R, Leeflang M, Moons K. Search filters for finding prognostic and diagnostic prediction studies in Medline to enhance systematic reviews. PLoS One. 2012;7(2):e32844

(“clinical prediction” OR clinical model*[All Fields] OR clinical score*[All Fields] OR decision rule*[All Fields] OR diagnostic accuracy OR diagnostic rule*[All Fields] OR diagnostic score*[All Fields] OR diagnostic value OR predictive outcome*[All Fields] OR predictive rule*[All Fields] OR predictive score*[All Fields] OR predictive value OR predictive risk*[All Fields] OR prediction outcome*[All Fields] OR prediction rule*[All Fields] OR prediction score*[All Fields] OR prediction value*[All Fields] OR prediction risk*[All Fields] OR risk assessment OR risk score*[All Fields] OR validation decision*[All Fields] OR validation rule*[All Fields] OR validation score*[All Fields] OR (derivation AND validation) OR (sensitivity AND specificity))

Reported in:

Keogh C, Wallace E, O'Brien KK, Murphy PJ, Teljeur C, McGrath B, Smith SM, Doherty N, Dimitrov BD, Fahey T. Optimized retrieval of primary care clinical prediction rules from MEDLINE to establish aWeb-based register. J Clin Epi. 2011;64(8):848-60.

Validated in:

Kavanagh PL, Frater F, Navarro T, LaVita P, Parrish R, Iorio A, Optimizing a literature surveillance strategy to retrieve sound overall prognosis and risk assessment model papers, J Am Med Inform Assoc 2021; ocaa232.

Keogh C, Wallace E, O'Brien KK, Murphy PJ, Teljeur C, McGrath B, Smith SM, Doherty N, Dimitrov BD, Fahey T. Optimized retrieval of primary care clinical prediction rules from MEDLINE to establish aWeb-based register. J Clin Epi. 2011;64(8):848-60.

(clinical[tiab] AND predict*[tiab]) OR (clinical[tiab] AND model*[tiab] ) OR (clinical[tiab] AND score*[tiab]) OR (decision [tiab] AND rule*[tiab]) OR (derive*[tiab] AND validat*[tiab]) OR (diagnos*[tiab] AND accura*[tiab]) OR (diagnos*[tiab] AND rule*[tiab]) OR (diagnos*[tiab] AND score*[tiab]) OR (diagnos*[tiab] AND value[tiab]) OR (predict*[tiab] AND outcome*[tiab]) OR (predict*[tiab] AND rule*[tiab] OR (predict*[tiab] AND score*[tiab] ) OR (predict*[tiab] AND validat*[tiab]) OR (predict*[tiab] AND value*[tiab]) OR (risk*[tiab] AND assessment*[tiab]) OR (risk[tiab] AND score*[tiab]) OR (sensitivity[tiab] AND specificity[tiab]) OR (symptoms[tiab] AND signs[tiab]) OR (validat*[tiab] AND decision*[tiab]) OR (validat*[tiab] AND rule*[tiab]) OR (validat*[tiab] AND score*[tiab]) OR (predict*[tiab] AND risk*[tiab])

Reported in:

Keogh C, Wallace E, O'Brien KK, Murphy PJ, Teljeur C, McGrath B, Smith SM, Doherty N, Dimitrov BD, Fahey T. Optimized retrieval of primary care clinical prediction rules from MEDLINE to establish aWeb-based register. J Clin Epi. 2011;64(8):848-60.

(allele OR amino OR animal OR apoptosis OR chromosome OR congenital OR dental OR dna OR endogenous OR endothelial OR epithelial OR mammalian OR mice OR molecule OR molecular OR mouse OR mutate OR mutation OR necrosis OR pathogenesis OR phosphorylation OR polymorphism OR receptor OR signal OR species OR tissue OR tumor OR tumour OR tyrosine OR vitro)

Reported in:

Keogh C, Wallace E, O'Brien KK, Murphy PJ, Teljeur C, McGrath B, Smith SM, Doherty N, Dimitrov BD, Fahey T. Optimized retrieval of primary care clinical prediction rules from MEDLINE to establish aWeb-based register. J Clin Epi. 2011;64(8):848-60

cohort studies[mh] or prognosis[mh] or disease progression[mh]

Reported on the Yale web site [Last accessed: 9 Dec 2022]

Validated in:

Kok R, Verbeek JAHM, Faber B, van Dijk, Frank JH, Hoving, Jan L A search strategy to identify studies on the prognosis of work disability: a diagnostic test framework. BMJ Open. 2015;5:e006315. doi:10.1136/bmjopen-2014-006315

• although it is unclear if the filter validated was this one of the Yale prognosis and natural history below.

cohort studies[mh] OR prognosis[mh] OR mortality[mh] OR morbidity[mh] OR "natural history" OR prognost*[tiab] OR course[tiab] OR predict*[tiab] OR outcome assessment[mh] OR outcome*[tiab] OR inception cohort* OR disease progression[mh] OR survival analysis[mh]

Reported on the Yale web site [Last accessed: 9 Dec 2022]

Validated in:

Kok R, Verbeek JAHM, Faber B, van Dijk, Frank JH, Hoving, Jan L A search strategy to identify studies on the prognosis of work disability: a diagnostic test framework. BMJ Open. 2015;5:e006315. doi:10.1136/bmjopen-2014-006315

• although it is unclear if the filter validated was this one or the Yale filter above.

prognos*[TIAB] OR cohort [TIAB] OR validat*[TIAB] OR predict* [TIAB] OR mortality [TIAB] OR follow up [TIAb]

Reported in:

Kavanagh PL, Frater F, Navarro T, LaVita P, Parrish R, Iorio A, Optimizing a literature surveillance strategy to retrieve sound overall prognosis and risk assessment model papers, J Am Med Inform Assoc 2021; ocaa232.

Validated in: none reported

((“clinical prediction”[tiab] OR “clinical model”[tiab] OR “clinical score”[tiab] OR “clinical scoring”[tiab] OR “validation of a clinical”[tiab] OR “decision guideline”[tiab] OR “validation study”[tiab] OR “validation studies”[tiab] OR “derivation study”[tiab] OR “screening score”[tiab] OR “decision rule”[tiab] OR “diagnostic rule”[tiab] OR “diagnostic score”[tiab] OR “predictive outcome”[tiab] OR “predictive rule”[tiab] OR “predictive score”[tiab] OR “predictive value”[tiab] OR “predictive risk”[tiab] OR “prediction outcome”[tiab] OR “prediction rule”[tiab] OR “prediction score”[tiab] OR “scoring”[tiab] OR “prediction value”[tiab] OR “prediction risk”[tiab] OR “risk assessment”[tiab] OR “risk score”[tiab] OR “risk scoring”[tiab] OR “prognostic score”[tiab] OR “prognostic index”[tiab] OR “prognostic rule”[tiab] OR “prospective validation”[tiab] OR (“risk”[tiab] AND “tool”[tiab]) OR ((“validate”[tiab] OR “validation”[tiab] OR “validating”[tiab] OR “develop”[tiab] OR “development”[tiab] OR “derivation”[tiab] OR “derive”[tiab] OR “deriving”[tiab] OR “performance”[tiab]) AND (“decision”[tiab] OR “predictive”[tiab] OR “prediction”[tiab] OR “rule”[tiab] OR "score"[tiab] OR “scoring”[tiab] OR “index”[tiab] OR “model”[tiab] OR “scale”[tiab] OR “tool”[tiab] OR “algorithm”[tiab])) OR (“development”[tiab] AND “validation”[tiab]) OR (“derivation”[tiab] AND “validation”[tiab]) OR “signs and symptoms”[tiab]) OR (((“PISA”[tiab] OR “PERC”[tiab] OR “PESI”[tiab] OR “Geneva”[tiab] OR “Wells”[tiab]) AND “pulmonary embolism”[tiab]) OR (“Leiden”[tiab] AND “rheumatoid”[tiab]) OR “Ottawa ankle”[tiab] OR “Ottawa knee”[tiab] OR (“Wells”[tiab] AND (“thrombosis”[tiab] OR "thromboembolism"[tiab])) OR ((“ATRIA stroke”[tiab] OR “ATRIA score”[tiab] OR “ATRIA risk”[tiab] OR “ATRIA bleeding”[tiab] OR “HAS-BLED”[tiab]) AND (“atrial fibrillation”[tiab] OR “anticoagulation”[tiab] OR “anticoagulant”[tiab])) OR ((“Centor”[tiab] OR “FeverPAIN”[tiab]) AND (“sore throat”[tiab] OR “pharyngitis”[tiab])) OR ((“AUDIT score”[tiab] OR “CRAFFT”[tiab] OR “AUDIT rule”[tiab] OR “AUDIT score”[tiab] OR “CAGE”[tiab]) AND (“alcohol”[tiab] OR “alcoholism”[tiab])) OR (“CAPRA”[tiab] AND “prostate”[tiab]) OR “San Francisco Syncope Rule”[tiab] OR ((“MMSE”[tiab] OR “MiniCog”[tiab] OR “Montreal Cognitive Assessment”[tiab]) AND (“dementia”[tiab] or “cognitive impairment”[tiab])) OR ((“ABCD2”[tiab] OR “ABCD”[tiab]) AND “transient ischemic attack”[tiab]) OR ((“CHADS2”[tiab] OR “CHA2DS2-VASc”[tiab] OR “CHADS-VASC”[tiab]) AND “atrial fibrillation”[tiab]) OR ((“PHQ-2”[tiab] OR “PHQ-7”[tiab] OR “two question screener”[tiab]) AND “depression”[tiab]) OR ((“GAD-2”[tiab] OR “GAD-7”[tiab]) AND “anxiety”[tiab]) OR “PC-PTSD”[tiab] OR “CRB65”[tiab] OR “CRB-65”[tiab] OR (“C-WATCH”[tiab] AND (“bleeding” OR “haemorrhage”[tiab])) OR “BAP-65”[tiab] OR “STARWAVE”[tiab] OR “CURB-65”[tiab] OR “CURB65”[tiab]))

Reported in:

Ebell MH, Fahey T, Murphy ME, Barry A, Barry H, Hickner J. An updated and more efficient search strategy to identify primary care relevant clinical prediction rules. J Clin Epidemiol. 2020; 125: 26-29 doi:10.1016/j.jclinepi.2020.05.013

Validated in: none reported

((validation[tiab] OR validate[tiab]) OR (“clinical prediction” OR clinical model*[All Fields] OR clinical score*[All Fields] OR decision rule*[All Fields] OR diagnostic accuracy OR diagnostic rule*[All Fields] OR diagnostic score*[All Fields] OR diagnostic value OR predictive outcome*[All Fields] OR predictive rule*[All Fields] OR predictive score*[All Fields] OR predictive value OR predictive risk*[All Fields] OR prediction outcome*[All Fields] OR prediction rule*[All Fields] OR prediction score*[All Fields] OR prediction value*[All Fields] OR prediction risk*[All Fields] OR risk assessment OR risk score*[All Fields] OR validation decision*[All Fields] OR validation rule*[All Fields] OR validation score*[All Fields] OR (derivation AND validation) OR (sensitivity AND specificity))

Reported in:

Ebell MH, Fahey T, Murphy ME, Barry A, Barry H, Hickner J. An updated and more efficient search strategy to identify primary care relevant clinical prediction rules. J Clin Epidemiol. 2020; 125: 26-29 doi:10.1016/j.jclinepi.2020.05.013

Validated in: none reported

cohort, incidence, mortality, follow-up studies, prognos*, predict*, course

Reported in:

McKibbon K, Eady A, Marks S. Evidence Based Principles and Practice.PDQ Series. Hamilton, Canada: BC Dekker; 1999

Validated in:

Hayden JA, Côté P, Bombardier C. Evaluation of the quality of prognosis studies in systematic reviews. Ann Intern Med. 2006 Mar 21;144(6):427-37. doi: 10.7326/0003-4819-144-6-200603210-00010.

predict:.tw. OR validat:.tw. OR develop.tw

Reported in:

Wong SS, Wilczynski NL, Haynes RB, Ramkissoonsingh R, Hedges Team. Developing optimal search strategies for detecting sound clinical prediction studies in MEDLINE. AMIA Annual Symposium Proceedings 2003;728-32.

Validated in: none reported

((predict* N3 rule* OR predict* N3 model OR predict* N3 models) OR (decision* N3 rule*) OR (TX validat*))

Reported in:

Keogh C, Wallace E, O'Brien KK, Murphy PJ, Teljeur C, McGrath B, Smith SM, Doherty N, Dimitrov BD, Fahey T. Optimized retrieval of primary care clinical prediction rules from MEDLINE to establish aWeb-based register. J Clin Epi. 2011;64(8):848-60.

Validated in: none reported

((predict* N3 rule* OR predict* N3 model OR predict* N3 models) OR (decision* N3 rule*))

Reported in:

Keogh C, Wallace E, O'Brien KK, Murphy PJ, Teljeur C, McGrath B, Smith SM, Doherty N, Dimitrov BD, Fahey T. Optimized retrieval of primary care clinical prediction rules from MEDLINE to establish aWeb-based register. J Clin Epi. 2011;64(8):848-60.

Validated in: none reported

1 exp Risk/

2 risk.tw

3 exp Cohort Studies/

4 cohort.tw

5 exp Prognosis/

6 "prognos*".tw

7 "predict*".tw

8 exp Incidence/

9 incidence.tw

10 exp Survival Analysis/

11 survival.tw

12 "causal factor".tw

13 course.tw

14 or/1–13

Reported in:

Stallings, E., Gaetano-Gil, A., Alvarez-Diaz, N. et al. Development and evaluation of a search filter to identify prognostic factor studies in Ovid MEDLINE. BMC Med Res Methodol 2022;22:107.

Validated in: none reported