Guidelines NOT Supported
Infectious Diseases Society of America (IDSA) Guidelines
NOT Supported By High Quality Evidence
1. Analysis of Overall Level of Evidence Behind Infectious Diseases Society of America Practice Guidelines
Dong Heun Lee, MD; Ole Vielemeyer, MD
Arch Intern Med. 2011;171(1):18-22. doi:10.1001/archinternmed.2010.482
Background Clinical practice guidelines are developed to assist in patient care. Physicians may assume that following such guidelines means practicing evidence-based medicine. However, the quality of supporting literature can vary greatly
Methods We analyzed the strength of recommendation and overall quality of evidence behind 41 Infectious Diseases Society of America (IDSA) guidelines released between January 1994 and May 2010. Individual recommendations were classified based on their strength of recommendation (levels A through C) and quality of evidence (levels I through III). Guidelines not following this format were excluded from further analysis. Evolution of IDSA guidelines was assessed by comparing 5 recently updated guidelines with their earlier versions.
Results In the 41 analyzed guidelines, 4218 individual recommendations were found and tabulated. Fourteen percent of the recommendations were classified as level I, 31% as level II, and 55% as level III evidence. Among class A recommendations (good evidence for support), 23% were level I (≥1 randomized controlled trial) and 37% were based on expert opinion only (level III). Updated guidelines expanded the absolute number of individual recommendations substantially. However, few were due to a sizable increase in level I evidence; most additional recommendations had level II and III evidence.
Conclusions More than half of the current recommendations of the IDSA are based on level III evidence only. Until more data from well-designed controlled clinical trials become available, physicians should remain cautious when using current guidelines as the sole source guiding patient care decisions.
2.
IDSA Practice Guidelines Scrutinized for Evidence Level
Emma Hitt, PhD
Other Health Care Provider Rating:
( 0 Votes )
Rate This Article:
- Print This Email thisShare
January 10, 2011 — More than half of the current recommendations among the 41 current IDSA guidelines are based on evidence from expert opinion (level III) only; therefore, physicians should remain cautious when using these guidelines as their only source for making clinical decisions, a new report suggests.
Dong Heun Lee, MD, and Ole VIelemeyer, MD, from the Drexel University College of Medicine in Philadelphia, Pennsylvania, reported the results of their analysis in the January 10 issue of the Archives of Internal Medicine.
"In daily clinical work, practitioners sometimes assume that adhering to practice guidelines means practicing evidence-based medicine," the researchers write. "However, the quality of supporting literature can vary greatly."
A total of 4218 individual recommendations from 41 current IDSA guidelines, released between January 1994 and May 2010, were analyzed according to strength of recommendation and quality of supporting evidence. Although 43% (n = 1796) were considered strong (level A) recommendations, only 14% of these level A recommendations were supported by a strong quality of evidence (level I) such as randomized controlled trials. More than half were supported by expert opinions (level III) only.
The researchers noted that guidelines on surgical prophylaxis, travel medicine, and asymptomatic bacteriuria had the highest percentage of recommendations supported by level I evidence.
A comparison of 5 current IDSA guidelines that had recently been updated revealed a significant increase in the number of level I evidence recommendations in only 2 of the updated guidelines; most updated recommendations were supported by level II or level III evidence only.
The researchers listed difficulties of conducting large randomized controlled trials in the field of infectious diseases and the limitations of the IDSA evidence-grading system as 2 reasons for the limited number of recommendations supported by level I evidence.
The current IDSA guidelines "constitute a great and reliable source of information that should be used," the researchers conclude. However, in atypical cases they encourage "reviewing the primary literature and using one's clinical judgment rather than relying solely on recommendations."
To improve patient outcomes, the authors recommend more research in the form of well-designed and controlled clinical trials in areas where only low-level quality of evidence is available.
Guidelines Are A Starting Point, Not the Finish Line
John H. Powers, MD, from Scientific Applications International Corporation, Bethesda, Maryland, wrote an accompanying editorial, stating: "[G]uidelines may provide a starting point for searching for information, but they are not the finish line."
"Evaluating evidence is about assessing probability," Dr. Powers commented in a news release. "Perhaps the main point we should take from the studies on quality of evidence is to be wary of falling into the trap of 'cookbook medicine,' " Dr. Powers continues. "Although the evidence and recommendations in guidelines may change across time, providers will always have a need to know how to think about clinical problems, not just what to think."
The study was not commercially funded. The authors have disclosed no relevant financial relationships. Dr. Powers' work is supported by the National Cancer Institute and National Institute of Allergy and Infectious Disease, National Institutes of Health. He also reports receiving consulting fees from Acureon, Advanced Life Sciences, Astellas, AstraZeneca, Basilea, Centegen, Cerexa, Concert, Cubist, Destiny, Forest, Gilead, Great Lakes, Johnson & Johnson, LEO, Merck, Methylgene, MPEX, Pharming, Octoplus, Takeda, Theravance, and Wyeth.
Arch Intern Med. 2011;171:15-17, 18-22.
http://www.medscape.com/viewarticle/735511
3. Infectious Disease Recommendations Largely Based on Low-Quality Evidence
By: MARY ANN MOON, Internal Medicine News Digital Network
01/10/11
FROM ARCHIVES OF INTERNAL MEDICINE
More than half of the current recommendations in practice guidelines concerning infectious disease are based on evidence derived only from expert opinion or descriptive studies, according to a report in the Jan. 10 issue of the Archives of Internal Medicine.
Only 14% of the 4,218 individual recommendations included in 41 Infectious Diseases Society of America (IDSA) guidelines published in 1994-2010 are based on the highest-quality, or level I, evidence, such as that from randomized controlled trials, said Dr. Dong Heun Lee and Dr. Ole Vielemeyer of Drexel University, Philadelphia.
"Guidelines can only summarize the best available evidence, which often may be weak. Thus, even more than 50 years since the inception of evidence-based medicine, following guidelines cannot always be equated with practicing medicine that is founded on robust data," the investigators noted.
"Physicians and policy makers should remain cautious when using current guidelines as the sole source guiding decisions in patient care."
The study authors assessed the quality of evidence underlying 41 of the 52 IDSA guidelines currently available, which cover a wide range of topics and use an IDSA evidence-grading system. About half of these 41 guidelines are new and half are updates of earlier guidelines.
In addition to the highest-quality (level I) evidence, the IDSA grading system designates evidence from well-designed, but nonrandomized clinical trials, from cohort studies, from case-controlled analytical studies, or "dramatic results from uncontrolled experiments" as intermediate-quality (level II) evidence. The lowest-quality (level III) evidence is that "from the opinions of respected authorities based on clinical experience, descriptive studies, or reports of expert committees," the investigators said.
They identified 4,218 individual recommendations among the 41 guidelines that could be charted according to the strength of the recommendation and the quality of the evidence supporting it. Only 14% were supported by level I evidence, 31% by level II evidence, and 55% by level III evidence (Arch. Intern. Med. 2011;171:18-22).
For example, greater than 80% of the recommendations concerning blastomycosis, which were published in 2008, were based on level III evidence and did not have any level I support. The findings were the same for recommendations concerning sporotrichosis, which were published in 2007.
The investigators also assessed the extent to which the quality of evidence has improved over time by selecting five guidelines that had recently been updated and comparing them with their respective earlier versions. The updates did include evidence from more studies, as well as evidence from more recent studies, than did the earlier guidelines. "However, only two updated guidelines had a significant increase in the number of level I quality-of-evidence recommendations; most additional recommendations were supported by level II or III quality of evidence only," Dr. Lee and Dr. Vielemeyer said.
In addition, "we came across imprecisions on more than one occasion and for more than one guideline, including illogical, erroneous, or missing references for recommendations and their associated grades," they added.
These findings are particularly concerning because guidelines are used not only for decision making in clinical practice but also "as benchmarks in the appraisal of quality of care provision," they said.
"We believe that the current clinical practice guidelines released by the IDSA constitute a great and reliable source of information that should be used. However, in circumstances when patient outcome is less than desirable, or when colleagues use diagnostic or therapeutic choices not included in the recommendations, it is prudent to remember that many of the individual recommendations are not supported by solid evidence.
"In such cases, we encourage reviewing the primary literature and using one’s clinical judgment rather than relying solely on recommendations," they concluded.
Dr. Lee and Dr. Vielemeyer reported that they had no relevant financial disclosures.
Rating: 5.0 of 5. 1 vote(s).
Click the rating bar to rate this item.
VITALS
Major Finding:Only 14% of 4,218 individual recommendations in 41 Infectious Diseases Society of America clinical practice guidelines are based on level I evidence such as that from randomized clinical trials, while more than half are based on level III evidence, such as that from expert opinion or descriptive studies.
Data Source: A review of 41 current IDSA clinical practice guidelines aimed at assessing the quality of evidence on which each recommendation is based.
Disclosures: Dr. Lee and Dr. Vielemeyer reported that they had no relevant financial disclosures.
VIEW ON THE NEWS
View on The News
Practice Guidelines Are Only a Starting Point
"Perhaps the main point we should take from the studies on quality of evidence is to be wary of falling into the trap of ‘cookbook medicine,’?" said Dr. John H. Powers.
"The existence of guidelines is probably better than no guidelines, but guidelines will never replace critical thinking in patient care."
For clinicians, guidelines "may provide a starting point for searching for information, but they are not the finish line.
"As with individual research studies, providers should critically evaluate guidelines and the evidence on which they are based and how relevant recommendations are locally at their institutions and in their patients," he said.
Dr. Powers is with the division of clinical research at the Scientific Applications International Corp. (SAIC) in support of the National Institutes of Health. He reports receiving consulting fees from several pharmaceutical companies. These comments were taken from his editorial accompanying the report by Dr. Lee and Dr. Vielemeyer (Arch. Intern. Med. 2010;171:15-17).
Share ] [ Close Window ]
Contact: Rachel Sparrow
215-255-7328
Many recommendations within practice guidelines not supported by high-quality evidence
More than half of the recommendations in current practice guidelines for infectious disease specialists are based on opinions from experts rather than on evidence from clinical trials, according to a report in the January 10 issue of Archives of Internal Medicine, one of the JAMA/Archives journals.
"During the past half century, a deluge of publications addressing nearly every aspect of patient care has both enhanced clinical decision making and encumbered it owing to the tremendous volume of new information," the authors write as background information in the article. "Clinical practice guidelines were developed to aid clinicians in improving patient outcomes and streamlining health care delivery by analyzing and summarizing data from all relevant publications. Lately, these guidelines have also been used as tools for educational purposes, performance measures and policy making."
Interest has been growing in critically appraising not only individual guidelines but also the entire sets of guidelines for specialists and subspecialists, the authors note. Dong Heun Lee, M.D., and Ole Vielemeyer, M.D., of Drexel University College of Medicine, Philadelphia, analyzed the strength of recommendations and overall quality of evidence behind 41 guidelines released by the Infectious Diseases Society of America (IDSA) between January 1994 and May 2010.
Recommendations within the guidelines were classified in two ways. The strength of recommendation was classified in levels A through C, with A indicating good evidence to support the recommendation, B indicating moderate evidence and C indicating poor evidence; some guidelines also included levels D and E. The quality of evidence was classified in levels I through III, with level I signifying evidence from at least one randomized controlled trial, level II indicating evidence from at least one well designed clinical trial that was not randomized and level III indicating evidence was based on opinions of respected authorities based on clinical experience, descriptive studies or reports of expert committees.
The 41 analyzed guidelines included 4,218 individual recommendations. Of these, 14 percent were classified as backed by level I evidence, 31 percent as level II and 55 percent as level III. Among class A recommendations, 23 percent were level I and 37 percent were level III.
In addition, the researchers selected five recently updated guidelines and compared them to their previous versions. In all but one case, the new versions cited an increased number of articles, and in every case the number of recommendations increased. However, most of these additional recommendations were supported only by level II or III quality of evidence. Only two updated guidelines had a significant increase in the number of level-I recommendations.
There are several possible explanations for these findings, the authors note. In comparison to other specialties, relatively few large multicenter randomized controlled trials have been conducted in the field of infectious diseases. "Many infectious diseases occur infrequently, present in a heterogeneous manner or are difficult to diagnose with certainty," the authors write. "For others, a randomized controlled trial would be impractical or wasteful or might be deemed unethical." In addition, some of the recommendations address questions about diagnosis or prognosis, neither of which could be studied in a randomized controlled trial and thus could never receive the highest quality rating.
"Guidelines can only summarize the best available evidence, which often may be weak," the authors conclude. "Thus, even more than 50 years since the inception of evidence-based medicine, following guidelines cannot always be equated with practicing medicine that is founded on robust data. To improve patient outcomes and minimize harm, future research efforts should focus on areas where only low-level quality of evidence is available. Until more data from such research in the form of well-designed and controlled clinical trials emerge, physicians and policy makers should remain cautious when using current guidelines as the sole source guiding decisions in patient care."
(Arch Intern Med. 2011;171[1]:18-22. Available pre-embargo to the media at www.jamamedia.org.)
Editor's Note: Please see the article for additional information, including other authors, author contributions and affiliations, financial disclosures, funding and support, etc.
Editorial: Guidelines No Substitute for Critical Thinking
"What are providers to make of recommendations in guidelines if most of these recommendations are based on opinion? First, these data reinforce that absolute certainty in science or medicine is an illusion. Rather, evaluating evidence is about assessing probability," writes John H. Powers, M.D., of Scientific Applications International Corporation, Bethesda, Md., in an accompanying editorial.
"Perhaps the main point we should take from the studies on quality of evidence is to be wary of falling into the trap of 'cookbook medicine,'" Dr. Powers writes. "Although the evidence and recommendations in guidelines may change across time, providers will always have a need to know how to think about clinical problems, not just what to think."
"As with individual research studies, providers should critically evaluate guidelines and the evidence on which they are based and how relevant recommendations are locally at their institutions and in their patients," Dr. Powers concludes. "Especially for subspecialists, guidelines may provide a starting point for searching for information, but they are not the finish line. The fact that many recommendations are based on opinion should also serve as a call to future researchers to critically evaluate and study the questions that need better answers."
(Arch Intern Med. 2011;171[1]:15-17. Available pre-embargo to the media at www.jamamedia.org.)
Editor's Note: This project was funded in whole or in part with federal funds from the National Cancer Institute, National Institutes of Health, under contract and in part by the National Institute of Allergy and Infectious Disease. Please see the article for additional information, including author contributions and affiliations, financial disclosures, funding and support, etc.
###
To contact corresponding author Ole Vielemeyer, M.D., call Rachel Sparrow at 215-255-7328 or e-mail rachel.sparrow@drexelmed.edu. To contact editorial author John H. Powers, M.D., call Frank Blanchard at 301-846-1893 or e-mail blanchardf@mail.nih.gov.
For more information, contact JAMA/Archives Media Relations at 312/464-JAMA (5262) or e-mail mediarelations@jama-archives.org.
http://www.eurekalert.org/pub_releases/2011-01/jaaj-mrw010711.php
5.
Editorial
This Article
Practice GuidelinesBelief, Criticism, and Probability
John H. Powers, MD
Arch Intern Med. 2011;171(1):15-17. doi:10.1001/archinternmed.2010.453
•
•
•
•
•
•
Citing Articles
Since this article does not have an abstract, we have provided the first 150 words of the full text and any section headings.
•
Related Content
Every day, health care providers aid patients in making decisions about their health. The process by which providers acquire, assimilate, and implement information to make decisions involves evaluation of published clinical research studies and reliance on early medical training, discussions with colleagues, local policies, personal clinical experience, and external influences.1 Another important source of information is practice guidelines developed and published by professional medical societies.
Guidelines can serve a useful purpose for providers by presenting a compilation of available evidence in a given therapeutic area. Guidelines can also help the provision of care, because standardization may help streamline processes for implementation of interventions. In some circumstances, quasi-experimental studies show an association between following recommendations in guidelines and improved outcomes for patients.
•
•
•
•
•
•
•
Topic Collections
Social Bookmarking
However, guidelines are not just summaries of the evidence. They are also interpretations of that evidence by guidelineauthors who bring to the process their . . . [Full Text of this Article]
AUTHOR INFORMATION
Author Affiliation: Division of Clinical Research, Scientific Applications International Corporation (SAIC) in Support of NIH, Bethesda, Maryland.