Korevaar 2020

Appraisal of: Korevaar DA, Salameh JP, Vali Y, Cohen JF, McInnes MDF, Spijker R, Bossuyt PM. Searching practices and inclusion of unpublished studies in systematic reviews of diagnostic accuracy. Res Synth Methods. 2020. DOI: 10.1002/jrsm.1389

Reviewers:

Caroline Higgins

Full reference:

Korevaar DA, Salameh JP, Vali Y, Cohen JF, McInnes MDF, Spijker R, Bossuyt PM. Searching practices and inclusion of unpublished studies in systematic reviews of diagnostic accuracy. Res Synth Methods. 2020 Jan 25. DOI: 10.1002/jrsm.1389

Short description:

There have been numerous studies reporting that anywhere from 25%-50% of diagnostic accuracy studies do not reach full-text publication in peer-reviewed publications, which may negatively impact systematic reviews of diagnostic accuracy (DTA-SR) by introducing biases if sources searched do not include unpublished studies (e.g., conference proceedings and trial registry data).

The aim of this study was to better understand the information retrieval practices among recently published DTA-SRs, specially the practice of searching for unpublished studies. Authors created two sets of DTA-SRs for this analysis: Cochrane DTA-SRs (n=100) and non-Cochrane DTA-SRs (n=100) and reviewed the publications to identify whether efforts were made to identify unpublished studies and whether unpublished studies were included.

Results indicate suboptimal searching practices in both sets:

Cochrane DTA-SRs set:

42 reported they considered or searched for unpublished studies for inclusion (10 reported they excluded non-peer-reviewed literature)

48 did not transparently report whether they did/did not consider unpublished studies, although 35 of these reported searching in sources of unpublished studies and 9 included at least one unpublished study in the analyses

Overall, of the 2,780 included primary studies in this set, unpublished studies represented 2.3% (64/2780)

Non-Cochrane DTA-SR set:

10 reported they considered or searched for unpublished studies for inclusion (36 reported they excluded non-peer-reviewed literature)

23 studies referred to unpublished or grey literature in general

54 did not transparently report whether they did/did not consider unpublished studies, although 13 of these reported searching in sources of unpublished studies and 10 included at least one unpublished study in the analyses

Overall, of the 2,082 included primary studies in this set, unpublished studies represented 1.9% (39/2082)

It is important to note that due to this study’s sampling methods, the authors did not perform a formal statistical comparison between the Cochrane and non-Cochrane sets.

Limitations stated by the study author(s):

The authors acknowledge that this study relies on the reporting of methods of the included DTA-SRs, which research has demonstrated often fail to include necessary information. They also acknowledge that there are different understandings and definitions of “unpublished studies”. To illustrate, they did find DTA-SRs which reported excluding unpublished studies, but in fact included conference abstracts. Finally, the included DTA-SR sets were created using convenience sampling and timeframes differed, which, as mentioned above, prevented the authors from conducting a statistical comparison between the Cochrane and non-Cochrane sets.

Limitations stated by the reviewers:

It was unclear why the authors decided to use convenience sampling techniques to create the two DTA-SR sets rather than via random sampling.

SuRe Info keywords (pre-defined tags):

diagnostic test accuracy studies

Study type:

Single study