Janssens 2015

Appraisal of: "Janssens, AC, Gwinn M. "Novel citation-based search method for scientific literature: application to meta-analyses." BMC Med Res Methodol 2015 15: 84."

Reviewer(s):

Elke Hausner
Ruth Wong

Full Reference:
Janssens, AC, Gwinn M. Novel citation-based search method for scientific literature: application to meta-analyses." BMC Med Res Methodol 2015 15: 84.

Short description:

In two independent studies, the authors aimed to reproduce the results of literature searches for sets of published meta-analyses (n = 10 and n = 42). For each meta-analysis, they extracted co-citations for the randomly selected ‘known’ articles from the Web of Science database, counted their frequencies and screened all articles with a score above a selection threshold. In the second study, the authors extended the method by retrieving direct citations for all selected articles.

In the first study, they retrieved 82 % of the studies included in the meta-analyses while screening only 11 % as many articles as were screened for the original publications. Articles that were missed were published in non-English languages, published before 1975, published very recently, or available only as conference abstracts. In the second study, they retrieved 79 % of included studies while screening half the original number of articles.

Citation searching appears to be an efficient and reasonably accurate method for finding articles similar to one or more articles of interest for meta-analysis and reviews.

Limitations stated by the author(s):

We evaluated the performance of our method conservatively by assuming that the original meta-analysis were comprehensive and complete.

Clearly, the method does not work when the topic of the meta-analysis is heterogeneous and the studies of interest are unlikely to have cited each other. The second study also included several meta-analyses with very small sample sizes, including one in which half of the studies were case reports that had few or no references [21], as well as a meta-analysis for which the ‘known’ studies were cited only four times in total [22]. The percentage of retrieved studies jumped to 89 % when these five metaanalyses were excluded.

The accuracy and efficiency of co-citation searching depends on characteristics of the underlying citation network. By design, our method misses the studies that the collective community of researchers apparently did not find worth citing. In our analysis, these included abstracts, articles in non-English languages, very old articles, and publications in semi-scientific journals, reports, websites, and theses. In addition, some newer and some very old articles were not cited often enough to rank high in our search.

Limitations stated by the reviewer(s):

The method was only applied to meta-analysis. The authors did not investigate whether the approach would have found studies missed and should have been included in the meta-analysis.

The authors performed 2 independent studies with slightly different methods (e.g. selection of the meta-analysis, selection thresholds used and analysis of only co-citations in study 1, analysis of both co-citations and “direct citations” in study 2). It is not clear in the discussion, which method (co-citation and/or direct citations) the authors recommend for further use. The impact of studies not found by these methods on the findings on the meta-analysis are unknown.

Study Type:

Single study

Related Chapters:

Tags:

• citation searching

• citation checking