Belter 2016

Appraisal of: "Belter CW. Citation analysis as a literature search method for systematic reviews. J Assn Inf Sci Tec 2016; 67(11): 2766-2777."

Reviewer(s):

Elke Hausner

Ruth Wong

Full Reference:
Belter CW. Citation analysis as a literature search method for systematic reviews. J Assn Inf Sci Tec 2016; 67(11): 2766-2777.

Short description:

The author proposed a method of searching the literature by systematically mining the various types of citation relationships between articles (cited by, citing, cociting, cocited articles).

The author tested the method by comparing its precision and recall to that of 14 published systematic reviews. The method successfully retrieved 74% of the studies included in these reviews and 90% of the studies it could reasonably be expected to retrieve. Indirect citation relationships clearly outperformed direct citation relationships in retrieving relevant publications. The method also retrieved fewer than half of the total number of publications retrieved by these reviews and can be performed in substantially less time.

Limitations stated by the author(s):

“This method may be less effective for disciplines and topics in which citations are not numerous”

“The proposed method may have also retrieved additional relevant publications that were not identified as such because I did not screen all 6,175 publications that it retrieved”

Although the method as implemented here performed relatively well, I made several arbitrary choices in implementing it in this analysis. My decision to use two key studies on which to base the method was one of these arbitrary choices.

Other arbitrary choices in this method were the thresholds I used to filter the cociting and cocited articles.

“This analysis clearly demonstrated that certain kinds of studies can rarely be identified using citation relationships. This is due both to the citation behavior of authors and the limitations of citation databases. Because authors rarely cite conference proceedings, and even more rarely cite studies in progress, it is extremely difficult for the proposed method to identify studies published or made available in these formats. Further, because these publication formats rarely include indexed cited references, the method is even less likely to identify them. This limitation means that the proposed method is even more prone to publication bias than the traditional method, meaning that its ability to identify descriptions of negative results is minimal. In addition, studies published in journals not indexed by the Web of Science were rarely identified by the proposed method. This means that certain kinds of studies, especially studies published in non-Western journals like the traditional Chinese medicine review excluded from this analysis, will rarely be identified by this method. Although using Scopus instead of, or in addition to, the Web of Science may be able to identify some of these publications, there are still a large number of journals not indexed by either database. The proposed method would be unlikely to retrieve studies published in such journals. Because of these limitations, it is unlikely that the proposed method could ever replace text-based searching for systematic reviews.”

“Although the method was designed for and tested in biomedicine, additional studies

would be needed to determine if the method is appropriate for literature retrieval in other disciplines.”

Limitations stated by the reviewer(s):

No further limitations

Study Type:

Single study

Related Chapters:

Tags:

• pearl growing

• citation checking

• databases (Web of Science)