How a Twitter HIBAR ends up as a published letter to the editor

Post date: Feb 17, 2014 6:53:27 AM

I’d like to share a fun Twitter discussion I had a month ago, and which lead to a letter to the editor that will be published in the near future in the Journal of Nervous and Mental Disease. One the 14^th of January, Keith Laws posted the following question on Twitter:

My response, underneath Keith’s message in the same screenshot, was that there was clearly something wrong with this table:

Keith found this table in a paper by Douglas Turkington and colleagues which reported an exploratory trial for cognitive behavioral techniques. At that moment, I didn’t even follow Keith on Twitter, but my Twitter buddy Åse (who wrote a blog post about the paper here: http://asehelene.wordpress.com/2014/01/15/through-the-looking-glass-into-an-oddly-analyzed-clinical-paper/) did, and if she shares something, it’s almost always interesting (#FF @asehelene). I don’t know much about the topic of cognitive behavioral therapy, but I have some interest in calculating and reporting effect sizes, so I’d thought I’d take a look.

Pretty soon, Tim Smits joined in:

as well as Stuart Richie:

As you can see, we were both joking around, as well as being amazed how this level of statistics reporting made it through peer review. In more formal terms, what we did was post-publication peer review (PPPR), or talking about the paper as a HIBAR (Had I Been A Reviewer). Note that Table 2 is all we have: there are no test statistics, means can only be gauged from Tables (that sometimes have no labels), and the 95% CI and error bars that are reported cannot possibly be correct.

Tim Smits (who had written a similar letter to the editor for another paper) took the lead and drafted a first letter. After we all added our comments, we submitted the letter (check out Tim’s blog for more about the content of the letter, as well as some things that we excluded from the letter for brevity’s sake), only to receive the message that our letter was accepted for publication a few days later. The original authors will probably publish a rejoinder (obviously, we are all very curious).

Now I understand that getting criticism on your work is never fun. In my personal experience, it very often takes a dinner conversation with my wife before I’m convinced that if people took the effort to criticize my work, there must be something that can be improved. What I like about this commentary is that is shows how Twitter is making post-publication reviews possible. It’s easy to get in contact with other researchers to discuss any concerns you might have (as Keith did in his first Tweet). Note that I have never met any of my co-authors in real life, demonstrating how Twitter can greatly extend your network and allows you to meet interesting and smart people who share your interests. Twitter provides a first test bed for your criticisms to see if they hold up (or if the problem lies in your own interpretation), and if a criticism is widely shared, can make it fun to actually take the effort to do something about a paper that contains errors.

It might be slightly weird that Tim, Stuart, and myself publish a comment in the Journal of Nervous and Mental Disease, a journal I guess none of us has ever read before. It also shows how Twitter extends the boundaries between scientific disciplines. This can bring new insights about reporting standards from one discipline to the next. Perhaps our comment has made researchers, reviewers, and editors who do research on cognitive behavioral therapy aware of the need to make sure they raise the bar on how they report statistics (if only so pesky researchers on Twitter leave you alone!). I think this would be great, and I can’t wait until researchers from another discipline point out statistical errors in my own articles that I and my closer peers did not recognize, because anything that improves the way we do science (such as Twitter!) is a good thing.