The presence of spin, or overinterpretation, in scientific papers that ultimately demonstrate statistically insignificant findings had been gnawing away at Thomas Forbes, MD, for quite some time.
The practice, the University of Toronto vascular chair reflects, is by no means unique to vascular surgery but would often spring to the forefront of his mind as he pored over randomized controlled trials (RCTs) comparing endovascular aneurysm repair (EVAR) to open repair, and carotid stenting to carotid endarterectomy (CEA).
In the case of EVAR vs. open repair, he recalls, he would see trial researchers with nonsignificant studies declare the former “no better” than the latter. “So, the new kid on the block is no better than the gold standard,” he says. “By saying no better, that has a negative connotation.” On the other hand, he would see authors involved with carotid stenting trials write that the new kid on the block—in this case stenting—was “just as good as” the gold-standard CEA. “Whereas,” Forbes points out, “statistically there was no difference. There was a positive connotation in one aspect, and a negative in the other.”
The problem percolated in his mind, and together with a team made up of two medical students and one cardiothoracic resident, he set about establishing the extent of the problem in an analysis of abdominal aortic aneurysm (AAA) and carotid stenosis studies.
What they found was that a large proportion of statistically nonsignificant RCTs yielded interpretations that were inconsistent with their results, and the research team disseminated their findings last year in a poster presentation at the 2022 Vascular Annual Meeting (VAM) in San Diego and again at the virtual annual meeting of the Canadian Society for Vascular Surgery (CSVS). They identified spin in nine abstracts and 13 main texts among 18 AAA articles, and seven abstracts and 10 main texts across 13 carotid stenosis studies.
In reality, Forbes says, whether findings have a positive or negative connotation should be left to the interpretation of the reader. “As a reader, I should say, ‘With the new treatment, should I be disappointed that it is no better, or should I be happy that it’s just as good?’”
The researchers further looked at whether overinterpretation might be related to the publishing journal’s impact factor, the source of trial funding, and where exactly in the papers the spin occurred. No significant association was detected in terms of funding source, and increasing journal impact factor was associated with a statistically significant lower likelihood of spin in the study title or abstract conclusion, the team reported. As for location, overinterpretation was “very common” in discussion sections, says Forbes.
“But I think more importantly, it was also quite prevalent in the titles as well as the abstract conclusions. Why I say that’s very important is I think readers of the scientific literature are deluged with massive amounts of information. And I’m sure most of us, if we’re honest, in our everyday practice, there’s very few papers that we read from front to back. There’s maybe a bit more where we actually read the entire abstract, and there’s maybe a bit more that we actually just read the conclusion of the abstract. So, spin, or overinterpretation of findings, in abstract conclusions and in titles has much more impact than the spin that occurs buried in the paper.”
It is incumbent on peer reviewers and journal editors to hold authors accountable for their wording so that overreach does not occur, Forbes muses. “As an associate editor of the Journal of Vascular Surgery myself, that’s one of the big things that we go back and say to the authors,” he explains. “We say, ‘Look, you had a negative finding. Don’t give that negative or positive connotation. Just say you were unable to find a change.’
“I think it is up to the peer review process to be the gatekeepers of the language and hold it true to the science. Remember that these are supposed to be scientific investigations and not necessarily opinion pieces. I think sometimes we err too much in that respect.”
Words matter, Forbes concludes. “Try not to become a salesman for your science. Let it speak for itself.”
