Inaccurate References; Guile, Duplicity, and Perfidious Publications

Good scientific writing is characterized by clarity of thought and often buttressed by appropriate peer reviewed references. This is of extreme importance when the scientific writing is that of consensus guidelines. But are references always faithfully quoted and even if that is true, are the original references in fact accurate and truthful?

Consider the recent AHA / ACC Guidelines for Coronary Artery Revascularization. According to these guidelines, class 1 recommendation arises when benefits >>> risks and class 2a when benefits >> risks. In the section dealing with left main disease, we find the above recommendation.

What is the evidence supporting the claim “PCI is reasonable to improve survival”? Reference 9 is a Bayesian cross-design and network meta-analysis of 12 studies (4 RCTs and 8 observational studies) comparing CABG with PCI and of 7 studies (2 randomized clinical trials and 5 observational studies) comparing CABG with medical therapy. This 2013 publication had a primary outcome of 1 year mortality and concluded;
1) no mortality difference between CABG and PCI
2) medical therapy is associated with higher 1-year mortality than PCI

So while the guideline writers have accurately quoted the reference, the actual publication seems more smoke and mirrors than hard evidence. Accepting the authors’ results that the posterior median for the comparative PCI / CABG mortality summary OR was 1.04 with 95% Bayesian credible interval 0.74 to 1.39 does not however provide evidence for concluding there is no mortality difference between the 2 techniques – unless, as first observed by the late George Diamond, the original Bayesian cardiologist, one believes potential relative mortality increases from 26% in favor of one technique to 39% in favor of the other are clinically unimportant.

The inability of these results to lead to definitive conclusions is illustrated by the following plot of this probability density function.

This demonstrates that the probability of increased mortality with PCI is 60% and 25% probability that the mortality OR is increased by at least 15% compared to CABG.

Moreover, there are other notable limitations to this publication which one may have expected an illustrious group of guideline writers to have noticed. First according equal weighting given to RCTs and observational studies may be questioned. Second an important quality element in a network meta-analysis is to assure consistency between the observed direct effects and the calculated indirect effects. However according to their figure below, there were in fact NO studies, observational or randomized, comparing PCI to medical therapy rather this OR was calculated only by considering the indirect evidence coming from studies comparing CABG to medical therapy and CABG to PCI.

Another major limitation to the use of this publication as a reference is that the outcome was mortality at 1 year and longer-term mortality is clearly important and now available from 5-year follow-up studies from 4 RCTs. Why were these very available 5 year comparative data not considered? Could it be because new Bayesian analyses of these RCTs show compared to CABG a 85% probability of increased mortality with PCI & 98% probability > 4 extra composite events / 100 PCI treated patients.

A “squeezing” of the difference between CABG and PCI for left main disease is also occurring via a downgrading of the strength of the CABG recommendation to level of evidence (LOE) B-R, defined as a moderate quality single RCT or moderate quality meta-analysis. In fact reference 10 supporting a CABG mortality benefit over medical therapy comes from a meta-analysis of 7 RCTs including 3 large ones of undisputable quality. The LOE supporting CABG in left main should definitively have been scored A. One could therefore reasonably argue that there has been a systematic misuse of references to attempt to narrow the difference between CABG and PCI in the treatment of left main disease.

More disconcerting than the questionable interpretation of the left main data in the body of the guideline text, is the slippery path towards complete equivalence between the revascularization techniques as articulated on page 1 of the “Top 10 Take-home Messages” in the guidelines document. Message #3 casts aside all pretense to evidence, objectivity and nuance stating “Percutaneous revascularization is a reasonable option to improve survival, compared with medical therapy, in selected patients with low to medium anatomic complexity of coronary artery disease and left main disease that is equally suitable for surgical or percutaneous revascularization.”