Inaccurate marketing claims and outcomes reports are proliferating. The Validation Institute has staked out a position as the leader in assisting/promoting vendors and consultants in the “Integrity Segment” of the healthcare services market.
How can you tell if your adviser is in the Integrity Segment? The easiest way: did they send you to this series or did you have to find it on your own?
Behind the scenes of peer review
Often, vendors will brag about being peer-reviewed. Most prospects of vendors will then assume that the data was carefully vetted and reviewed by independent highly qualified third parties before seeing the light of publication because, after all, no journal would ever publish an article that was obviously flawed, right?
Well, certainly not for free.
It turns out most journals now charge for publication, and good luck finding any disclaimers to that effect in their published articles. Even airline magazines, where the expectation of objectivity is quite low to begin with (“Waco: Visit the Texas Panhandle’s Undiscovered Jewel”), multi-page spreads are marked “(advertisement)” on every page.
Not so in clinical research. A whole industry of journals with distinguished-sounding names all using various combinations of basically the same words, like Journal of Medical Economics, has sprung up to satisfy the demand for articles by vendors who want to purchase the ability to say they are peer-reviewed.
Let’s look at that very same Journal of Medical Economics (JME). Here is their rate card.
They aren’t the only ones. Here is a list of other journals offering the same thing.
You might say: “Well, that’s a big list but now that we know articles in these journals are tainted, we’ll just trust other peer-reviewed journals instead.”
Haha, good one. This list is just from one single publisher. This field has attracted multiple for-profit publishers, each publishing journals with more erudite-sounding titles than the others. Fact is, the vast majority of journals are now profit-making enterprises, and the vast majority of “peer-reviewed” articles are bought-and-paid-for by vendors, instead of the readers. Most of the resulting articles are known as “open-access,” for just that reason—they’re free to read.
Let’s dig a little deeper to see what this means in practice, using the JME as an example. And let’s see what JME published for Livongo, that the field’s most respected general-interest (non-profit!) journals — New England Journal of Medicine, the Lancet, or the Journal of the American Medical Association — would never do.
First, they published an article comparing participants to non-participants, a fallacy debunked in the first installment of this series. You could scan the entire databases of those three journals without finding a single article using non-participants as a control for participants. Nor does the Validation Institute validate such programs without a major disclaimer announcing that much or most of the differences between the two cohorts in these outcomes reports can be traced to that study design, rather than the intervention.
Next, the supplier of Livongo’s diabetes strips funded the study. Livongo claims that its outcomes are in large part due to more Type 2 diabetics checking their glucose much more often, a practice that requires using more strips. This conflict of interest was not disclosed in the article.
A headline about this study, with (presumably) unintended irony, announces the result:
- New England Journal of Medicine: 74.6
- Journal of the American Medical Association: 5
- The Lancet: 4
- Journal of Medical Economics: 8
The take-aways from Parts Five and Six
Previous installments [1][2][3][4] have cautioned to watch out for words and phrases like: “participants” “matched” “high-risk” “as compared to trend” We can now add to that list more trigger words/phrases:- “We can’t get the data”
- actuaries
- peer-review
- and even (with a small “v”) validated
Endnotes:
Page1:
- first installment: https://validationinstitute.com/blog/how-to-tell-if-your-vendors-claims-are-valid-part-one/
Page2:
- https://www.ajmc.com/view/do-wellness-outcomes-reports-systematically-anddramatically-overstate-savings
- https://thehealthcareblog.com/blog/2015/12/16/genetic-testing-the-new-frontier-ofwellness-madness/
Page 4:
- https://www.washingtonpost.com/national/health-science/nortin-hadler-author-ofseveral-books-on-medical-overtreatment-turns-his-attention-to-what-he-calls-themedicalization-of-aging/2012/02/03/gIQAYoUnPR_story.html
- http://thehealthproject.com/wp-content/uploads/2017/10/EastmanEval-original.pdf
Page 5:
- https://validationinstitute.com/certifications/
- https://theincidentaleconomist.com/wordpress/workplace-wellness-programs-dontsave-money/
- https://www.healthaffairs.org/do/10.1377/hblog20141222.043494/full/
Page 6:
- https://journals.sagepub.com/doi/10.4278/ajhp.130731-LIT-395
- https://www.cnbc.com/2019/04/16/for-millions-of-workers-into-wellness-tests-are-backand-not-good.html
- https://www.nber.org/papers/w24229
- https://experts.illinois.edu/en/publications/effects-of-a-workplace-wellness-program-onemployee-health-health
Page 8:
- https://validationinstitute.com/wp-content/uploads/2021/04/Part-3-How-to-tell-if-yourvendors-claims-are-valid.pdf
