Inaccurate marketing claims and outcomes reports are proliferating. The Validation Institute has staked out a position as the leader in assisting/promoting vendors and consultants in the “Integrity Segment” of the healthcare services market.
How can you tell if your adviser is in the Integrity Segment? The easiest way: did they send you to this series or did you have to find it on your own?
Part One showed that regression to the mean was responsible for a large chunk of claimed savings, and one vendor even admitted it publicly. Part Two showed that separating a population into active, willing participants and comparing their performance to non-participants demonstrates exactly why the FDA nixed that study design in 1962.
Part Three will now explore the role of “trend inflation” in generating savings…and how to spot it as a fallacy. As with the other two, vendors have been known to accidentally admit that inflating a trendline is an easy way to show savings. Seth Serxner, of Optum, said it best (behind paywall):
“We can conclude that choice of trend has a large impact on estimates of financial savings.”
Of course it does. Vendors retain actuarial firms or consulting firms – preferably those that also consult to their customers – to show savings, not to determine whether there are savings in reality. And one of the easiest ways to show savings is to concoct a strawman trend inflating fast enough that you can, in Mr. Serxner’s words, “estimate financial savings” quite optimistically via a simple comparison to that trend.
Mr. Serxner returned to that theme during a taped debate. It was pointed out that despite the massive investment that employers had made implementing increasingly comprehensive and punitive wellness programs, the rate of wellness–sensitive medical events (WSME) in the employer–insured population had remained virtually unchanged, according to the Agency for Healthcare Research and Quality (AHRQ).
He suggested [2] that the rate would have increased dramatically otherwise, so that the WSME rate would have been much higher absent wellness programs. Unfortunately for him, the AHRQ also collects data on non-employer-insured patients, who by definition would not have access to employee wellness programs. It turns out that people without access to wellness programs trended slightly better than people with access. (That red line is higher in general because Medicaid, Medicare and uninsured populations have worse health than commercially insured people.)
Another example of trend inflation would be our friends from Part One: Wellsteps. Their wellness program – the subject of a detailed exposé in STATNews and The Boston Globe, claimed massive savings simply by drawing a trendline and crediting themselves for the difference between the “predicted” 15% cumulative increase and the allegedly “actual” 15% decrease.
There is a saying that “in wellness, you don’t have the challenge the data to invalidate it. You merely have to read the data. It will invalidate itself.” Wellsteps was the poster child for this observation in Part One on regression to the mean, as they admitted their fabrication.
This time they didn’t actually admit their fabrication. Their own data did. They published a chart, subsequently deleted, showing that costs per person increased over that period, from $3308 to $3330 for participants and slightly more for non-participants.
How to spot this fallacy
Look for reports where everything declines vs. “predicted.” For example, consider North Carolina Medicaid. North Carolina Medicaid had instituted a patient-centered medical home model through a vendor, Community Care of North Carolina (CCNC). CCNC paid several teams of actuaries, starting with Mercer, to show savings. The (subsequently deleted) Mercer report “showed” that per–member–per–month savings had been realized in every age group, with savings exceeding 50% per baby per month.
The way to know that each of these figures is fictitious is that while by far the largest savings was realized in babies, babies were not even eligible for the program, which was for adults only. (Neonatal days, by far the largest avoidable expense, also rose slightly. Those would have needed to dramatically decline for this result to be valid — assuming babies were included in the measurement, which they weren’t.)
Mercer also broke out the comparison by category of spending.
This is where your fraud-o-meter should be flashing red. Every category of spending can’t decline. If you insulate your house, you’ll save money…but not on insulation. Looking at this slide, ask yourself: “Where is the insulation expense?” Normally one attempts to reduce inpatient expense by shifting to outpatient, getting more office-based care, or increasing use of meds. In this case, everything declined.
Eventually the state of North Carolina was shown that none of these figures were accurate, and subsequently terminated the program.
Did Accolade do exactly the same thing?
Now that you’ve seen how trend inflation has worked in the past, try applying the same logic to Accolade’s savings claims, which are summarized below. By way of background, Accolade paid Aon to show savings, and Aon delivered. Aon drew a trendline and “found” that Accolade reduced utilization by 8.3% after only one year. They saved money across all conditions — including anxiety, mental mood, neurological disorders and hypothyroidism, four categories in which no other vendor has ever claimed savings.
They somehow also achieved large reductions in asthma, which no other vendor has ever achieved because the number of asthmatics you need to medicate in order to avoid an ER visit is much higher than the value of the ER visit theoretically avoided. Cancer cost was reduced 18% in the first year alone, and 26% after 2 years. Even cancer vendors don’t allege that.
Highlights of Aon’s Mischaracterization of Accolade Savings
As mentioned, we’d recommend reading the entire report, which Accolade has also reviewed and found no fault with. Here is a highlighted excerpt albeit a bit out of context, on the inappropriateness (pretty obvious from the result) on using a cherrypicked undisclosed control group instead of a clearly appropriate control group comprised of employees from the same organizations who almost certainly did not use the Accolade service due to good health. Clearly, the people who likely used the service (though higher cost overall) should trend much better than people who didn’t, if indeed the entire group achieved massive savings.
The baseline trend should not be one that Aon alleges would have happened anyway based on some cherrypicked random unnamed companies. Rather, it should be the trend of the 0 (or 0 and 1) morbidity categories in the companies that signed up with Accolade. There is virtually nothing that care navigation/support can do for people who mostly (Study Group 1) or totally (Study Group 2) don’t need care navigation/support because they aren’t spending money on high-cost care to begin with.
But everything else is the same if the same companies are used as the controls, making “same companies” the correct control group. You are holding everything constant except the possibility of benefiting from care navigation. Isolating those who would qualify for care navigation allows the reader to see the impact of care navigation when everything else is held constant.
The central tenet of a “control group” – which Aon would know if they’ve read either of my award-winning books or taken a basic course (mine or anyone else’s) in study design – is to be “identical in all respects” except the respect you are evaluating. Using cherrypicked anonymous random companies as a control is exactly the opposite. Here is an easily understood table that explains how to do basic case-control analysis, using the “identical in all respects” standard:
The percent increase in costs in the zero (or 0-1, in Accolade’s first cohort) morbidity group would therefore be the obvious proxy for the likely percent increase in costs of the comorbid categories if they did not have access to Accolade’s support.
Using the same-company control group methodology yields the following:
“Savings” in Control Group doing nothing | “Savings in Support-sensitive group | |
First Cohort | 8.3% | 8.4% |
Second Cohort | 7.0% | 6.6% |
Quantum vs. Accolade
We’ll leave it to the reader to decide whether Aon’s analysis is valid, or whether it exemplifies Seth Serxner’s admission that “the choice of trend” determines your claimed savings.
Quantum and Accolade are direct competitors. The difference is that Quantum is validated by the Validation Institute. We have reviewed their data carefully…and stand behind our findings with our $25,000 Credibility Guarantee payable to Quantum clients who feel misled by our validation. To our knowledge, no consultants or actuaries stand behind their reports in this manner. Nor has any consultant or actuary ever failed to “find savings” when paid to do so, which is one reason none offers a credibility guarantee.
And if you relied on this misrepresentation and would like to get your money back, we are happy to perform a forensic review on Accolade. Indeed, we just did.
Endnotes:
Page1:
- first installment: https://validationinstitute.com/blog/how-to-tell-if-your-vendors-claims-are-valid-part-one/
Page2:
- https://www.ajmc.com/view/do-wellness-outcomes-reports-systematically-anddramatically-overstate-savings
- https://thehealthcareblog.com/blog/2015/12/16/genetic-testing-the-new-frontier-ofwellness-madness/
Page 4:
- https://www.washingtonpost.com/national/health-science/nortin-hadler-author-ofseveral-books-on-medical-overtreatment-turns-his-attention-to-what-he-calls-themedicalization-of-aging/2012/02/03/gIQAYoUnPR_story.html
- http://thehealthproject.com/wp-content/uploads/2017/10/EastmanEval-original.pdf
Page 5:
- https://validationinstitute.com/certifications/
- https://theincidentaleconomist.com/wordpress/workplace-wellness-programs-dontsave-money/
- https://www.healthaffairs.org/do/10.1377/hblog20141222.043494/full/
Page 6:
- https://journals.sagepub.com/doi/10.4278/ajhp.130731-LIT-395
- https://www.cnbc.com/2019/04/16/for-millions-of-workers-into-wellness-tests-are-backand-not-good.html
- https://www.nber.org/papers/w24229
- https://experts.illinois.edu/en/publications/effects-of-a-workplace-wellness-program-onemployee-health-health
Page 8:
- https://validationinstitute.com/wp-content/uploads/2021/04/Part-3-How-to-tell-if-yourvendors-claims-are-valid.pdf
