This is Part Two of the teaching tool that Aon was gracious enough to provide (See Part One here), and consent to our usage. It is part of the Advanced Critical Outcomes Report Analysis (CORA Pro) curriculum, and we have encouraged Validation Institute (VI)-validated entities and individuals to share it with others, to increase the general level of sophistication in outcomes measurement. We are also encouraging Aon to distribute it internally, and have offered them a discount on the CORA Pro course, in order to guide their actuaries in their professional development.
Ideally, Aon’s report and this teaching tool should be read in conjunction with Why Nobody Believes the Numbers (John Wiley & Sons, 2012). This book is universally considered the seminal textbook on outcomes measurement. Why Nobody Believes identifies and even anticipates the questions that should be asked about this report.
Why Nobody Believes sets forth the Seven Rules of Plausibility, starting on Page 57. We recommend asking Aon to reconcile their methodology and conclusions about outcomes with some of those Seven Rules.
I: Questions about the savings figures themselves
The participants-vs-non-participants study design would appear to violate Plausibility Rule #2: Control Group Equivalency, by comparing the study cohort to “matched controls.” The latter are people who look the same as the study group on paper but likely have a much different frame of mind, that can’t be controlled for. The need to ask questions about matched-control equivalence as regards the study design itself was addressed in Part One.
Part Two is more about the plausibility of the outcomes this study design generated. The way to find out if this study design was actually valid is to ask Aon these questions about their non-participant control group:
- How does a minimum of one phone call with a mental health provider reduce overall spending by almost a third? Wouldn’t this violate Plausibility Rule #3, which is that no voluntary behavior-change program can reduce costs by more than 25%?
- Wouldn’t costs increase at least initially due to the expense of these provider interactions?
- How is it that the “efficiency ratio” (See Part One for Aon’s unique definition) deteriorates after the first year? Shouldn’t mental health counseling have a more positive impact over time as more sessions take place?
- Can you comment on why, over this period, participant spending rose 40% faster than non-participant cost? (31% vs 22%)? Shouldn’t it be the opposite?
II: Specific Impacts on Elements of Cost
Plausibility Rule #4 is the Nexus Rule: There must be a connection between the type of intervention and the source of the savings. The best example of a violation of Rule #4 was a claim by a wellness vendor that workplace cholesterol and blood glucose screenings reduced “diseases of the blood and blood-forming organs.” In a follow-up question at a conference, the author of the study was unable to name one such disease other than hemophilia.
Aon made the following claims that appear to violate this Rule. We are not saying that they do violate this rule, just that they appear to. It is up to you to ask Aon to explain.
- While it is understandable that mental health ER visits declined (though a reduction of almost 90% seems like a stretch considering that most of Lyra’s providers can’t prescribe drugs), how did you achieve a reduction of 64 visits/1000 in non-mental health ER visits?
Likewise, how were you able to achieve a 45% reduction in specialty drug spend by 2021, when Lyra’s talk therapists are not trained in pharmacology, can’t prescribe, don’t influence drug prices, and didn’t go to medical school? A reduction of this magnitude would appear to violate Plausibility Rule #3 (large declines cannot be traced to voluntary programs) as well as Rule #4.
Like Lyra, we are not experts by any means on specialty drugspend. We would recommend contacting arguably the country’s leading expert on this topic, pramod@viviohealth.com, to get his opinion.
III: Dose and Response of Counseling
In pharmacology and epidemiology, one tests the response of the body to the drug or chemical being studied. Almost invariably, a dose-dependent response is shown, though here is an example showing the opposite, in an alcohol study. Whichever way it goes, it is a basic tenet of health services research to find the optimal dosing. Or as it is said: “The difference between remedy and poison is dose.”
While no one is going to be poisoned by a point solution, there is a corollary in the Seven Rules of Plausibility Rule #5 starts by stating that costs and utilization can’t decline faster than the “dose” of the intervention. in this case, spending on or utilization of drugs, emergency visits, and mental health services other than Lyra – against the “dose” of Lyra. Curiously, in this case, the costs declined the most in the first year.
More importantly, Rule #5 also states that the greater the dose of the intervention, the greater the response. In this case, that would mean that people who had more visits over time would have lower costs than people who had fewer visits.
Aon segmented the findings by age, modality, drug use and more, but not by the single most meaningful comparison: number of visits with the providers. Clearly, more visits should mean more impact. Or maybe, as in the linked alcohol study, it doesn’t, in which case Lyra would be having no impact on costs, other than the first call perhaps, causing the 31% reduction in overall spending in 2018.
A cynic might conclude that this omission from the Aon study is the “dog that didn’t bark in the nighttime,” and was deliberately excluded because there was no dose-response correlation. We are more likely to attribute this omission to Hanlon’s Razor. After all, no one from Aon has attended any of our in-person workshops, taken either of our Critical Outcomes Report Analysis courses, or even (to our knowledge) read Why Nobody Believes the Numbers.
So the perfectly innocent explanation is that they didn’t realize they were supposed to do that. Hence we are not jumping to a conclusion, but we do recommend asking Aon about this omission to draw your own.
