Validation Institute will never criticize companies publicly, but nonetheless sometimes a company’s measurement errors make for a good lesson. Let’s face it, engineers learn more from one bridge falling than from 100 staying up. So, let’s do the same here.
We want to share a spot-on example of what exactly we have been seeing lately with performance measurements and market claims. We aren’t going to name names both because we don’t name names, and because our example is only one of many non-validated vendors that inadvertently will allow us to highlight important lessons.
Let’s map a few of the market claims from an example vendor to the fallacies which we also identified in our How To Tell If Your Vendor Claims Are Valid- A Nine-Part Series multi-part series.
Lesson One: Peer Review
Not all peer-review is created equal. Here is one of many examples where peer-review publications and articles come from unreliable sources. Take an article from a publication called JMIR Formative Research, which is one of 15,500+ “open access” journals that currently exist. These open-access journals are the opposite of journals like JAMA the articles and NEJM in that the authors pay for the articles, not the readers. We covered publishing in open-access journals in Part Six of our series. Keep in mind, when you read an article in an open-access journal that it is subject to nowhere near the level of peer review that authoritative journals insist on.
As a result of these lax publication standards, open-access journals have a low “impact factor,” which is journal-speak for their low citation numbers. JAMA’s impact factor is 63. NEJM’s is 96. Formative Research’s impact factor is 2.
So, don’t trust everything you read.
Lesson Two: Outcomes Measurement
Let’s look at this one sentence from a real vendor statement which demonstrates three fallacies we will cover.
1934 participants with a baseline HbA1c [sic] >8 were enrolled into the Program. Of those enrolled, 58.3% (1128/1934) provided follow-up laboratory HbA1c values
Part Two of the multipart series covers participation bias of cherry-picking participants and features several examples (from industry sources unsuccessfully attempting to prove the opposite) that willing participants will always outperform unwilling non-participants, even when there isn’t a program to participate in.
Also, 42% of those 1934 dropped out within three months, and 76% within a year – leaving only 455 stalwart souls who completed the program. Not counting the dropouts creates “survivor bias,” which is a special case of participation bias. Those few (24% in this case) who are willing to stick with a program will always as a cohort have better results than those who get discouraged and drop out. Here is the “funnel”:
Lastly, to qualify for this program in the first place, a participant needed to have an Hb A1c of at least 8%. In any population for any measurement, people who are high as a group (some individuals will be exceptions) will decline, while people who are low (again, as a group) will increase. This is classic regression to the mean, as covered in Part One of our multi-part series.
A former wellness company that is now defunct inadvertently proved this perfectly with this chart:
It’s a bit tough to read this chart because it’s a screenshot of a screenshot, and we can’t exactly ask them for a copy since they don’t exist anymore. But the bottom line is that low-risk people’s risk went up and high-risk people’s risk went down, as is always the case in a population.
Unlike this defunct company, the vendor who made the statement we shared above, along with many others vendors, only shows high-risk members declining, not low-risk members increasing. Combining this regression to the mean with the initial participation bias and then the follow-up survivor bias could cause one to wonder why they couldn’t reduce Hb A1c by more than 1.4 points among those active participants who stuck it out with the program for a full year.
Lesson Three: Savings Calculation
The savings claim is also driven by regression to the mean. The only way to save money in diabetes in the short term is by deprescribing insulin, GLP-1s, and other high-cost meds. You can’t get there by simply reducing Hb A1c. Especially not by only 1.4%. And yet…
How did they arrive at that number? Recall the discussion of regression to the mean: the worse the starting point, the greater the improvement: In this case, the more hospitalizations and ER visits people had in the baseline, the greater the reduction in hospitalizations and ER visits.
Just a minor clarification here: $2,040 in the previous display should not be labeled “Cost savings per member.” It should be labeled “Cost savings for the members with the highest utilization in the baseline, unadjusted for participation bias, survivor bias, and regression to the mean.”
Likewise:
This slide should similarly be more accurately labeled just like the above (substitute “heaviest” for “highest utilization”) with the added disclosure that this is after only one year for those stalwart souls, and most people who lose weight later regain it.
Lesson to be Learned
Don’t immediately believe what solution vendors tell you about their performance. There is a lot of information behind those claims which your consultants and your team may not be able to identify—and not for a lack of trying. A solution provider choosing to apply for Validation Certification through Validation Institute demonstrates their commitment to transparency and their confidence in the ability to achieve what they promise. So, look for Validated providers today or ask your solution providers if they have been Validated. For a list of Validated providers please refer here: https://validationinstitute.com/validation-reports-archive/.
