How to tell if your vendor’s claims are valid: Part Five

Inaccurate marketing claims and outcomes reports are proliferating. The Validation Institute has staked out a position as the leader in assisting/promoting vendors and consultants in the “Integrity Segment” of the healthcare services market. 

How can you tell if your adviser is in the Integrity Segment? The easiest way: did they send you to this series or did you have to find it on your own?

Part Five covers questions to ask the vendor. It is best to ask them in person, but in an RFP as well.

Then Part Six will deconstruct the latest fad, which is vendors getting themselves peer reviewed in academic journals. You will learn to distinguish true peer reviewed publications from ads disguised as peer review. Peer review itself has come under fire, too. That means you should consider even the best peer review to be a minimum threshold, rather than a substitute for your own critical thinking.

Questions to ask the vendor

[Spoiler Alert] The first question is specifically about the Validation Institute.

(1) “Are these outcomes claims validated by the Validation Institute?”

A full list of validated organizations can be found here. In descending order, the responses you will get will be as follows:

·      “Yes.”

·      “We have already contracted for validation and it should be completed soon.”

·      “We aren’t validated by them because no one has ever asked us to.” (A good rejoinder here is: “So you are proposing that we be penalized because no one knows what questions to ask you?”)

·      “We’ve been validated by actuaries.”

The last one takes us right into Question #2.

(2) “Do your actuaries warrant their work, so that they will agree in writing to be held responsible if indeed it can be shown to be invalid?”

Few actuaries ever stand behind their work. (The Validation Institute does, of course, with its Credibility Guarantee.)

As in this hilarious three-part series, vendors pay actuaries to “show savings,” not to determine whether indeed there are savings. Who would hire an actuary who wouldn’t support their product? Or, to put it another way:

“You know that point solution you paid me to validate? It doesn’t save money,” said no actuary ever.

If in fact the vendor insists that the actuaries did this validly, ask to talk to the actuaries. You will find that they relied upon at least one of the fallacies in the first three installments [1] [2] [3] of this curriculum:

Comparing participants to “matched” non-participants

Only measuring people who pre-identified as having the condition in question or being high-risk or having a high Hb a1c to begin with

      Measuring against “trend”

You will also probably also find that there is no relationship between the change in health status and the change in claimed costs., the subject of Part Four in this series. Aon’s analysis of Lyra is a perfect example of this. (Lyra is no longer VI-validated, nor could they be, without renouncing this report.). Or they deliberately did the math wrong, as in Aon’s analysis of Accolade.

But the next question will determine whether the emperor has no clothes. This may be asked of whoever did the analysis, whether it’s an actuary or not. The question is a template. We’ll use diabetes as an example, since that is a popular point solution.

(3) “What was the rate of primary-coded diabetes admissions per 1000 before you instituted the program and what is the rate now?”

Care to bet they will not know?  (Although two vendors validated by VI, Virta and Twin Health don’t even make that claim. Rather, they focus on deprescribing, which is more easily measured.)

Somehow, though, they are sure there were savings by reducing admissions. They just don’t know how or why, because they can’t know how much the diabetes admissions rate declined without knowing the starting point or the ending point.

They will probably say they can’t get the data. A good follow-up question might be to ask how, if they don’t have the data and don’t know what the starting point or the endpoint is, they are so sure they got savings.

Here is a trick question, meaning one that any actuary should be able to answer, but most can’t: “What is the approximate diabetes admission rate in the commercially insured population, per 1000 covered people?”  The answer is about 1. (You read that

right.)

So if, for example, they got a reduction of 10% in the admission rate, they would reduce admissions by 0.1 for every 1000 covered people. Figure $30,000 per admission. That creates a reduction in cost of $3 per covered person per year.

That example was diabetes. If you do musculoskeletal, there will be more opportunity to reduce utilization, beyond hospital admissions. Therefore, as opposed to just asking what the reduction in admissions is, you might even ask the vendor which procedure and test codes they typically tally to determine the reduction in utilization of high-cost procedures and tests. Then you can measure the change in those procedure codes, for the entire population.

This is also the way the Validation Institute’s RFP PowerTool is designed. If you are reading this in conjunction with the Critical Outcomes Report Analysis course, it is included. Otherwise, you can just review the redacted version here.