Cutoff Prediction Results for 2017 Star Ratings
New Star Rating cutoffs for 2017 are out. As expected, the new cutoffs deviated from last year’s cutoffs, and will substantially affect the calculation of the overall Star Rating. Once again, many MA plans will be surprised.
We, Accordion Health, developed a sophisticated prediction model for these notorious ever-changing cutoffs. Our data scientist team analyzed hundreds of public datasets, found temporal patterns of Star Rating measures, and simulated CMS’s methodology of deriving cutoffs. Our clients have been using our predicted cutoffs from the beginning of this year. Given the new cutoffs have been publicly announced, we think it is a good time to see how we scored!
The Medicare Advantage Star Rating system consists of 44 different measures. Therefore, the prediction results vary by measures. Some measures are much more difficult to predict than others due primarily to the lack of data. For example, Star measures related to Special Needs Plan (SNP) are extremely difficult to predict their cutoffs, since only a subset of MA plans are eligible for the measures.
Cutoffs for Medication Adherence Measures
Presenting the results for all the measures can be quite laborious. Thus, we will focus on the prediction results of the Medication Adherence measures — diabetes, hypertension, and cholesterol medications. We picked these three measures, not just because they are highly weighted measures but also improving these measures positively impact many other clinical outcome measures.
Let’s see the results!
The chart above shows the average errors (measured in Root Mean Squared Error) of two different cutoff predictions: one from a baseline model that uses last year’s cutoffs as predictions, and our model that uses hundreds of public datasets and machine learning algorithms. As can be seen, the average errors from our model are significantly less than the ones from the baseline model.
300% More Accurate Cutoff Predictions
The average RMSEs from the two models are as follows:
- Baseline: 3.041
- Accordion: 1.039
Our prediction model, on average, was off by ±1 from the true cutoffs, while the baseline was off by ±3. In short, our model was approximately 3 times more accurate than the baseline.
Let’s take a look at the cutoff predictions for the Cholesterol Medication Adherence measure (D14, MA):
The cutoffs for this measure have been widely fluctuating in the past, and this year was not an exception. See the 2 star cutoff was 50 last year, but 66 this year. 16 points difference! If a plan was shooting for 3 stars relying on last year’s cutoffs, it is possible that they may not even be able to achieve 2 stars. On the other hand, check out our cutoff predictions. Our predicted values are very close to the truth.
Our Predictions are Getting Better Every Year
The beauty of our model is that it is “data-driven”. More data, better predictions. As we have more data points in 2017, we believe our predictions for 2018 and 2019 will be even more accurate than ever before.
Better strategy starts with better prediction. If you want to know more about your 2018 or 2019 Star Rating predictions, please contact at info [at] accordionhealth [dot] com.
