Proving Energy Savings

I develop deemed savings for TRMs. I know they are a great representation of average energy savings and are well designed by stakeholder groups of engineers, consultants, program administrators, and independent evaluators. Furthermore, they are vetted and approved by regulatory bodies such as public utility commissions, so you should have a good measure of confidence in the savings.

However, most homeowners implement multiple efficiency measures and it’s important to validate the savings in the field. My team takes on the challenge of testing the difference between the deemed savings and the meter-verified savings.

For a recent project we designed and applied a paired t-test: a statistical test that compares the mean values of meter-verified vs deemed on the same population of homes.

• HYPOTHESIS: We proposed that the difference between the mean meter-verified result and the mean deemed result was zero.
• STATISTICAL SIGNIFICANCE: The p-value is the statistical significance of the findings. If the p-value was below 0.05, we rejected our hypothesis (and concluded that there is in fact a difference) as this indicates that there is less than a 5% probability that the difference is zero.

We found that sometimes meter-verified savings were higher than deemed, and sometimes they were about the same.

So, what’s the big difference in HVAC? For HVAC equipment that gets replaced at the end of its useful life, the TRM assumes the old system would have been performing equivalent to a minimally code-efficient system, and uses NREL system performance curves to estimate consumption. However, those systems were performing well below code, and we can see that in the AMI data when we stratify HVAC by replacement type: end of life or early retirement.

We also dug into the attic insulation measure and found some interesting patterns when we stratified by baseline R-value. Savings in the categories of lowest baseline R-value (R-0, effectively no insulation) were significantly lower at the meter than the TRM estimated. But the population of homes in those lower baseline categories was relatively small. Most homes had existing baseline attic insulation in the R-9–14 range, and those homes outperformed the TRM estimate.

Ultimately, our findings resulted in changes to some of the TRM assumptions, which helped the deemed savings do a better job of estimating the true average savings in a portfolio of efficiency measures.

Is it time for you to check on how your portfolio is performing? Email me to get started. Want a job like mine? Visit our careers page!

How we did it: A Meter-Verified Study of Deemed Savings

Here is how we tested if advanced metering infrastructure (AMI) data measures up against deemed savings from the TRM:

1. We obtained project tracking data for 21,772 homes that installed attic insulation, air sealing, duct sealing, air conditioners, and heat pumps between May 2017 and June 2019.

2. Then we classified the homes based on the measure(s) installed (it is common to see more than one measure installed at a home), as well as by key parameters such as heating and cooling type, attic insulation R-value, and air infiltration reduction level. All in all, we had nine different measure combinations and 14 key parameter categories.

3. Applying the deemed savings from the TRM, we calculated ex ante energy and demand impacts for each home.

4. Next, we obtained hourly interval AMI data for those same homes from May 2016 through June 2020 because our analysis required at least one year of usage data for the periods before and after the measures were completed.

5. It was too much data for a spreadsheet — it crashed our computers. We wrote multiple scripts in Python to read the data.

6. To identify and remove AMI data that wasn’t suitable for analysis, we executed a rigorous data cleaning procedure. We were left with 11,969 homes after the data cleaning.

7. For each home that remained after data cleaning, we established customized best-fit multivariate regression models for pre- and post-installation periods, incorporating statistically significant predictor variables for heating- and cooling-degree hours, time of day, and a COVID variable to account for consumption changes resulting from stay-at-home lockdowns. We automated a linear regression for each °F combination of heating- and cooling-degree hours that output the model coefficients associated with the reference temperatures that produced the lowest prediction error.

8. Then we removed any homes whose models showed high prediction errors. That left us with 10,017 homes, which is just 46% of the original 21,772.

9. To produce weather-normalized hourly annual load profiles, for both pre- and post-installation periods, we applied the coefficients generated in the regression analysis to a typical annual weather profile (we used TMY3 data).

10. Oh yes, we also referenced a control group of non-participants to explore exogenous effects; those social, cultural, and market factors unrelated to DSM program activities. We found that the average weather-normalized consumption for the control group was slightly higher in the post-period than the pre, indicating that our analysis in the treatment groups (i.e., those 10,017 homes) is likely somewhat conservative in general. Meaning, a full accounting for exogenous effects is likely to increase savings.

Originally written by Angel Moreno, Frontier Energy — https://frontierenergy.com/blog/#proving-energy-savings