Vivint Smart Home Energy Savings Study Heating and Cooling Results September 2017 For Vivint Element Thermostat enabled with Sky Smart Home Assistant DISCLAIMER: This study was commissioned by Vivint Smart Home and was independently conducted by WattzOn. WatzOn retains complete editorial control. The results reported here are averages across a large sample of homes. Actual savings results by home will vary, and depend on a number of factors including climate, utility rates, type of heating and cooling equipment, seasonal weather patterns and more. SUMMARY WattzOn conducted a study on nearly 7,000 homes across the U.S. to evaluate energy savings from the Vivint Element Thermostat enabled with the Sky Smart Home Assistant. The study period, June 2016 through May 2017, includes both cooling and heating seasons. The results show that homes using Vivint s thermostat technology experienced as much as a 12% reduction in heating and cooling energy use throughout the year, resulting in utility bill savings of up to $146 per year. The study measured energy savings in the same manner as ENERGY STAR, the U.S. government program that provides a product label for consumer assurance of energy savings. The study homes maintained comfortable temperatures, averaging 74 F indoors with minimal fluctuations, and achieved energy savings that exceed the ENERGY STAR standard. KEY RESULTS WattzOn recently completed a detailed study of energy savings from connected thermostats for over 7,000 homes across the U.S. that had installed the Vivint Element Thermostat enabled with the Sky Smart Home Assistant. The study evaluated energy savings using important data for each home, including minutes of air conditioner or heating run time per day, indoor temperature achieved, indoor temperature fluctuations, and outdoor temperature. All data were available at 5-minute intervals, allowing for precise measurements. The key results are: Vivint s national average savings results exceed the ENERGY STAR standard. 1 1 The ENERGY STAR savings standards are 8% for heating and 10% for cooling. The national averages are based on weighted average of results across U.S. climate zones. (Please note that ENERGY STAR excludes the Marine climate
Vivint integrates intelligent thermostat controls with security system sensors, enabling more precise control of HVAC runtimes, and increased savings. The study found an average 16% improvement in energy savings when the Sky Smart Home Assistant is enabled, as compared to homes that used other thermostat control systems. The improvement in savings is statistically valid at the 95% significance level. Preliminary results suggest the Vivint thermostat technology uses periods when the resident is out of the home to save additional energy. The results are preliminary because not all sub-sample results are statistically significant. Additional data and study is needed. Vivint has designed its intelligent thermostat controls for home comfort. Data analyses for a small sample of homes highlights the the narrow indoor temperature range residents experience, even when outdoor temperatures fluctuate significantly. METHOD AND ENERGY SAVING FINDINGS Data was available from over 7,000 homes that have operated the Element thermostat since June 2016. Summer cooling season savings and winter heating savings were calculated for each home in the study using the same method and software that is used by the ENERGY STAR connected thermostat certification process. Use of the ENERGY STAR method and software allows for a direct comparison of this study s results to the standard. 2 Savings are estimated by calculating the actual heating and cooling (HVAC) runtime minutes, and the expected HVAC runtime minutes. Residents have a variety of indoor temperature preferences, and an estimate of expected runtime minutes is calculated based on this preference. This is the baseline. Savings are achieved when actual runtime minutes are less than expected. The method used incorporates a resident s temperature preferences into the baseline, and thus does not overestimate savings. For example, if a family likes a cold home in the winter, this is not counted as high energy savings. The reduction in runtime by each home during the cooling or heating season is then averaged across the homes by climate region. A U.S. weighted-average savings result is calculated based on the distribution of the homes by climate region. Results are calculated separately for heating and cooling seasons, per the ENERGY STAR protocol. The results are summed by home to provide an annual perspective on thermostat performance. The cooling season in this study is June 22 October 1, 2016. The heating season in this study is November 1, 2016 May 1, 2017. zone from the standard.) The standard can be found at: https://www.energystar.gov/sites/default/files/energy%20star%20program%20requirements%20for%20connected%20thermostats%20ver sion%201.0_0.pdf 2 This study is not an official ENERGY STAR certification. A multi-year multi-stakeholder process was used at the ENERGY STAR program to develop the certification standard. This study makes use of the final draft standard, issued in December 2016. https://www.energystar.gov/products/spec/connected_thermostats_specification_v1_0_pd 2
Within the study sample, there were two important splits in the data. First, is whether the consumer chose to opt-in or opt-out of Vivint s Sky Smart Home Assistant, the intelligent controls. The homes in this study were nearly evenly split amongst homes that opted-in and homes that opted-out. Homes that switched groups during the sample period were dropped from the study. Homes that opted-in to the smart home controls experienced a 16% increase in energy savings. Second, the homes were grouped by the average number of weeks during the heating or cooling season in which the residents were at home at least 30 hours during the week. Residents in the AWAY ALOT group were out of the house more than 30 hours per week in more than 75% of weeks of the heating or cooling the season. The AT HOME group was the opposite, in that they had fewer than 25% of weeks in the season in which they were out of the house more than 30 hours during the week. The away behavior was analyzed separately for heating and cooling. In this sample, there were three times more homes in the AWAY ALOT group than the AT HOME group. While the regional sub-samples and away results are interesting, it appears that there are not a sufficient number of homes in many sub-samples to produce statistically significant results. It is suggested that the study be repeated with more homes included to obtain a more accurate picture. Initial results for the regional sub-samples with high significance levels show that residents in the AWAY A LOT group do experience higher energy savings. While preliminary, this finding illustrates the ability of the Vivint home automation sensors to dynamically manage heating and cooling to fit residents behaviors. 3
Vivint Smart Home Thermostat Study September 2017 UTILITY BILL SAVINGS To estimate of the benefit of the Vivint intelligent controls on the resident s utility bill, the avoided runtime minutes were converted into reduction of the heating or cooling load (eg kwh or therms of energy use.) Local electricity and natural gas rates were applied to estimate the monetary savings. The average savings results by region are shown below. FIGURE 1 ANNUAL AVERAGE SAVINGS OF $112 PER YEAR USING VIVINT ELEMENT THERMOSTAT SAVINGS WITH SKY SMART HOME ASSISTANT ENABLED *The ENERGY STAR certification does include data from homes in the Marine climate zone. Source: US Energy Information Administration climate zones: https://www.eia.gov/consumption/commercial/maps.php Because of variations in heating and cooling loads, rates, climate and so on, a range of potential utility bill impact was calculated. Assuming a 30% variation in typical home energy load, utility bill savings range from $79 to $146 per home per year. See Table 3 in the ADDITIONAL DETAIL section below for more information. COMFORT AND SAVINGS For nearly 20 years, energy experts have struggled to balance the competing goals of energy savings and comfort. Typically the experts recommend that users program their thermostats, 4
pushing temperatures to cooler indoor temperatures in the winter and higher indoor temperatures in the summer. But, asking Americans to program their thermostat is the first snag. Recent data from the Energy Information Administration shows that only 12% of Americans program their thermostat during the summer cooling season. 3 In addition, the typical temperature setting recommendations reinforce the old notion that that saving energy means an uncomfortable home. For example, the U.S. Department of Energy advises summer indoor temperature settings of 78 F when at home, and 7 to 10 F higher when away. An indoor temperature of 68 F is recommended in the winter. 4 These sharp swings in indoor temperatures remain the standard recommendation, and many connected thermostats attempt achieve savings by implementing these rigid schedules in various forms. But, the leading professional HVAC engineering society, ASHRAE has found that 80% of individuals are most comfortable indoors with temperatures of 74 F to 79 F in the summer, and 68 F to 73 F in the winter. 5 So it is no surprise that families often fight against the schedule, constantly tweaking the indoor temperature, only to have it revert back to the uncomfortable pre-set temperature in 30 minutes or an hour. In contrast, indoor temperature data from Vivint homes shows a very different pattern. The Vivint intelligent controls maintain a steady indoor temperature with minimal fluctuations, while saving energy. There is little evidence in the Vivint temperature data of the constant tweaking behavior. A sample of data from approximately 50 homes, was reviewed and all homes showed very little night/day fluctuations, indicating consumer comfort with the opt-in settings. To illustrate these phenomena, a week of indoor and outdoor temperatures is shown for a typical home in the study below. 3 https://www.eia.gov/todayinenergy/detail.php?id=32112 4 https://energy.gov/energysaver/thermostats 5 https://www.3rhsec.com/files/ashrae-55-2010.pdf 5
FIGURE 2 TYPICAL INDOOR TEMPERATURE PATTERN VIVINT ELEMENT WITH SKY HOME ASSISTANT JUNE 15-21, 2016 As the graph illustrates, the home experienced a very narrow range of indoor temperature fluctuations, even as the outdoor temperatures cycled up and down by 25 to 30 degrees Fahrenheit per day. As following table shows, the Vivint intelligent controls used the Away Mode to reduce runtime and save energy. It is not surprising that the indoor temperature drifted up a bit during the Away Period. TABLE 1 DETAIL FOR SAMPLE HOME June 15-21, 2016 Outdoor Indoor Away Period Maximum 96.8 78.5 81.0 Temperature ( F) Minimum 66.2 74.0 76.0 Temperature ( F) Average 82 76 79 Temperature ( F) Max Min ( F) 31 4.5 5.0 AC Runtime* 56% 27% * AC Runtime for Indoor = Minutes AC Running / Total Minutes in Sleep or Home mode. AC Runtime for Away Period = Minutes AC Running / Total Minutes in Away Mode 6
It is reasonable to ask whether this sample home is unusual: Perhaps the home is well insulated and thus retains cooled air nicely, making it easy to maintain a constant indoor temperature. Visual inspection for a sample of homes across climate zones and seasons found that this indoor temperature pattern is actually typical. Homes using Vivint Element with the Sky Smart Home Assistant experienced very minimal indoor temperature fluctuations. Statistical validation of this conclusion should be undertaken in a further study. ADDITIONAL TABLES The Range of Utility Bill Savings The following table shows the range of utility bill savings found in this study. The range reflects a 30% variation in heating and cooling energy load for typical homes by region. The savings results used (runtime reduction) combine the opt-in and opt-out groups and are statistically significant at the 95% significance level. TABLE 2 THE RANGE OF UTILITY BILL SAVINGS ALL VIVINT STUDY HOMES Region Average High Low Cold $154.66 $201.06 $108.26 Hot-Humid $93.48 $121.53 $65.44 Mixed Dry $114.93 $149.42 $80.45 Mixed Humid $87.44 $113.68 $61.21 Weighted Average* $112.63 $146.42 $78.84 * Weighted by number of homes in each region. 7
Thermostat Savings Results by Climate Zone Climate Zone TABLE 3 THERMOSTAT SAVINGS BY CLIMATE ZONE (Runtime reductions) SUMMER, 2016 Energy Savings with Vivint Intelligent Control Energy Savings without Vivint Intelligent Control Mixed-Humid** 10% 9% Hot-Humid 9% 8% Notes: ** Difference in thermostat savings between opt-in and opt-out significant at the 95% significance level. Marine climate zone not shown, as ENERGY STAR excludes it from its methodology. TABLE 4 THERMOSTAT SAVINGS BY CLIMATE ZONE (Runtime reductions) WINTER, 2016/2017 Climate Zone Energy Savings with Vivint Intelligent Control* Energy Savings without Vivint Intelligent Control* Hot-Dry/Mixed- 10% 10% Dry** Cold/Very Cold 12% 9% Hot-Dry/Mixed- 15% 14% Dry** Cold/Very Cold 26% 24% Mixed-Humid** 16% 14% Hot-Humid 14% 10% Notes: ** Difference in thermostat savings between opt-in and opt-out significant at the 95% significance level. Marine climate zone not shown, as ENERGY STAR excludes it from its methodology. SAMPLE SIZE AND STATISTICAL SIGNIFICANCE As WattzOn reviewed the study results, we used a careful chain of reasoning to test the results for statistical validity. First an F-test was used to determine the form of a t-test statistic. Then the value of the t-test statistic was calculated and as was a p-value, which indicates whether the null hypothesis of a difference in means should be accepted. 8
As the above tables show, not all regions and sub-samples had statistically significant results. The sample size required that might possibly achieve statistical significant results, given the thermostat variance observed, was explored in various ways. We estimate that more than 800 homes are needed within each sub-sample group to obtain statistically significant results. This finding is highly dependent on the size of the mean savings results by group, the standard deviation of savings results and the size of the difference in means. In addition, WattzOn reviewed other studies on thermostat savings and discussed the statistical significance issue with other experts in the field. We found that almost all studies of thermostat savings ignore the matter of statistical validity. This is unfortunate, as it prevents study results from informing product development. If one wants to create a feature to save energy, then one really does not know if it adds savings without statistically significant results. Product features can be confidently added only when there is sufficient data for potential statistical significance and careful review of clear hypotheses. Otherwise, produce refinements could actually be reducing energy savings. STUDY METHODOLOGY The Data. The study was based on data from 9416 homes across the U.S. that installed the Vivint thermostat on or before June 15, 2016. Data on indoor temperatures, weather, and air conditioning run time from June 22, 2016 through May 1, 2017 were used. Data from the home was collected at 5-minute intervals. Homes were removed from the sample if they switched between the opt-in to the opt-out groups. How Savings Were Estimated. ENERGY STAR is U.S. government standards and product certification program that enables energy-efficient product choices in the market. For the past three years, ENERGY STAR staff and industry leaders have been developing a method to certify energy savings from Connected Thermostats; thermostats that use sophisticated systems and a connection to the internet to provide heating and cooling control. 6 This study, conducted in the summer 2017, uses the ENERGY STAR savings methodology, including the calculation software ENERGY STAR has made available. 7 8 Note that the results of this study do not qualify the Vivint thermostats for an ENERGY STAR certification. However, the results provided here are entirely comparable to the ENERGY STAR standards. 6 Details on the ENERGY STAR Connected Thermostat certification are here: https://www.energystar.gov/products/spec/connected_thermostats_specification_v1_0_pd. 7 The software, methodology and extensive documentation are available here: https://www.energystar.gov/sites/default/files/energy%20star%20program%20requirements%20for%20conne cted%20thermostats%20version%201.0%20final%20draft.pdf. 8 In December 2016 ENERGY STAR announced their final standards before going live with the certification program in 2017. Once the standards go live, thermostat providers will register their data sets with authorized third-party labs that will conduct a study using the ENERGY STAR savings calculation software. 9
For the summer season, the ENERGY STAR method estimates energy savings from the reduced air conditioner (AC) use enabled by a connected thermostat as follows: First, an expected AC runtime minutes per home is estimated, conditional on the indoor temperature preferences of the residents and local weather. The actual runtime minutes are compared to the expected, and savings are achieved when the actual runtime minutes are less than expected. The calculation of savings is done separately for each home, with the savings per day summed over the days in the cooling season. The results for each home are averaged over the climate zone region. The national savings result is the weighted-average of regional results, using a weighting scheme that reflects the national pattern of cooling energy use. 9 The same scheme is used to calculate winter season energy savings, based on heating run time. Vivint data on heating equipment is insufficient to distinguish between systems (furnace, heat pump, baseboard heating, boilers) and fuel type for heating (natural gas or electric). The most frequent system and fuel type by climate zone was used in this study to calculate expected bills and bill savings. Note that the lack of home-specific heating system data may lead the study results to overstate heating savings. The ENERGY STAR connected thermostat testing protocol excludes-speed furnaces, and these are included in this study. Variable speeds can be more efficient than a single speed. Also the protocol omits an adjustment factor for heat pumps. It will be important to have data heating house type to improve this study. The ENERGY STAR method is advancement over previous methods to estimate savings because it directly accounts for a user s temperature preferences and habits. Previously, it was assumed that all residents preferred an indoor temperature 72 F year-round, 24/7. This created a false baseline, as it is clear that users have varied preferences. The ENERGY STAR method is the first to use observed indoor temperatures to capture realized preferences, and to use a calculation method that accounts this variety in preferences. Studies show that very few people bother to set the detailed thermostat schedules that energy experts recommend. For example, one study found that only 10% of homes programmed setbacks. 10 As mentioned, a recent EIA study showed that only 12% of homes programmed their AC. Previous studies assumed that a high rate of consumer compliance would take place after appropriate education. In contrast, the ENERGY STAR method reflects how users actually engage with thermostats, which is to maintain indoor comfort. The ENERGY STAR method fits well with the Vivint thermostat strategy of a simple user experience and intelligent controls to deliver comfort in an energy-efficient manner. How Monetary Savings Were Estimated. To convert runtime minutes saved into a monetary amount, the following additional data were used to translate runtime minutes saved into utility bills saved. 11 9 See the source cited in footnote 3 for details. 10 https://eec.ucdavis.edu/files/usability_of_residential_thermostats.pdf 11 Sources: https://www.eia.gov/consumption/residential/ 10
A state with a large population center was selected for each climate zone. Typical heating and cooling loads were calculated per home in that state The most frequent heating system and heating fuel type were identified for that state Local electric and natural gas rates were obtained Typical cooling and heating bills were calculated Savings rates from Vivint thermostats were applied separately for the cooling and heating seasons. The results are monetary savings per season. The seasonal savings were added to an annual total. How the Range in Monetary Savings Was Estimated. The biggest point of variation between homes in a locale is their level of use of energy, and thus the typical heating and cooling loads were varied within each region to create an estimated range of monetary savings. Based on other WattzOn data, we used the typical energy load +/- 30%. The average thermostat savings from the pooled sample by region was then applied to get a range of kwh or therms saved. All other inputs and assumptions above apply. ABOUT WATTZON WattzOn provides utility data and analytics to business partners in the connected home, solar and consumer credit markets. WattzOn s unique national platform for residential utility data covers 94 million U.S. homes, and our software tools make it easy for consumers to provide utility data. Our products such as Snapshot, which allows a resident to take a photo of their utility bill. High-powered, machine-learning based analytics provided by WattzOn allow for customer segmentation and targeting based on energy use patterns. WattzOn has worked with leading smart thermostat manufacturers and has been involved in several connected thermostat field trials, including calculation of combined utility bill savings from residential solar systems and smart thermostats. The WattzOn team is highly credentialed, and is led by Dr. Martha Amram (Ph.D MIT.) WattzOn has been granted two patents on its use of analytics for utility bill disaggregation, and has one patent application pending for its use of machine-learning type techniques to process utility bill images. Based in Mountain View, CA, WattzOn has received numerous awards for its analytics and utility data access products, including a grant from the U.S. Department of Energy for Consumer Insights from Smart Meters, invited presentations at the White House, and being named a Top 20 Energy Software Company by CIO Review. To learn more, please go to www.wattzon.com. https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_6_a https://www.eia.gov/dnav/ng/ng_pri_sum_a_epg0_prs_dmcf_m.htm https://www.eia.gov/todayinenergy/detail.php?id=30672#&src= Consumption Residential Energy Consumption Survey (RECS)-b3 https://www.energystar.gov/ia/business/bulk_purchasing/bpsavings.../calc_cac.xls 11