Regional Uses and Comparisons of National and Massachusetts Survey Data on ENERGY STAR Awareness

Transcription

1 Regional Uses and Comparisons of National and Massachusetts Survey Data on ENERGY STAR Awareness Timothy Pettit, Nexus Market Research, Arlington, VA Lynn Hoefgen, Nexus Market Research, Cambridge, MA Stephen Bonnano, NSTAR Electric and Gas, Westwood, MA Lomont White, Northeast Utilities, Hartford, CT Kim Ihrig, National Grid, Northborough, MA Deborah Jarvis, Unitil, Hampton, NH ABSTRACT For each of the past several years, a partnership of the Consortium for Energy Efficiency (CEE) and the U.S. Environmental Protection Agency (EPA) has produced a study on national awareness of ENERGY STAR. CEE members fund the survey, design the survey instrument, and oversee the data collection, which often includes oversamples at the state or regional level, in addition to the nationallevel sample. EPA then analyzes the national-level data. One of these oversamples was conducted in September and October of 2002 for the Massachusetts investor-owned electric utilities. Analysis for the Massachusetts utilities was conducted by Nexus Market Research as part of a Market Progress and Evaluation Report (MPER) on the Massachusetts ENERGY STAR Appliances Program. From the perspective of the program, the CEE WebTV survey is extremely useful for addressing the question of attribution: whether positive changes in the market have been brought about at least in part by the Massachusetts utilities program. The pattern across several measures suggests that higher recognition and understanding of the ENERGY STAR label in Massachusetts than nationally is largely attributable to the program. For assessing other stated objectives of the MPER, the Massachusetts CEE WebTV survey is insufficient. By clarifying the value of the CEE WebTV survey with respect to Massachusetts analytical objectives, this paper demonstrates the value of regional comparison through the CEE WebTV survey effort, and helps facilitate a common interpretation of the meaning and direction of the ENERGY STAR label through an analysis at the regional level. Background and Overview For the past several years, a partnership by the Consortium for Energy Efficiency (CEE) and the U.S. Environmental Protection Agency (EPA) has produced an annual report on national awareness of ENERGY STAR. CEE members fund the survey, design the survey instrument, and oversee the data collection, which often includes oversamples of data at the state or regional level that is, parallel, geographically specific surveys in addition to the national-level sample. CEE makes these data publicly available. EPA has typically undertaken an analysis of key indicators of ENERGY STAR awareness using the national-level data. CEE members who collect oversamples conduct their own analyses of the oversample data and compare them to the national-level data. The national-tostate or -regional comparisons make it possible for members to compare the effects of their own local efforts to promote the ENERGY STAR label as separate from efforts at the national level. One of these parallel regional surveys was conducted in September and October of 2002 for the Massachusetts investor-owned electric utilities (Massachusetts Electric Company, Nantucket Electric Company, NSTAR Electric, Western Massachusetts Electric Company, and Fitchburg Gas and Electric Light Company).

2 This survey, exactly paralleling the national CEE survey, involved WebTV data collection by Knowledge Networks, Inc. Analysis for the Massachusetts utilities was conducted by Nexus Market Research as part of a Market Progress and Evaluation Report (MPER) on the Massachusetts ENERGY STAR Appliances Program; the EPA commissioned a separate analysis of the national data. This paper compares the Massachusetts and national results, discusses the analytical objectives of the national and regional surveys, and examines the use and value of the national survey for regional comparisons. Description of the Massachusetts Program and Evaluation Components The long-term goal of the Massachusetts ENERGY STAR Appliance Program is to help establish a market for high-efficiency appliances in the Northeast. On the supply side, the program is designed to help develop the manufacturer and retailer infrastructure necessary to make these appliances widely available. On the demand side, the program is designed to increase consumer awareness and demand for ENERGY STAR-labeled appliances. The logic of the program can be summarized as follows: Awareness increases demand increases supply increases prices drop standards supported The program covers ENERGY STAR-labeled refrigerators, dishwashers, clothes washers, and room air conditioners. Key elements include: Marketing to consumers through advertising, educational programs, and selective rebates. Support of retailers through co-operative advertising, promotional point-of-purchase (POP) materials, sales training, and incentives for sales associates (spiffs). Support of manufacturers through co-operative advertising. Working with federal agencies and manufacturers to support increasing energy efficiency standards. This study integrates data and findings from a variety of evaluation activities besides the WebTV survey, including: Secondary research on other ENERGY STAR appliance programs and consumer behavior related to appliances. A random-digit dial (RDD) telephone survey of recent Massachusetts purchasers of and shoppers for appliances, used to estimate market size and share, and to assess the role of ENERGY STAR in purchasing and shopping decisions. A telephone survey of appliance retailers and purchasing executives in Massachusetts to assess marketing and stocking of ENERGY STAR-labeled appliances.

3 Mystery shopping visits in Massachusetts appliance retail stores to assess ENERGY STAR appliance inventories, promotional materials, and salespersons knowledge, attitudes, and selling practices with respect to ENERGY STAR. An assessment of the Massachusetts utilities ENERGY STAR appliance marketing plan in comparison with other regional programs, and in relation to overall program goals and theory. A brief assessment of non-energy benefits (NEBs) identified through the current evaluation and secondary data sources. Besides questions about awareness and understanding of the ENERGY STAR label, the CEE WebTV survey asked respondents a number of questions similar to some questions asked in the RDD survey related to appliance purchases and decision-making, in order to assess the role of ENERGY STAR in purchasing and shopping decisions. Description of the National CEE WebTV Survey and Analytical Objectives In fall of 2001, CEE conducted its first national WebTV survey, building on the national mail survey it fielded in 2000, and on other regional pilot WebTV surveys conducted in the Pacific Northwest and Wisconsin. The national WebTV survey was repeated in fall of The 2002 sample for Massachusetts consisted of 289 households selected randomly from all households that had been pre-recruited to Knowledge Networks Massachusetts WebTV sample. The national sample included the Massachusetts sample, plus another 879 households (1168 total). The non-massachusetts national sample was divided into three strata based on the largest Nielsen Designated Market Areas (DMAs) that account for approximately 70% of all U.S. households. These three strata are: High-publicity areas. Areas in which there have been at least two recent years of sustained promotions and publicity from non-federal activities. These areas include DMAs in states with a high degree of utility activity such as California, Connecticut, New Jersey, New York, Oregon, Rhode Island, Vermont, Washington, and Wisconsin. DMAs in Massachusetts, of course, are also high publicity areas, but they were treated as a separate stratum for purposes of this survey. Low-publicity areas. Areas in which there have been federal campaign activities only and no significant regional program sponsor activities. Other. All other DMAs (Pettit et al. 2002, 3).

4 The 2002 national analysis report, sponsored by EPA, is based on the 2001 analysis and is largely focused on label management concerns, evaluating several ENERGY STAR indicators on a national basis, and by publicity level for regional comparison. The 2002 analysis is organized as follows: Awareness (label recognition) Respondents were asked whether they had seen or heard of the ENERGY STAR label without seeing it (unaided recognition). Those who said they had seen it were then shown the label and were asked to confirm that this was in fact the label they were thinking of; those who had not seen it were also shown the label and asked if they recognized it (both of these together are aided recognition). Respondents were also asked to recall the products on which they had seen the label. The study compares estimates of awareness and product associations in high- and low-publicity categories. Understanding The study assessed messaging and label meaning by asking respondents what the label meant to them. Messages were grouped according to levels of understanding and were analyzed by high- and low-publicity categories. The report also presented label understanding by awareness. Decision-making (influence) Respondents reported actual purchases of ENERGY STAR-labeled products, the extent to which the ENERGY STAR label influenced their decision-making process, the degree to which financing influenced purchasing behavior, and their likelihood to purchase again and recommend to a friend (brand loyalty). The study compared self-reported purchases and the degree of label influence by high- and lowpublicity levels. Information sources The survey asked respondents to state on what media or other (nonlabeled product) sources they recall seeing or hearing about the ENERGY STAR label. The survey also asked respondents where they look for information on products that carry the ENERGY STAR. The study compares the sources seen versus the sources consulted (Pettit et al. 2002). For the federal ENERGY STAR program, which is co-sponsored by EPA and DOE, the data collection efforts of CEE and its members provide timely information on program impacts. In the EPA-sponsored analysis, the macro-level indicators allow the federal program to assess progress and to focus strategy. The analysis also helps ENERGY STAR partners understand EPA s perspective on the program s progress by defining key indicators of program impacts and focusing the analysis accordingly.

5 Summary of Massachusetts CEE WebTV Data The entire Massachusetts data set was analyzed and compared with the national data. To correctly compare the Massachusetts data to the national data, the national data were weighted by strata according to weights provided by the EPA s analysis contractor 1. Data were analyzed at the 90 percent level of confidence. The primary findings of CEE WebTV survey for Massachusetts are as follows: Thirty-one percent of Massachusetts residents indicate unaided awareness of the ENERGY STAR label higher than the 22% in the nation as a whole, and about the same as in other high publicity areas (32%). Aided recognition is at 56% in Massachusetts, again significantly higher than in the U.S. as a whole (39%), and about the same as in other high publicity areas (52%). This shows that the efforts to promote ENERGY STAR by the Massachusetts utilities (and by the utilities and market transformation organizations in other high publicity areas) have been successful in increasing recognition of the label. Massachusetts respondents (56%) are significantly more likely than respondents in the U.S. as a whole (49%) and those in other high publicity areas (48%) to show a high understanding of the ENERGY STAR label. The difference comes mainly from a higher proportion of Massachusetts respondents (12%, vs. 2% nationally and 2% in other high publicity areas) who mention energy or environmental product standards an indication that people in Massachusetts are likelier than others to use the label for consumer guidance. This is a key aim of the program, as consumers are told to look for the ENERGY STAR label. Asked where they have seen the ENERGY STAR label, Massachusetts respondents most often say on appliances or electronic equipment (34%), displays in stores (31%), TV commercials (21%), utility mailings or bill inserts (17%), or the Yellow EnergyGuide label (12%). Massachusetts respondents are likelier than those in the U.S. as a whole to have seen the label on all five of these sources, and are likelier than those in other high publicity areas to have seen the label on four out of the five (all except TV commercials). Thus Massachusetts appears to be doing better than other areas where consumers are closer to a purchase decision that is, when they see the labels on products or in stores. People in Massachusetts are likelier than those in the U.S. as a whole to recall seeing the label on all appliance types (43% vs. 23%), lighting products (18% vs. 7%), and all building materials (19% vs. 11%). Appliances and lighting are where the Massachusetts utilities have been concentrating their resources, so the effort appears to have paid off in terms of association of the ENERGY STAR label with these products. 1 The analysis is similar to the method used in the 2002 national analysis, using slightly different assumptions on distributions and Don t Know responses. Any differences, however, are small and usually within the sampling error of both reports estimates.

6 Discussion Similarly, people in Massachusetts are likelier than those in the U.S. as a whole to have seen the label on several products they have bought, including their refrigerators (40% vs. 19%), washing machines (39% vs. 20%), microwave ovens (mistakenly, probably because they associate the label with appliances in general 43% vs. 12%), and lighting products (40% vs. 14%). Massachusetts respondents (40%) are likelier than respondents in other high publicity areas (22%) to recall seeing the label on their new lighting products. Massachusetts respondents are less likely than those in the U.S. as a whole to recall seeing the label on their new copying machines, fax machines, and all home office equipment in general. Compared to those in other high publicity areas, Massachusetts respondents are less likely to recall seeing the label on their new central air conditioners, fax machines, and office equipment in general. The higher recall of the ENERGY STAR label on home office equipment in areas other than Massachusetts may simply be a function of respondent fatigue: they may only indicate the first few products that come to mind which suggests that the Massachusetts utilities are doing a good job of making sure appliances and lighting are at the top of their minds. Massachusetts consumers (65%) are significantly likelier than those in the U.S. as a whole (45%) to say that the presence of the label very much or somewhat influenced their purchase decision. There is no significant difference between Massachusetts and other high publicity areas (55%) on responses to this question. Massachusetts consumers (39%) are much more likely to report receiving rebates for their ENERGY STAR-labeled products than are consumers in the U.S. as a whole (13%) or those in other high publicity areas (16%). Nearly all Massachusetts buyers of ENERGY STAR-labeled products, like those in other areas, say they would have purchased the products without rebates. However, given the differences between Massachusetts and other areas in the results of this survey, it would appear that the success of the Massachusetts program may be driven to a large extent by rebates. Loyalty to ENERGY STAR, expressed as likelihood to recommend it to a friend, is not significantly higher or lower among Massachusetts consumers who have bought an ENERGY STAR-labeled product (59%) than among people in the U.S. as a whole (48%) or in other high publicity areas (68%) who have bought such products. Among all respondents, however because the base of ENERGY STAR buyers is proportionally larger those in Massachusetts are likelier than those in the U.S. as a whole (but not other high publicity areas) to say they would recommend it to a friend. From the perspective of the Massachusetts ENERGY STAR Appliances Program, the CEE WebTV survey is extremely useful for addressing one major component of the MPER namely, the question of attribution: whether positive changes in the market have been brought about at least in part by the Massachusetts utilities program. One alternative hypothesis is that the national, umbrella ENERGY STAR Program, administered by EPA and DOE, is responsible for virtually all of the observed market effects. The Massachusetts MPER, without having the benefit of a comprehensive baseline study, was not intended to address the actual magnitude of the incremental effects of its ENERGY STAR Appliances Program in the absence of other programs. Such quantification would be very expensive,

7 highly complicated, imprecise, and would result only in a qualified statement about the effectiveness of the Massachusetts utilities efforts that has limited operational value. Moreover, the national ENERGY STAR program is intended to have synergies through its partnering approach, and efforts to decouple partners programs from the national program do not measure intended program effects. While the Massachusetts ENERGY STAR Appliance Program does leverage off the national effort, the results of the evaluation in Massachusetts, of which the WebTV survey was a part, show a pattern that would be hard to explain without the conclusion that the Massachusetts program is responsible for a large share of the observed market changes. A good way to differentiate the effects of the Massachusetts program from those of the national program is to compare Massachusetts with other parts of the country, especially those with no active promotion of ENERGY STAR by utilities or other regional organizations. Levels of key indicators in these other areas should be largely attributable to the national program, and anything above these levels in Massachusetts should be largely attributable to the Massachusetts program. As noted earlier, a sample of Massachusetts residents is significantly more likely to remember seeing or hearing of the ENERGY STAR label without a visual representation (unaided recognition) than a national sample of households. Unaided recognition of the label among Massachusetts residents is similar to that found in other areas with high publicity for ENERGY STAR. Aided recognition is also significantly higher than found in the nation as a whole and similar to the level found in other high-publicity areas. In addition to recognition, Massachusetts residents exhibit a significantly higher level of understanding of the ENERGY STAR label than respondents in the nation as a whole and those in other high publicity areas. Similarly, Massachusetts residents are more likely to have seen the ENERGY STAR label on at least one appliance than are respondents in the nation as a whole and those in other high publicity areas. Thus, the pattern across several measures suggests that higher recognition and understanding of the ENERGY STAR label in Massachusetts than nationally is largely attributable to the Massachusetts ENERGY STAR Program. These outcomes are consistent with those that would be expected as a result of the program activities specifically, that increased consumer awareness would be fostered by advertising, educational programs, selective rebates, and support of retailers advertising and promotional efforts. The WebTV survey is also useful for the Massachusetts utilities (and for EPA and DOE) in that it can produce estimates of both aided and unaided recognition of the ENERGY STAR label, unlike either a mail survey or a telephone survey. Because the WebTV format like a mail survey can show the label, it is useful for measuring aided recognition. However, unlike with a mail survey, a WebTV survey can also be used to measure unaided recognition because the sequence in which respondents see questions can be controlled. Hence unaided recognition estimates from a WebTV survey should be comparable with those from a telephone survey, and aided recognition estimates should be comparable with those from a mail survey. For the stated objectives of the MPER, in order to estimate market size and share, and to assess the role of ENERGY STAR in purchasing and shopping decisions, the Massachusetts CEE WebTV survey is insufficient for two reasons. First, it cannot provide a sufficient number of records from its panel, given Knowledge Networks recruitment rate, to provide information at desired precision levels and detail for assessing market size, market share, shopping behavior, and purchasing verification of ENERGY STAR-qualifying appliances; future CEE WebTV survey efforts in Massachusetts will also be limited by the recruitment rate 2. The sample of 289 in the 2002 Massachusetts WebTV was the maximum that could be achieved; in comparison, the Massachusetts telephone RDD survey reached willing respondents in 2669 households, of which 791 qualified for a full interview because they had 2 For September 2003, the investor-owned utilities of Massachusetts have already committed to sampling annually with the national survey effort; however, only 135 cases will be collected due to Knowledge Networks relatively slow recruitment rates, limiting precision of Massachusetts compared to the rest of the country to 7.6% (at the 90% confidence level).

8 recently bought or shopped for an appliance. Second, questions on consumer decision-making require considerable probing on energy efficiency attitudes, shopping behavior, and verification of actual purchased models. The CEE WebTV survey asks which products consumers have seen the label on, which products they have bought, and which of the products they bought had the label. It also asks about the influence of the label and rebates on purchases and likelihood to recommend to a friend, but only for all ENERGY STAR-labeled products as a group rather than specific products. In contrast, the Massachusetts RDD survey asks similar questions about each of the four appliance types, in addition to questions about who selected the appliance (e.g., respondent, landlord, builder), where the appliance was purchased, reasons for purchase (e.g., old one stopped working, wanted more efficient one, remodeled, etc.), where they looked for information, whether or not retailers/catalogs/web sites/magazines discussed ENERGY STAR or energy efficiency and what they said about it, purchase intentions and awareness before the actual purchase, and more. In addition, the RDD survey asked refrigerator and dishwasher purchasers to provide us their model numbers so we could look them up and verify whether or not they had actually purchased an ENERGY STAR model; this served as the basis for estimating market share. The WebTV survey simply does not have the sample size or the detailed questioning necessary to provide such information. Conclusions On balance, the CEE WebTV survey is an excellent tool for collecting data on top-level indicators of ENERGY STAR awareness, understanding, consumer decision-making and purchasing behavior. The national CEE WebTV survey provides information on the impacts, over time, of the national ENERGY STAR partnership program which seeks to increase market penetration of ENERGY STAR-labeled products through synergies produced by its partnership approach. EPA s findings largely concern ENERGY STAR brand management issues and reflect policy objectives of the national partnership program, such as reducing air pollution related to energy consumption, including carbon emissions. Many of the variables and analytical methods from the national analysis, however, can be useful at the regional level, even where theory-based evaluations require more rigorous determinations of specific program effects. For the Massachusetts MPER, the greatest value of the WebTV survey data is the capability it provides for national and regional comparisons and thereby assessing the effects of the Massachusetts program; the WebTV survey is the only source of data we know of that permits this type of comparative analysis at such a reasonable cost. Program support for ENERGY STAR in Massachusetts includes a wide array of program elements, starting with public education and outreach to raise public awareness. Without the national WebTV survey data, a Massachusetts-only survey effort would provide a substantially incomplete analysis of awareness and understanding. Moreover, the cost to Massachusetts utilities of fielding their own concurrent national survey would take resources from other aspects of the MPER, leaving a substantially weaker overall evaluation effort. The remainder and majority of the CEE WebTV data are considerably less valuable to the MPER. For example, the MPER collected data in detail on four appliances to assess the market for those appliances, including respondent shopping behavior, and not simply for ENERGY STAR-labeled appliances. The CEE WebTV survey collects purchasing data on all ENERGY STAR-labeled products, and the survey would be prohibitively onerous on respondents to collect data in that level of detail for all ENERGY STAR-labeled products. The complete CEE WebTV data set, however, is not completely disregarded in the MPER. Some WebTV data were analyzed in conjunction with data from other consumer survey efforts. The entire Massachusetts WebTV survey data set was analyzed, compared to the national data, and

9 included as an appendix in the MPER. This appendix was useful in providing information on relationships between broad indicators of consumer decision-making and shopping behavior, and as supporting evidence for conclusions reached through specific indicators established by other, more directed survey efforts. Although the CEE WebTV survey meets a valuable data collection need for Massachusetts investor-owned utilities, considerations for other regions are as follows: Utility ENERGY STAR programs lacking significant outreach and public education components may find it difficult to measure the effects of their programs with the WebTV approach. The differences in awareness and understanding in Massachusetts compared to elsewhere in the country reflect substantial investment on the part of the Massachusetts utilities. Some states with small populations are not adequately represented in Knowledge Networks WebTV sample, so that oversampling is not an option no matter how extensive the outreach and public education effort. In addition to the availability of sample, small regions outside the top 70% of household DMAs have not been included in the sampling strategy since the first national WebTV survey. Comparisons of estimates from small regions to the nation, therefore, are of limited value since national estimates are inherently biased toward large media markets. An individual small region that wishes to participate does not have a good reference for comparison except as part of a larger program, such as a state program. Consistency of analytical methods between regions fielding their own surveys concurrent with the national WebTV survey is necessary to maximize the usefulness of the survey effort into the future. The analytical methods for weighting strata and regional groups are documented in EPA s report. States or regions fielding oversamples should consider supplemental consumer surveys to collect data on market penetration and consumer behavior for specific product programs and market assessments, or, for market penetration, other methods such as retailer surveys. In summary, the incorporation of the 2002 CEE WebTV effort into the Massachusetts MPER was generally a positive experience for all parties involved, including CEE, EPA, and the investorowned utilities of Massachusetts. The Massachusetts MPER included analyses from the WebTV data that was central to its program tracking needs, and CEE and EPA bolstered data collection and analysis efforts by increasing the Massachusetts sample size. By clarifying the value of the CEE WebTV survey with respect to Massachusetts regional analytical objectives, this paper demonstrates the value of regional comparison through the CEE WebTV survey effort, and helps facilitate a common interpretation of the meaning and direction of the ENERGY STAR label through an analysis at the regional level.

10 References Pettit, T., V. Goepfrich, M. Goldberg, L. Hoefgen National Analysis of the 2001 ENERGY STAR Household Surveys: Final Report. Washington, D.C.: U.S. Environmental Protection Agency.