New thinking on benchmarking customer experience Vicki Howe, Head of Product Development April 2015
Contents 1 Key points... 3 2 Introduction... 4 2a Satisfaction benchmarking from HouseMark... 4 3 Star benchmarking and its benefits... 5 3a Latest sector-wide results from HouseMark s Star benchmarking... 5 3b Key drivers of satisfaction... 6 4 Star, tracking and transactional surveys and the need to review... 7 4a The consultation... 8 4b What the sector said... 8 5 StarT: Our way forward for transactional surveys... 10 5a Star developments in response to the consultation... 11 Star and StarT, and the intellectual property rights in them, are owned by HouseMark Ltd ( HouseMark ). We give express permission for anyone who is a social landlord, or their subcontractor, to use, store and transmit Star and StarT. This includes the right to copy the Star and StarT questions and the approach set out in written guidance provided by HouseMark. Star and StarT surveys and questionnaires do not have to be acknowledged to HouseMark. The Star logo (available at www.housemark.co.uk/star) may be reproduced on questionnaires, reports and other materials based on Star on the basis that you use it in the same way that HouseMark uses it, including without limitation, its format, colours and proportions. You are expressly prohibited from using the Star logo in a way that is detrimental to the reputation of HouseMark or Star or in a way that is not in keeping with the way that HouseMark uses the logo. You acknowledge that Star and StarT has not been developed to meet your individual requirements. Whilst care has been taken to ensure the accuracy and completeness of the questions, we exclude all conditions, warranties, representations or other terms, so far as is permitted by law, concerning Star and StarT, including without limitation that relating to fitness for purpose, accuracy, completeness or intellectual property. We hereby disclaim all liability and responsibility for your use of Star and StarT, or any part thereof, so far as it is permitted by law. 2
1 Key points HouseMark owns and manages the Star satisfaction framework for the social housing sector. Using this framework, over 350 UK social housing landlords regularly collect and benchmark perception satisfaction survey data. Landlords use satisfaction data, of which Star is a part of, for their own performance monitoring and to inform business decisions. Taken as a whole, the HouseMark Star satisfaction dataset reveals overall sector trends and typical drivers of satisfaction. Our latest data shows overall satisfaction levels have remained constant on average for general needs and housing for older people tenants. Since Star s launch in 2011, a wider range of survey methods have become popular with landlords than is covered by the Star framework. Transactional surveys, such as repairs, lettings or ASB surveys, are not covered directly by the Star framework. However, some HouseMark members wanted to explore the appetite for guidance and a benchmarking framework specifically for transactional surveys. HouseMark, in association with market research company Acuity, conducted a major consultation exercise with HouseMark members in late 2014 / early 2015 to find out more about the extent to which transactional surveys are utilised, current expectations for customer satisfaction benchmarks and the continuing role of Star. We found that Star remains popular and continues to have an important role to play, with 91% telling us that Star still has a place. The evidence base arising from this consultation exercise has led to the development of StarT by HouseMark a transactional satisfaction survey framework that is complementary to, but separate from, Star. HouseMark is delivering new and refreshed guidance and benchmarking services to help our member compare satisfaction for both perception (Star) and transactional (StarT) surveys. 3
2 Introduction 2a Satisfaction benchmarking from HouseMark In 2011, HouseMark launched the Star (Survey of Tenants and Residents) framework - a periodic perception-based survey that provides social housing landlords with the means to compare satisfaction results with each other. Today, nearly four years later, around 350 landlords regularly collect and benchmark this information using HouseMark services. Landlords say they continue to find Star an invaluable resource for understanding how tenants and residents feel about the services they provide and they use this information to inform business planning and resource allocation. As a wider range of survey methods has become available for measuring and understanding satisfaction, there has been a growing appetite to combine this over-time perception survey data with real-time customer feedback. Many landlords are now carrying out transactional surveys triggered by specific events or interactions with customers; for example following a responsive repair, complaint, or letting. The purpose of this report is three-fold. Firstly, it provides a recap on the benefits of Star, and what HouseMark s Star dataset tells us about the latest trends in satisfaction. Secondly, it outlines the results of our research into: The extent to which transactional surveys are being used, and are evolving, in the sector The demand for benchmarking data gathered from transactional surveys What guidance the sector might need on transactional surveys and understanding their relationship to perception customer satisfaction surveys (such as Star surveys). And, thirdly, to introduce StarT; our new framework for benchmarking transactional survey data. Used together, the Star and StarT frameworks will provide a fuller picture of comparable customer satisfaction than has ever before been available. 4
3 Star benchmarking and its benefits HouseMark developed Star for the social housing sector. Our website 1 contains details about how to run a Star survey together with a full set of questions covering the core areas of activity; as well as several optional and model questions for specific landlord services. This standardised approach to satisfaction measurement enables landlords to make meaningful performance comparisons with other landlords working in the social housing sector and monitor trends over time. Satisfaction measurement provides a valuable source of data for measuring and reporting tenant and resident-focused outcomes and engaging tenants and residents in what their landlord does. Most landlords will carry out a variety of surveys to capture satisfaction data. Star is not designed to meet all the surveying needs of a typical landlord; however, it will deliver a robust, central foundation for capturing tenants and residents general perceptions about their landlord. The framework is designed with sufficient built-in flexibility to ensure that landlords are able to measure what they need for Star compliance in ways that best suit their organisational requirements and those of their tenants and residents. Satisfaction measurement is also about gathering robust, actionable data to inform business decisions on changes to service delivery. Without subsequent action, satisfaction measurement has no purpose; without robustness any action may be misdirected. For landlords in Scotland, HouseMark teamed up with SHBVN (Scottish Housing Best Value Network) to produced Star in Scotland which combines the regulatory requirements under the Scottish Social Housing Charter with Star's established framework for comparing satisfaction. Once a Star (or Star in Scotland) survey has been conducted and collated, the results can be entered into HouseMark s Star module. Star data, when combined with cost and performance data in HouseMark s core benchmarking service, provides the basis upon which to judge value for money and identify the best opportunities for improving performance. 3a Latest sector-wide results from HouseMark s Star benchmarking HouseMark s annual report on Star benchmarking data based on surveys carried out in 2013/14 found that overall satisfaction has remained constant for leaseholders, general needs and housing for older people tenants between 2012/13 and 2013/14. There has been a small increase (1% point) in overall satisfaction for supported housing tenants and a larger increase (8% points) for shared owners. 2 1 https://www.housemark.co.uk/subscriber-tools/benchmarking/survey-of-tenants-and-residents 2 Comparisons used a balanced panel of landlords that submitted performance data for both years, and excluded those which submitted data from the same survey two years running. 5
As in previous years, this year s annual report found that tenants are more satisfied with the service provided by their landlord than home owners (leaseholders and shared owners). In 2013/14, general needs tenants were less satisfied than those living in housing for older people, and shared owners are more satisfied overall than leaseholders. When the results are broken down by region, they show that satisfaction for London-based organisations is around 10 percentage points lower than other regions across all quartiles. Outside London, the scores are remarkably consistent, with only a few points separating the regions at each quartile threshold. Although less up-to-date than the data captured by HouseMark s Star benchmarking 3, the CLG s English Housing Survey publications 4 also offer some supplementary insight into recent trends in satisfaction. The 2012/13 survey findings show that the general trend in satisfaction with accommodation, neighbourhood and repairs among social housing tenants over the five years between, 2008/09 and 2012/13, is very slightly upwards. 3b Key drivers of satisfaction HouseMark s annual Star benchmarking report uses correlation and regression analysis to assess how much each benchmarked Star question influences overall satisfaction. This demonstrates that every core measure 5 is significantly correlated with overall satisfaction, and that this correlation is positive i.e. as one core measure increases in value, so does overall satisfaction. Repairs and maintenance has the biggest influence on overall satisfaction, as it did in in 2012/13 and 2011/12. Landlords whose residents are satisfied in this area tend to have the highest rates of overall satisfaction too. The next biggest influencer is listening and acting to views, closely followed by the Value for Money (VFM) of rent and the quality of home. This implies that getting resident involvement right is an important factor in achieving good overall satisfaction 3 The English Housing Survey s breakdowns by tenure (including social housing tenants) for its 2013/14 survey are not published until July 2015. 4 https://www.gov.uk/government/collections/english-housing-survey 5 The seven core Star satisfaction measures cover overall satisfaction, the home, neighbourhood, VFM rent, VFM service charge, repairs and listens/acts on views 6
scores, as well as VFM and the quality of home, as lower ratings in these areas will have a negative impact on overall satisfaction. 4 Star, tracking and transactional surveys and the need to review Since the launch of Star, a much wider range of survey methods has become popular with landlords for measuring and understanding customer satisfaction. Notwithstanding this broader range of survey options, 91% of our consultation respondents are clear that they want to continue deploying Star as part of their mix of approaches. A Star survey is a perception survey aimed at measuring all 6 customers views, impressions and opinions about their landlord and the services it delivers. Historically, perception surveys were carried out at specific points in time, typically every two or three years. Today, however, many landlords carry out their Star perception surveys as tracking surveys which means running the same survey at set intervals throughout the year, such as weekly, monthly or quarterly. We define transactional surveys as those that collect customer feedback about an interaction or event, gathering data linked to a recent service while the interaction is still fresh in customers minds. The distribution of a transactional survey to a respondent (e.g. by way of posting a survey or making a phone call) is triggered by an event such as a responsive repair, a complaint or a letting. These surveys are an invaluable way of measuring the customer experience and views on service quality at the operational level, obtaining feedback very quickly after the event. The distinction between the two survey types of perception and transactional is not the actual questions asked, but the trigger for distributing the survey. For one type, the survey distribution follows an interaction or an event (transactional) whereas the other type (perception) does not. HouseMark recognises the need to be even clearer in our benchmarking guidance about what is and is not a Star survey, and specifically how transactional surveys relate (or not) to STAR. The purpose of our review was therefore, to understand the extent to which transactional surveys are used in the sector and to assess the demand for the benchmarking of customer satisfaction data from transactional surveys. The review also sought to understand what guidance the sector might require on transactional surveys and their relationship to perception customer satisfaction surveys (such as Star surveys), and to develop a new framework for the use of both surveys to successfully capture robust and representative data. 6 To capture all views, the landlord will either conduct a census survey or a carefully constructed sample survey to ensure statistically valid representation 7
4a The consultation The consultation set out to understand the extent to which transactional surveys are used in the sector and to assess the demand for the benchmarking of customer satisfaction data from transactional surveys. It also sought to understand what guidance the sector might require on transactional surveys and their relationship to perception customer satisfaction surveys. We also took the opportunity to seek feedback on the current Star survey framework. HouseMark began the project in October 2014 with a roundtable discussion to set the scope of the review. Participants were invited from a cross-section of HouseMark members, some of whom were critics of Star. Following the event, two online member surveys and ten in-depth telephone interviews were carried out in December 2014 and January 2015. HouseMark members were also asked to submit examples of their Star and transactional surveys. The online survey was aimed at research and policy staff, directors and managers of specific service areas. It was split into two parts: Part A focused on benchmarking and the role of transactional and Star surveys; Part B asked further in-depth questions about the particular transactional surveys conducted by the landlord. Part A of the survey was completed by 178 staff from 166 landlords, with a further 73 landlords continuing to complete the more in-depth Part B. A charitable donation was made for each survey response received. In addition to the online consultation responses, a total of 341 examples of transactional and Star questionnaires were sent to HouseMark. In-depth structured telephone interviews were conducted with senior executives in the social housing sector. These interviews captured their opinions on the use made of customer satisfaction information in improving, developing and tailoring services to meet customer requirements, monitoring operational performance and contract management and establishing a sector-wide approach to benchmarking transactional surveys. Detailed findings from the review are available for HouseMark members on our website. 4b What the sector said The review confirmed the popularity of Star surveys with 91% of respondents confirming that they use the framework and plan to keep on doing so. Virtually all landlords carry out transactional surveys (99%). Almost all of the landlords conduct responsive repairs surveys (94%), with a high percentage carrying out ASB complaints (84%), new lettings (76%), and complaints (77%) surveys. Telephone (85%) and postal surveys (83%) are the most popular research method used by social landlords to survey residents; but many landlords also use online (56%), face-to-face (51%), mystery shoppers (48%) and focus groups (47%). Two of the challenges facing many landlords are falling response rates for postal surveys and the often disappointing levels of take up for online surveys. Although many landlords use social media to communicate with customers, few are 8
conducting customer research over these newer platforms. Of the landlords who conduct Star surveys, the majority continue to carry out oneoff surveys (66%), but a third now use tracking surveys (33%). Postal surveys remain the most popular survey method for Star surveys (68% compared to 25% telephone), but a third of those carrying out postal surveys are considering changing survey methods. This desire to investigate alternatives could be related to the concern about falling response rates for postal surveys. Those adopting the second most popular method of telephone surveys are much more likely to retain this method in future only 4% would consider a change. We asked about the extent to which landlords include the seven core questions in their Star surveys. Six of the seven core Star questions are included by well over 90% of landlords (94% to 100% depending on the question); only the question on whether service charges provide value for money, which is not applicable for all landlords, is less frequently used (81%). In our discussions with HouseMark members prior to the consultation, we had picked up a limited amount of concern about two specific core Star questions and we sought opinions on their future in this consultation. We found the majority of landlords favour keeping the question on whether rent provides value for money (66% find it useful). Just over half of the landlords who responded wanted to keep the question about landlords listening and acting on residents views (53%). In order to compare satisfaction results, the requirement for all participants in a benchmarking exercise to work to the same answer scale for a particular question is as important as the wording of that question being the same. For all core Star questions, a five-point descriptive text scale of very satisfied, fairly satisfied, neither, fairly dissatisfied, very dissatisfied was adopted when the framework was set up in 2011. We asked about response scales again in our latest consultation. A majority of respondents (53%) favoured retaining a five-point scale (which includes the middle ground option neither satisfied nor dissatisfied ). The use of descriptive text for the answer scale (rather than a numeric scale) remains the most popular option (78%). Two-thirds of landlords would like to see a set of survey questions developed for transactional surveys by HouseMark (65%), with a higher number wanting to see guidance and advice from HouseMark (as thought leaders ) on survey methods and sample sizes for transactional surveys (87%). Landlords want to see more customer satisfaction measures added or included in HouseMark s benchmarking modules and a link to comparable data sets available outside the sector. The consultation found a strong demand for guidance, sharing of good practice and learning events for the sector. 9
5 StarT: Our way forward for transactional surveys In response to the feedback from the consultation, we are developing StarT (Survey for Tenants and Residents Transactions), our new benchmarking service for transactional surveys. Just as Star provides a comparable framework for a landlord s main tenant and resident perception satisfaction survey, StarT will provide a framework for comparable transactional surveys. The Star and StarT frameworks are distinct and separate, but together they provide a fuller picture of landlords customer satisfaction than Star was able to deliver alone. The feedback from our consultation confirms our belief that when it comes to comparing satisfaction results, Star (perception) data and StarT (transactional) data are also distinct and separate. We will not be mixing data from the two different types of survey in our benchmarking services. Our new StarT guidance on the use of transactional surveys is available to HouseMark members. It includes a set of questions for transactional surveys as well as providing advice on survey methods, sample sizes and other areas highlighted in the review. Through the introduction of StarT, we are expanding our benchmarking services to include clearly defined transactional customer satisfaction measures, starting with the most popular surveys identified in the review repairs, lettings, ASB and complaints. During 2015/16, landlords who wish to take up this offer of new transactional satisfaction benchmarking services for repairs, lettings, ASB and complaints will be able to adapt their survey methods and questions to meet our new StarT framework. From April 2016, the new online benchmarking systems will be available to HouseMark subscribers to input their survey data and see how they compare. We have developed standard StarT questions for each of these four transactional survey areas. There is built-in flexibility and landlords do not have to ask all of the questions in each service area in order to participate in our benchmarking. Question sets may be expanded from other sources such as optional StarT questions and your own questions to tailor the survey to meet your exact needs. The choice of StarT questions was strongly influenced by the information obtained during our consultation from the on-line questionnaire respondents, in-depth interviews and hundreds of examples of customer survey questionnaires/telephone scripts. 10
StarT will enable more landlords to take part in comparable survey measurement appropriate to the survey type, and a wider range of measures to be included in HouseMark satisfaction benchmarking. 5a Star developments in response to the consultation Our priority has been to respond to the consultation feedback in relation to providing StarT guidance and benchmarking solutions for transactional satisfaction surveys for those HouseMark members who want this. We also learned from the consultation that, while there is very widespread (91%) support for the Star framework, there are a number of areas where landlords would like further guidance and there were various suggestions for development of the Star framework. During the spring and summer of 2015, we will be considering and consulting further on: The future form of the listens and act core question in Star The introduction of social value and/or ONS well-being measures into the Star optional question set How to address some concerns about whether the neither responses are included or excluded in the calculation of satisfaction results Expanding the range of filters available in on-line benchmarking of both Star and StarT for those landlords who want to narrow their peer group of comparison to surveys and organisations that are most like their own. The overall timeline for HouseMark delivery on both a refreshed Star and a new StarT framework to our members is: This new thinking report on benchmarking customer experience 30 April 2015 Detailed findings from our consultation 30 April 2015 StarT a guide to running transactional surveys, including StarT questions Upgrade of the key Star documents: Star features A guide to running Star Star questions Introduction of on-line StarT benchmarking services Upgrade of on-line Star benchmarking services, including more filter options 30 April 2015 July 2015 April 2016 2018 11
4 Riley Court, Millburn Hill Road, University of Warwick Science Park, CV4 7HP. Company Reg. No 3822761 VAT Reg. No. 744 0353 53 T: 024 7646 0500 E: info@housemark.co.uk W: www.housemark.co.uk