A review by the performance measurement practice July Performance Frameworks and Board Reporting

A review by the performance measurement practice Juy 2009 Performance Frameworks and Board Reporting

Our vision is to hep the nation spend wisey. We promote the highest standards in financia management and reporting, the proper conduct of pubic business and beneficia change in the provision of pubic services. The Nationa Audit Office scrutinises pubic spending on behaf of Pariament. The Comptroer and Auditor Genera, Amyas Morse, is an Officer of the House of Commons. He is the head of the Nationa Audit Office which empoys some 900 staff. He and the Nationa Audit Office are totay independent of Government. He certifies the accounts of a Government departments and a wide range of other pubic sector bodies; and he has statutory authority to report to Pariament on the economy, efficiency and effectiveness with which departments and other bodies have used their resources. Our work eads to savings and other efficiency gains worth many miions of pounds: at east 9 for every 1 spent running the Office.

Performance Frameworks and Board Reporting a review by the performance measurement practice Juy 2009

The responsibiity for taking forward strategic aims is invested in Departmenta Boards. The effectiveness of Board decision making is therefore centra to achieving Government objectives and vaue for money.

Contents Part One Summary 4 Part Two Deveoping a framework 6 Part Three Reporting performance information 16 Part Four Using the framework 24 Part Five Concusion 30 Appendix One Methodoogy 31 Appendix Two Maturity Mode for Performance Measurement Frameworks 32 This review was conducted by Diane French, Peter Gratzke, Vincent McCarthy under the direction of Nick Soan. For further information about the Nationa Audit Office pease contact: Nationa Audit Office Press Office 157-197 Buckingham Paace Road Victoria London SW1W 9SP Te: 020 7798 7400 Emai: enquiries@nao.gsi.gov.uk Nationa Audit Office 2009

4 Part One Performance Frameworks and Board Reporting Part One Summary 1.1 Government Departments are responsibe for deivering the ambitions of Government, incuding tacking the ong-term chaenges facing the UK. Whether a Department is responsibe for reducing chid poverty, tacking cimate change, or improving heathcare it must make decisions about how to prioritise its interventions and secure the greatest possibe efficiency for every pound of taxpayers money it spends (CO, 2009). 1.2 The responsibiity for taking forward strategic aims is invested in Departmenta Boards 1. The effectiveness of Board decision making is therefore centra to achieving Government objectives and vaue for money. Performance information presented to Boards must aow the Board to: focus on the things that matter, understand and agree on what the business needs to do to deiver organisationa objectives, expain business performance, and take we-informed decisions. 2 The best way to structure and present performance information has been the subject of much anaysis and debate in both pubic and private sectors. This paper examines current centra government practice by reference to a survey and case studies (see Appendix 1 for methods). Overview 1.3 In the ast decade performance measurement has been a key theme in Centra Government and a the organisations we surveyed have some form of performance measurement framework. In most cases a broad framework exists to structure performance information, and there is some evidence of good practice; with frameworks being used to bring about performance gains and support performance management. But there is cear scope for improvement, and a need to get more vaue from the considerabe, if unquantified, costs incurred in creating and operating the frameworks. We have structured our findings in three Parts: deveoping a framework; reporting performance information; and ensuring performance information is used. The main messages are: performance measurement frameworks are widey used, and focus on strategic objectives, but rarey reate to a coherent business mode or provide a comprehensive picture of performance. 1 HM Treasury (2005). Corporate governance in centra government departments: Code of good practice 2 KPMG (2008). Nationa Audit Office: Board Reporting & Management information Discussion Pack.

Performance Frameworks and Board Reporting Part One 5 It is important for Boards to focus on what matters, but agreeing priorities can often be contentious with differing perspectives across different poicy areas or Directorates. Many organisations we surveyed used a performance framework that catered for both resuts (incuding outputs and outcomes) and enabers (incuding inputs, panning and processes). But we saw itte evidence of frameworks being supported by cear business modes that persuasivey inked inputs through processes to outputs and outcomes. And we noted organisations either measuring outcomes over which they have itte infuence, or undertaking outputs and activities which had itte infuence on the outcomes sought. Better use of modes can maintain a ogic to the summarisation of information for a Board, and aid its interpretation. organisations need to devote more attention to data quaity, and improve its presentation, to support decision-making. Producing high quaity data is essentia to the utiity of performance information and was identified by 13 out of 36 survey respondents as the greatest chaenge, and critica success factor to deveoping an effective performance measurement framework. Routine vaidation of the data systems underpinning Government priorities reguary highights poor data quaity and our foow-up work provided severa exampes of frameworks operating with itte attention to the quaity of the measures used to popuate them. Presenting performance data in context generates performance information. For exampe, by presenting measures against benchmarks Boards can compare performance and judge vaue for money. Our review of Board reports showed that they often acked carity and woud benefit from a more anaytica expanatory commentary. performance measurement frameworks did not ink financia information and performance information satisfactoriy, hindering informed strategic decision-making. We found that few organisations had a good understanding of the inks between financia or cost information and performance information. And around a third of respondents did not use performance information to make strategic decisions about resources or process improvement. In our foow-up interviews, however, we identified organisations that, with the hep of a performance measurement framework, had created a performance improvement cuture by invoving and unocking the potentia of staff. 1.4 The case studies in this report give exampes of how some organisations have met these chaenges and impemented performance measurement frameworks to improve service deivery. In Appendix 2 we provide a matrix to enabe organisations to assess their current stage of maturity and pan improvements.

6 Part Two Performance Frameworks and Board Reporting Part Two Deveoping a framework 2.1 An effective performance measurement framework (see Figure 1) heps a Board to take informed strategic decisions about how best to pursue the organisation s objectives. 3 From our survey, we identified three factors in the deveopment of a performance measurement framework that contribute to its effectiveness: Formuating cear and agreed strategic objectives is essentia given the potentia for confusion afforded by changing priorities, numerous stakehoders, and compex deivery chains; Understanding the business and taioring the performance measurement framework to focus on the key deivery drivers for achieving strategic objectives; Getting a comprehensive but concise picture of organisationa performance to aow the Board to monitor enabers as we as resuts and so have some confidence in managing future performance. Formuating cear and agreed strategic objectives 2.2 Uncear or conficting objectives can distort the focus of performance measurement and ead an organisation to aocate resources inefficienty or engage in activities that do not provide Vaue for Money. 4 Furthermore, if objectives are not shared throughout an organisation or amongst partners, there is a risk of working towards different, incompatibe goas and faiing to achieve desired outcomes. 5 Figure 1 Performance Measurement Framework A performance measurement framework is a management too that inks a the performance information in the organisation. it enabes the seection and prioritisation of performance indicators. A good performance measurement framework wi te the whoe performance story of an organisation and ink inputs, outputs, and outcomes. it wi aow a board to take decisions that are based on the best evidence and provide vaue for Money. 3 HM Treasury et a., Choosing the right FABRIC: A Framework for Performance Information, 2001. 4 NAO, Good Government, 2008, p. 12. 5 NAO, Joining up to improve pubic services, p. 9. The report goes on to ist faiures due to ack of shared objectives, p. 20, p. 47.

Performance Frameworks and Board Reporting Part Two 7 2.3 Eeven per cent of survey respondents considered focusing on strategic objectives to be a major chaenge in deveoping effective performance measurement. The main difficuty identified was aigning stakehoders and senior management behind an agreed but concise set of objectives. Centra Government organisations aso have to integrate numerous cross-government initiatives, ike the efficiency or better reguation agendas. For Executive Agencies and Non-Departmenta Pubic Bodies (NDPBs) formuating strategic objectives means engaging with parent or ead Departments, Cross- Government poicies, other Centra Government deivery bodies, Loca Authorities, and stakehoders outside of Government, such as service users or citizen groups. 6 2.4 Figure 2 provides a seection of survey responses from Finance Directors of Centra Government organisations. The foowing case exampe shows how one organisation deat successfuy with some of these chaenges. Figure 2 Chaenges to focusing on strategic objectives (seection of responses to NAO survey of Centra Government Financia Directors) Handing the issues of changing priorities around key objectives. Diversity of the organisation. finding the baance between the various eements of the organisation and the need to disti the outputs to a manageabe eve. Getting aignment between the objectives set for us by Government, and the objectives that we set for our contractors. Part of the probem for us is defining the high eve poicy objective in strict [operationa] terms. Proper, informed negotiation of the PSA in the first pace. Sometimes we think what we have to deiver was not ceary thought through. Case exampe 2.5 A arge goba investment bank identified that senior managers and their business units often have differing understandings of what the overa business strategy is and how it shoud be impemented. Foowing a review of numerous strategy documents, interviews with senior managers identifying 45 different monthy reports and 2,300 metrics, the bank re-formuated its strategic objectives into three distinct goas against which its key performance metrics coud be mapped: Buiding cient intimacy Acceerating growth businesses Creating a factory to industriaise interna processes 6 Tabot, et a., Exporing Performance Regimes: A New Approach to Understanding Pubic Sector Performance, Report for the Nationa Audit Office, 2005.

8 Part Two Performance Frameworks and Board Reporting 2.6 The review process increased the use of supporting evidence in the business strategy. The objectives were then agreed by the board as the most important evers against which to measure the success of the business strategy. The carity and conciseness of the objectives articuated the intended goas in a way that aowed them to be easiy transated in tangibe and measurabe indicators. Understanding the business 2.7 Performance information must enabe a Board to exercise strategic contro over the organisation. A performance measurement framework must therefore cover the key processes in the organisation and refect a robust understanding of the business. To deveop an effective performance measurement framework an organisation must: identify the main drivers of objective achievement, and deveop a causa mode of how they interact with each other and with the environment; and understand who, within or outside, the organisation is responsibe for deivery by deveoping a deivery map and schedue. Causa modes 2.8 Causa modes articuate what an organisation must do to achieve its objectives, and show how the seected actions wi produce the intended effects. Government s topeve objectives are now specified around improving societa outcomes: in most cases these are chaenging objectives, with many pausibe actions which coud contribute, as we as choices for each action about the deivery route (pubic, private or vountary sectors, for exampe). If objectives are to be achieved cost-effectivey, a Department must fuy understand not ony the ikey cost-effectiveness of options for action, but aso how they interact with each other, and how they shoud be schedued. 7 2.9 Causa modes tend to be compicated, and make heavy demands on the research base on what works and on data systems. Spending Review guidance has recommended the preparation of causa modes since SR2000. The Cabinet Office found, however, in its recent Capabiity Reviews that Departments often do not understand or communicate their business modes we. 8 Whie the modes may be compicated, they can be simpified (as shown in Figure 3) to hep those in the deivery chain understand the significance of their activities to overa performance, and to faciitate performance management overa. 7 Nationa Audit Office, Good Government, 2008. 8 Cabinet Office, Capabiity Reviews: Progress and Next Steps, 2007.

Performance Frameworks and Board Reporting Part Two 9 Figure 3 Causa Mode outcome Key drivers strategies outputs Things we can contro Things we can infuence but not contro Things we cannot contro or infuence Horizon scanning 2.10 A causa mode is necessary to get most vaue from monitoring progress, as we as panning appropriate action. Many of the outcomes the Government seeks to achieve are not infuenced by Government action aone, but aso by many externa factors, for exampe; achieving better heath and webeing for a and word cass science and innovation in the UK, as expressed in two of the current Pubic Service Agreements. Many Centra Government organisations find it difficut to estabish their contribution to intended outcomes. 9 It is important to understand the contribution of the three sets of drivers iustrated above to outcomes being monitored both to choose appropriate responses to emerging circumstances and to get an accurate view of Departmenta performance. Case exampe 2.11 The former Department for Innovation, Universities and Skis (DIUS) (now the Department of Business Innovation and Skis) deivers a broad agenda. One of the Department s objectives is customer satisfaction. In a report commissioned by the NAO, KPMG assisted DIUS to expore a causa mode that shows how the Department s activities ead to the achievement of customer satisfaction 10 (see Figure 4 overeaf). 9 Nationa Audit Office, Acting on Information, 2005: Two thirds of Departments have difficuties estabishing measures which show the Department s contribution to fina outcomes. 10 KPMG, On Deivery, 2008, p. 4. Avaiabe at: http://www.nao.org.uk/guidance_and_good_practice/good_practice/ performance_measurement.aspx

10 Part Two Performance Frameworks and Board Reporting Figure 4 Causaity mode for DIUS Customer Satisfaction target outcome Key driver strategy output To improve customer satisfaction by three percentage points each year from 2009 to 2011 as measured by question six ( overa satisfaction ) on the annua customer satisfaction survey, using the 2008 survey as a baseine. being abe to get in touch with the right person in the organisation. Transactions between customers and the organisation being processed prompty. The organisation s staff having a positive, friendy attitude when deaing with customers. improving customer access. improving processing times. improving staff attitudes. Customers to be directed to the right person at their first point of contact. for a significant transaction streams, transactions to be processed within agreed timescaes. Staff to dea with customers in a friendy and positive manner. 2.12 DIUS popuated the causa mode by first ceary defining the scope of the objective in terms of outcomes. Then they identified key drivers that have a positive impact on customer satisfaction, and deveoped strategies to achieve them. The strategies specified outputs that contribute to the overa outcome of customer satisfaction. 2.13 For a causa mode to deiver vaue, the cause-and-effect reationships must be based on evidence and vaidated. Research shows that private sector organisations that deveop and vaidate a causa mode consistenty outperform other organisations and achieve on average a 5.1 per cent higher return on equity. Vaidation in this context means using actuas data to check on the correation between individua performance measures and progress in achieving strategic objectives. 11 Vaidating the causa mode has the benefits of increasing confidence in the mode, uncovering fase assumptions and faciitating accurate earning from experience. By vaidating the inks between the metrics in its performance measurement framework, the goba investment bank in Section 2.5 was abe to determine the importance of individua drivers in contributing to the intended outcome, as we as identifying eading indicators (see Figure 5) for the achievement of strategic objectives. 12 11 Ittner, Christopher D. and David F. Larcker, Coming Up Short on Nonfinancia Performance Measurement, Harvard Business Review, November 2003, p.91. 12 Accenture, Performance Measurement Frameworks In Board Reports: Insights Into Leading Practice From The Private And Internationa Pubic Sectors, 2009,p.37.

Performance Frameworks and Board Reporting Part Two 11 Figure 5 Leading indicators An indicator is eading if it is inked to, and precedes, the vaue of some subsequent resut. Leading indicators can therefore be used as predictive toos. for exampe, an increasing eve of staff turn-over coud be a eading indicator for decining customer satisfaction and increasing training costs. Deivery maps 2.14 Deivery maps demonstrate how an organisation wi deiver the required actions, by mapping the processes within an organisation as we as the responsibiities of deivery partners. In practice, many different bodies are often invoved in achieving the Government s objectives (see Figure 6). We found organisations that had significanty improved their performance by estabishing how to deiver and execute their strategy by mapping the deivery chain and using process improvement methodoogies. 13 An exampe is the Crimina Injuries Compensation Authority. Figure 6 Deivery partners of PSA 28 PSA deivery agreement 28 Secure a heathy natura environment for today and the future shows that a ead department, three NDPbs, 9 other Departments, Regiona Deveopment Agencies, Loca Authorities, devoved Government bodies and Third Sector organisations are invoved in the deivery of a singe PSA. 13 Nationa Audit Office, Deivering Efficienty Strengthening the inks in pubic service deivery chains, 2006.

12 Part Two Performance Frameworks and Board Reporting Case exampe 2.15 The Crimina Injuries Compensation Authority (CICA) was strugging with ong processing times before it used process improvement methods to map the deivery cyce of its core business. As a resut, the process was grouped into four stages and performance indicators were attached to each stage to monitor the cyce (see Figure 7). 2.16 CICA specified a rationae for each indicator emphasising its purpose and putting it into an organisationa context (Figure 8). Since focusing on the key indicators, CICA can mode and predict the case oad and has set ambitious targets for the future. Figure 7 CICA s process indicators process stages Stage A Appication to fina Registration Key performance indicator Time to Register an appication ineigibe appications and ni awards Stage B Registration to first Decision Size of ive caseoad Active caseoad cyce time to first decision Stage C Request to Review to Review Decision Active caseoad cyce time to compete review Cases outstanding over 2 years Stage D Appea Appea stage response times Decisions overturned Figure 8 The rationae for indicators The objective underying the first indicator, Time to register an appication for exampe, is: To record initia appications as quicky as possibe after they are received. (Target = 3 days) its rationae is, To provide victims of crime with as fast a service as possibe by ensuring the Authority does not distort its statistics by deaying the addition of new appications to its case oad.

Performance Frameworks and Board Reporting Part Two 13 Getting a comprehensive picture 2.17 Our survey showed that one in five respondents considered seecting the most reevant measures to be both a critica success factor and major chaenge. The main difficuty was striking the right baance between comprehensive and concise performance reports, whie accommodating the compexities of the organisation (see Figure 9). 2.18 55 per cent of Centra Government organisations in our survey adopted a Baanced Scorecard (BSC) approach to structure their performance information. The best BSCs are deveoped in conjunction with causa modes, 14 so seection of measures are reevant and inked to achieving strategic objectives. The origina BSC inked Financia Resuts to Customer Satisfaction, underpinned by the quaity of Interna Processes in turn supported by Learning and Growth. However, our anaysis of Board performance reports showed that many organisations appy BSCs ineffectivey. For exampe, we found BSCs composed entirey of programme miestones, with no indicators of resuts. In other cases, the main categories of the BSC had been used as buckets into which avaiabe indictors had been grouped, without cear inks to each other or to the business mode. BSCs constructed purey as a presentationa device wi not yied the insights into future performance that the baanced technoogy was designed to provide. Figure 9 Chaenges to seecting reevant measures (seection of responses to NAO survey of Centra Government Finance Directors) The board s desire for comprehensive performance information on the one hand and, on the other, a short, concise board report. Keeping it simpe yet meaningfu. There is a need for constant vigiance in ensuring the board is not overwhemed by too much information, yet has sufficient to enabe effective risk management and panning. in a compex organisation with a matrix management structure and mutipe externa stakehoders, designing a framework that wi meet the priority needs internay and externay is a chaenge. Managing compexity agreeing the appropriate baance of data to measure/assess to be abe to contro programme deivery and take appropriate interventions, whist not overburdening deivery teams with onerous data coection requirements. bringing together a aspects which infuence Departmenta performance, incuding resources, capabiity, and in our case, assessing the deivery of [the Department] s remit [ ]. 14 Kapan, R. S. and D. P. Norton, The Baanced Scorecard Measures That Drive Performance, Harvard Business Review, 1992.

14 Part Two Performance Frameworks and Board Reporting Case exampe 2.19 The former Department for Business, Enterprise and Reguatory Reform (BERR) (since merged with DIUS to form the Department for Business, Innovation and Skis) is responsibe for a wide range of poicy areas incuding consumer issues, better reguation, empoyment matters, and economic deveopment. This arge portfoio poses a chaenge in providing the Board with a comprehensive picture of the organisation s performance. To structure the information and make it accessibe to Board members, BERR deveoped a BSC that takes account of its intended strategic outcomes, the resources it uses, stakehoder views, and its capabiities (see Figure 10). 2.20 The scorecard integrates the Department s strategic objectives and has a strong narrative, which shows how BERR s objectives can be achieved by stakehoder engagement, efficient resource management, and organisationa capabiity. These four perspectives provide the Board with a comprehensive overview of the determinants of BERR s success. The headine indicators and traffic ight ratings make it simpe to read. Additionay, the scorecard is suppemented by more detaied reports on each indicator, so Board members can dri down to obtain information on the headine indicators. Board members can navigate from a broad overview of the organisation to a robust assessment of ower eve performance. Figure 10 BERR s Corporate Baanced Scorecard (Ratings are iustrative ony) Deivering our objectives Part 1: Board priorities Baanced Scorecard 2008-09 Managing our resources Top 16 Managing within CSR settement AR 1 PSA indicators Admin budgets/workforce panning AR 2 DSO indicators Deivering vaue for money savings AG 3 Being the voice for business in Government Improving our capabiity Communicate BERR's roe Gather/anayse business issues Work successfuy with OGDs Buiding our reputation with business AG AR AR AG 4 Driving improved performance Deivering effectivey with our partners Deveoping our peope AG AG AG

Performance Frameworks and Board Reporting Part Two 15 Seecting a concise set of measures 2.21 To prioritise and reduce the number of reported measures to ony the most important measures, organisations can fiter and map them according to their importance. Typicay, an organisation wi aready record a arge amount of information. To seect the most important indicators to report to the Board, the goba investment bank from Section 2.5 appied a fiter to eiminate redundant metrics. It assessed existing indicators to identify groups of indicators measuring simiar things, or those in a cear hierarchy, or not reevant to business concerns. By eiminating overapping measures, the bank reduced the number of metrics throughout the organisation from 2,300 to 73 and mapped a measures to its three strategic objectives. The 73 incuded a few new indicators where the process had identified gaps in measurement. The bank then used its vaidated causa mode to determine which indicators had the greatest impact on its objectives and grouped them according to their importance. With the prioritisation iustrated in Figure 11, the bank seected the most important measures in ight of its strategic outcomes. Figure 11 Prioritising Metrics financia sensitivity high ow watch factors Metrics in this category require monitoring to ensure financia performance targets are met Distractions Metrics in this category can be eiminated from top eve consideration vaue Drivers Metrics in this category directy reate to strategic imperatives tactica Metrics in this category wi be used to manage high priority but ow financia impact areas of the business ow manageabiity high

16 Part Three Performance Frameworks and Board Reporting Part Three Reporting performance information 3.1 The best seection of measures is of itte use if monitoring data are unreiabe or not reported effectivey. The foowing factors are important when reporting performance information: Constructing vaid measures is necessary to ensure that indicators refect what they are intended to and provide a soid basis for decision making; Producing high quaity data is a precondition for the utiity of performance information and generates confidence in the performance report; Reporting information in context, for exampe in reation to targets or benchmarks, aows the Board to evauate how we the organisation is performing and achieving Vaue for Money; Presenting information with anaysis and insight or commentary enabes Boards to easiy understand, and engage in, performance discussions and provides a good basis for making decisions. Constructing vaid measures 3.2 Our recent survey found that one in four respondents considered deveoping vaid measures a chaenge to impementing effective performance measurement frameworks. This resuts corroborates previous survey findings that showed that identifying high eve quantifiabe measures of intended outcomes to be a significant chaenge for Government Departments. Our recent anaysis of the data systems underpinning Departmenta Strategic Objectives showed that amost one fifth had weaknesses in matching the specification of the intended indicator. Case exampe 3.3 The British Counci (BC) seeks to achieve engagement and trust for the UK through the exchange of knowedge and ideas between peope wordwide. The aim is achieved through programmes in areas such as education, Engish anguage and cuture, and governance, ooking to infuence and faciitate change in four key ways: atering perceptions: changing the eve of understanding or opinion hed by our target audience on a particuar issue; agenda setting: changing the priority given to a particuar issue among opinionformers and decision-makers;

Performance Frameworks and Board Reporting Part Three 17 capacity buiding: increasing the capacity skis, personne, institutions and resources to engage in or react to a particuar issue; and institutiona change: changing poicy, organisationa or civic structures, ega or reguatory frameworks reating to a particuar issue. 3.4 BC has deveoped a tempate to structure its assessment of the impact of its programmes. The tempate breaks down the potentia impacts above into five dimensions (see Figure 12) which taken together can hep assess the scae of impact. The contribution of a suite of reated programmes can then be assessed on a common basis at portfoio eve, against higher eve objectives. This exampe shows how an organisation can gain traction on compex muti-ayered outcomes in a way that heps programme and corporate management. Figure 12 The British Counci s Impact mode (3P2L) Dimension Participants Practitioners rationae These dimensions measure the size and type of audience reached by a particuar programme. Poicy eaders Leverage Legacy This dimension measures the knock-on effects of a project beyond the directy reached audience. This may incude media attention or foow-up events. This dimension measures the ong-term tangibe impacts of a project such as changes in schoo curricua or education poicy. This measure aows bc to assess whether or not intended outcomes have been achieved. Producing high quaity data 3.5 In our survey, Centra Government organisations identified data quaity as the most important success factor in deveoping an effective performance measurement framework: Data quaity needs to be supported by robust systems of contro which can withstand organisationa changes; Data that are inaccurate or unreiabe wi reduce the Board s confidence in the information and distort the Board s decisions; Information that is out-of-date reduces a Board s abiity to respond effectivey to events, and hinders forecasting.

18 Part Three Performance Frameworks and Board Reporting Accuracy, reiabiity, and robustness of data 3.6 The NAO vaidates PSA data systems to assess whether they are robust, and capabe of providing reiabe, vaid information. For the Spending Review period 2005 2008 we found ony 50 per cent of systems to be fit for purpose. In interviews to foow-up our survey we found itte awareness at executive eve of data quaity issues. We can infer that Boards are often unaware that they are making decisions on, or taking assurance from, poor quaity data. Case exampe 3.7 The Independent Living Fund (ILF) is an Executive Non-Departmenta Pubic Body that assesses the eigibiity of appications for financia support for disabed peope and processes direct payments to successfu appicants. With over 21,000 service users and cycica variations in the number of appications, ILF s Board and senior management team depend on accurate and reiabe data to aocate resources appropriatey to ensure efficiency and short processing times. ILF uses a series of contro checks on each data stream to vaidate resuts. 3.8 The measure award accuracy, for exampe, informs the Board of the eve of accuracy with which appications are awarded ILF benefits. To record the measure and ensure high eves of accuracy, each regiona team manager performs a preauthorisation check of 10 per cent of a awards made by Assessment Officers in their team. This check invoves revisiting each stage of the decision making process to determine whether the correct poicies and procedures were appied and the correct information was used to make the decision. The software used to process appications randomy seects 10 per cent of a awards to the manager. Once the team manager is satisfied that the award has been made correcty he or she then authorises the award for payment. Any identified errors wi be recorded in the system and the manager corrects the errors before the award is made. Additionay, the Senior Operations Manager performs monthy vaidation checks. This invoves a random number of pre-authorisation checks covering a the regiona teams. As a resut, award accuracy is at 95.8 per cent 15 and the Board reported having confidence in the figures. Fuctuations in accuracy or probems with individua team members are addressed by additiona training and if necessary empoyment contracts are terminated where the Assessment Officer is not performing to a satisfactory standard. Timeiness of information 3.9 Our survey showed that the information Boards receive was often more than one month od, and in some cases more than three months od. A recent study by Deoitte aso found that ack of up-to-date information for decision making is a significant barrier for pubic sector organisations. 16 15 ILF Performance report for period Apri 2008 June 2008. 16 Deoitte, Mastering Finance in Government: Transforming the Government Enterprise Through Better Financia Management, 2008, p. 8.

Performance Frameworks and Board Reporting Part Three 19 Case exampe 3.10 The Highways Agency (HA) is a arge executive agency with over 3,500 staff and a budget of over 6bn in 2008. Despite its size and compexity, HA maintains a strict 10 working-days target for the timeiness of information to the Board. Data is coected from six directorates and communicated through severa organisationa eves. HA attributed their success in meeting the target to two factors: A direct ink between objectives, operationa deivery, and the data coection and coation process; Board eve ownership of the timeiness of measures ensures a strong incentive for directors to provide the information in time to avoid gaps in the Board report. Reporting Information in context 3.11 Performance information is most powerfu when it is presented in the context of expectations or comparisons. This type of additiona information in form of targets and benchmarks enabes stakehoders to make a judgement on the organisation s performance and the Vaue for Money it deivers. Targets and benchmarks can act as performance evers to incentivise the behaviour of staff, contractors, and deivery bodies. 17 Target setting 3.12 Targets are important in setting the eve of expectation for Government spending and action, but must be appied with care to avoid producing adverse or unintended consequences: 18 By specifying excessive detai targets can reduce the autonomy of pubic service managers to make on-the-ground decisions; 19 Increasing bureaucracy as a resut of the administrative burden can have a negative impact on the motivation of the workforce. For exampe, HM Treasury found, for each headine PSA, at east a 10 fod mutipier of additiona indicators and targets aong deivery chains. 20 Designing targets that are too stretching can potentiay have a negative impact on managers and staff by being unachievabe and may require empoyees to focus on those targets to the excusion of a others, causing other aspects of the organisation to suffer. 17 Nationa Audit Office, The Use of Sanctions & Rewards in the Pubic Sector, 2008. 18 Nationa Audit Office, Sanctions & Rewards, p. 32. 19 HM Treasury, ibid., p. 11. 20 HM Treasury, Devoving decision making: 1 Deivering better pubic services: refining targets and performance management, 2004, p. 13.

20 Part Three Performance Frameworks and Board Reporting Case exampe 3.13 The Environment Agency (EA) is an Executive Non Departmenta Pubic Body with a arge poicy portfoio, ranging from food protection to ooking after widife; the EAs mode of deivery is aso highy regionaised. To ensure the most effective deivery, EA has shifted away from a prescriptive approach to target setting at output eve. Instead, EA sets high eve outcome targets at a corporate eve and deegates decisions over how to achieve them to regiona and business units. EA requires a business pans to map activities to the outcomes they are deivering. Outcome panning is supported by cear guidance for operationa units. Benchmarking 3.14 Ony 33 per cent of respondents to our survey reported using benchmarks from other UK pubic sector organisations and even ess (24 per cent) used benchmarks from comparabe internationa organisations. Comparison against peers aows organisations to identify scope for improvement and efficiencies. Even between dissimiar pubic bodies, there are common activities that can be compared, for exampe: ICT support functions; Human resources; Asset management; and Procurement of common goods and services. 21 Case exampe 3.15 The German Federa Empoyment Agency uses taiored benchmarking of oca agencies to motivate staff and ocate good practice. The Agency s new structure uses benchmarking to rate how oca agencies are performing against a number of performance indicators. Loca agencies are cassified according to the nature of abour market conditions they face; whether the agency is in a rura or urban environment, and whether there is high or ow unempoyment. By comparing agencies operating in simiar environments, the regiona and head offices have better visibiity of how we oca agencies are being managed given their particuar circumstances. More importanty, the agencies themseves know where they need to improve performance. This added transparency is motivating staff to ensure their office compares we against its peers. Tabes and graphs based on key performance indicators are dispayed at the staff entrance. 22 21 NAO, Progress in improving government efficiency, 2006, p. 64. 22 NAO, Progress in improving government efficiency, 2006, p. 65.

Performance Frameworks and Board Reporting Part Three 21 Presenting performance information 3.16 Boards ony have a imited amount of time to discuss performance information. Performance reports must therefore be presented to focus attention on the most important strategic issues. Poory structured reports increase the time it takes for the Board to understand and focus on the most important issues. Board performance reports can make performance information more accessibe and usefu to Board discussions by: Incuding commentary expaining performance and providing anaysis to hep Board members understand the issues; Making performance information easy to read, by using graphs and charts; Designating senior responsibe owners for individua indicators to increase accountabiity, carify responsibiities and incentivise behaviour; and Updating previousy agreed actions to aow the Board to foow-up on decisions and ensure their impementation. 3.17 Our anaysis of the Board reports showed that whie 78 per cent use commentary to expain performance, 44 per cent of Board reports do not use graphics, 59 per cent do not show previousy agreed action points, and 67 per cent did not show designated target owners. Some of the reports we anaysed were over 100 pages in ength or had no coherent organising principe. Case exampe 3.18 As described in section 3.13, the Environment Agency oversees a diverse poicy portfoio. To aow the Board to focus on the most important performance issues, EA deveoped a Baanced Scorecard aong with a tempate to report on individua performance measures. The consistent tempate makes it easy for Board members to identify important issues and quicky understand performance information. The tempate is concise and ceary structured and provides the Board with a necessary information and anaysis to take action (see Figures 13 and 14).

22 Part Three Performance Frameworks and Board Reporting Figure 13 The Environment Agency s Scorecard 1 2 1 Perspective of set of indicators 2 3 4 5 Rationae and context 3 Rag status and trend 4 Page number of anaysis 5 Last year s status

Performance Frameworks and Board Reporting Part Three 23 Figure 14 The Environment Agency s Indicator Tempate 1 2 3 4 5 6 7 8 9 1 Context of the indicator 2 Owner and business area 3 Reporting period 4 Target and RAG 6 break-down of performance 7 Data source and imitations 8 expanation and anaysis 9 Record of actions 5 Graph of performance

24 Part Four Performance Frameworks and Board Reporting Part Four Using the framework 4.1 The best performance measurement framework is ineffective if it is not used by the Board, senior management, and indeed the whoe the organisation to drive performance. Performance information is used at many eves throughout an organisation and produces different benefits at each eve. 4.2 We have identified three areas in which performance information can prove particuary vauabe by: Linking performance to financia information aows Boards to assess Vaue for Money and make decisions based on cost-effectiveness; Guiding decision-making to improve performance, aocate resources, and enhance organisationa capacity; Creating a performance cuture is essentia to drive performance throughout the organisation. Linking performance to financia information 4.3 Linking financia information to performance information is crucia to the Board s abiity to determine the Vaue for Money of an organisation s activities and make strategic decisions on resource aocation or process improvement. Presentationa reporting, where cost and performance information is reported side by side, is the most basic form of integration. But decision making is best served by providing margina cost information to answer questions such as how much it woud cost to achieve another unit of the intended outcome?, or what investment is necessary to increase the rate of improvement from x to y?. Achieving the eve of integration between financia and performance information needed to assess margina cost is much more difficut and requires a robust costing methodoogy to be appied, for exampe activity-based costing. 23 23 NAO, Managing financia resources to deiver better pubic services, 2008.

Performance Frameworks and Board Reporting Part Four 25 4.4 Ony 29 per cent of the respondents to our survey inked financia information to performance information at programme eve through activity-based costing. This resut is corroborated by a previous NAO survey which found that 53 per cent of Departments find aocating costs between outcomes is a [ ] chaenge. 24 A further NAO survey found that ony eight per cent of Departments have integrated financia information and performance information at Board eve. 25 There are a number of reasons why organisations find inking financia and performance information so chaenging: Attributing and vaidating the impact of activities to intended outcomes is often difficut to evidence; Costing objectives across organisationa divisions requires new forms of recording costs; and Poicy programmes often support more than one objective and therefore a strict aocation of costs to outcomes is not possibe. Case study 4.5 In a recent report, the NAO commissioned Accenture to examine the possibiities of the Department for Internationa Deveopment (DfID) appying a costing methodoogy to one of its Departmenta Strategic Objectives (DSO). 26 DFID s DSO framework has objectives that stretch across divisions. Therefore, costing by division does not give managers a cear sense of how much activities cost by DSO objective. To generate this information, costs need to be aocated to activities that contribute to the DSOs 27 (see Figure 15 overeaf). 4.6 Achieving a Line of Sight through inputs, outputs, and outcomes is essentia to determining the costs associated with a DSO. Ony where outcomes can be inked to the costs of inputs and outputs, can cost-effectiveness of DSOs be assessed. Where it is not possibe to estabish a direct ink, as in some of DFID s DSOs, performance informed budgeting is the best aternative. 24 NAO, Acting on information, 2005. 25 NAO, Managing financia resources to deiver better pubic services, 2008, p. 20. 26 Accenture, Performance Management in Whiteha, 2008. 27 Accenture, Performance Management in Whiteha, 2008, p. 9.

26 Part Four Performance Frameworks and Board Reporting Figure 15 Activity-based costing cost by Activity country Divisions centra Divisions corporate Divisions DSO 1 Promote good governance, economic growth, trade and access to basic services DSO 2 Promote cimate change mitigation and adaptation measures and ensure environmenta sustainabiity cost by Division DSO 3 Respond effectivey to confict and humanitarian crises and support peace in order to reduce poverty DSO 4 Deveop a goba partnership for deveopment (beyond aid) DSO 5 Make a biatera and mutiatera donors more effective DSO 6 Deiver high quaity and effective biatera deveopment assistance DSO 7 improve the efficiency and effectiveness of the organisation Figure 16 Exampe of Activity Based Costing The foowing exampe demonstrates how above methodoogy can be appied to Hiv and AiDS, which is a sub indicator within DfiD s DSO 1. DSO 1.4 (iii) Percentage of most-at-risk popuations reached with Hiv prevention programmes The tota cost for Hiv and AiDS can be derived as foows: miion Cost of DfiD s own managed programmes for Hiv and AiDS 120 % of tota budget spend based on partner government s spend ratio for Hiv and AiDS 84 % of tota mutiatera spend based on historica spend on Hiv and AiDS 31 % of admin spend apportioned to Hiv and AiDS 5 Tota HIV and AIDS spend 240 NOTe The fi gures are for iustrative purposes ony

Performance Frameworks and Board Reporting Part Four 27 Guiding decision-making 4.7 Performance information, especiay when inked to financia information, has enormous potentia to benefit Board decision-making. To get the most out of performance information, the most engaged Boards use it to make decisions about strategic direction, resource aocation and organisationa capacity. Using performance information to inform Board decisions ensures that funds are directed at the most pressing issues and the most effective activities of the organisation. 4.8 Our survey showed that Boards frequenty use performance information to monitor performance, but use it ess to aocate resources, enhance capacity, and adopt new approaches or change work processes (see Figure 17). Figure 17 The use of Board reports Activity Aways % mosty % rarey % never % no response % Monitor achievement against 83 14 0 2 0 performance objectives identify programme probems to 61 32 7 0 0 be addressed Manage risks 61 32 5 0 2 Coordinate efforts within 38 44 10 5 3 organisation or with externa organisations Manage resources operationay 37 40 14 9 0 Aocate resources to achieve 33 36 20 10 0 strategic aims enhance organisationa 28 41 28 3 0 capacity Adopt new approaches or change work processes 16 41 35 8 0 NOTe based on respondents from 42 organisations Case exampe 4.9 Since the Crimina Injuries Compensation Authority (CICA) introduced its new performance measurement framework, it has become a major infuence on decision making. For exampe, the Board and senior management take decisions based on the scorecard to shift resources from ower to higher priorities. The scorecard aso informs the Board and senior management of what performance is possibe at a given eve of funding and makes a trade-offs visibe, eading to some KPIs being parked whist others are focused on. An additiona benefit is the fact that CICA can now mode different outcomes and funding eves and base bids for additiona funding on robust evidence.

28 Part Four Performance Frameworks and Board Reporting Creating a performance cuture 4.10 The cuture of an organisation has a significant impact on performance. A dynamic performance cuture wi see motivated staff engaging criticay with existing practices, and coaborating and communicating to effectivey impement the organisation s mission. Creating a performance cuture, where each member of staff knows how he or she contributes to organisationa objectives can be amongst the greatest benefits of introducing a performance measurement framework. During the impementation process of a performance measurement framework three factors prove particuary important to successfuy deveoping a performance cuture: Leadership in the form of active support and a firm commitment from senior executives to impement a framework throughout the organisation; Communication of the impementation process must be transparent and comprehensive, expaining the framework and how the performance of each individua inks to organisationa objectives; and Engagement from a eves of the organisation must be sought at each stage of the process to increase buy-in and ownership by a staff. 4.11 Our survey found generating buy-in amongst the greatest chaenges in impementing a performance measurement framework. Survey respondents gave exampes of these difficuties; resistance from staff in accepting a transition to persona objectives, resistance to changes in organisationa processes, and communicating the framework to a arge number of staff from a variety of discipines. The Cabinet Office s Capabiity Reviews recenty found that two thirds of Departments have probems in their organisationa cuture, incuding the need for staff to fee abe to speak up and chaenge. 28 Case exampes 4.12 Our foow-up interviews found exampes of organisations in Centra Government that have had success in impementing performance measurement frameworks and transforming the organisationa cuture and staff attitudes in the process. The Crimina Injuries Compensation Authority s (CICA) response in 2008 to criticism of its performance incuded a business process review and a new performance measurement framework. CICA adopted a traditiona Baanced Scorecard approach. It wanted to become a singe sight organisation, which has one objective and a cear vision of how to achieve it. As a resut it reviewed, mapped and streamined key processes. At first the changes caused concerns among staff about the mechanisation of their work. However, the commitment of senior management to the programme ed to benefits and efficiencies which have become increasingy accepted by staff, and more staff now make suggestions as to how further benefits can be reaised. The Board review the scorecard quartery, 28 NAO, Assessment of the Capabiity Review programme, 2009.