OLACEF Controller General s Office of the Republic of Bolivia November Pablo A. Herrera Suárez

Transcription

1 EUROSAI -OLACEFS: IV EUROSAI OLACEF SYMPOSIUM CEDEIR: Special Technical Commission for Evaluating the Performance of SAIs and Performance Indicators METHODOLOGY FOR MEASURING AND EVALUATING THE AUDITING IMPACT ON THE SAVINGS AND ON THE GOOD USE OF PUBLIC RESOURCES (Process, Product, Results, Impact) PRESENTATION Lima, Peru, November 17-18, While methods of evaluating results are limited by the complexity of the State s actions, there will always be areas in which to apply them. Furthermore, it must be taken into account that the process of transition towards an administration aimed at results and service to the client, has been developed, according to CEPAL (1998), at two levels: (i) (ii) The redefinition of the logic of internal functioning of public organizations (governability structure of organizations, incentive systems, evaluation and control, supervision model, human resources management, and financial management, model among others); The interaction among public organizations and the rest of the economy (separating the functions of the State, of government and the functions of the private sector). In a market economy, private companies know that they are effective when they make a profit, and they are sustainable when that continues to occur. Their indicators to measure it are: profitability and the market value of the company itself. But how can the effectiveness and the sustainability of institutions and programs that produce public value be measured? With what criteria can a fiscal control program be evaluated? Can the effectiveness of an entity that regulates administration and governmental control systems be measured? With what indicators? And how can one know when effectiveness is sustainable? How to know if the allocation of resources was more or less efficient? If for private companies, knowledge about their current and future earnings is a matter of survival, for the people and their representatives, knowledge about effectiveness and sustainability in achieving public value is a right and, for public decision makers, it is a duty. As Shack et. al. indicate (1996), the key to an effective modernization process is especially represented by changing the current organizational culture based on regulations to one in which emphasis is placed on the results of public management clearly oriented toward user satisfaction and efficient, effective, economic operations through a substantial change in the incentive system for public servants. The evaluation of the results is usually based on adhering to effectiveness, efficiency, economic and quality criteria in the State s intervention. Follow-up on these criteria is of vital importance, since they are basically determined by the work of the individuals responsible for the different areas of public activity. Additionally, these criteria form part of the aspects to consider in a process of changing the organizational culture of the public sector that involves the substitution of a bureaucratic public 1

2 administration model (emphasizing regulations and procedures) with a managerial one in which processes and results prevail. The measurement of performance is usually carried out using indicators (quantitative and qualitative). As specified by the World Bank (1998), the use of indicators helps to deal with the issue of knowing when a certain action was successful or not. Performance indicators, defined as input, product, results and impact (IPRI) indicators, are generally instruments for measuring the main variables associated with attaining objectives, which in turn constitute a concrete qualitative or quantitative expression of what is to be achieved in relation to a specific, established objective. CONTENTS I. INTRODUCTION I.1 PRESENTATION I.2 OVERALL OBJECTIVE I.3 SPECIFIC OBJECTIVES I.4 JUSTIFICATION OF THE ISSUE I.5 IMPORTANCE OF THE ISSUE IN RELATION WITH THE INTOSAI STRATEGIC PLAN II. III. IV. PERFORMANCE EVALUATION BASED ON STRATEGIC MAPS II.1 Conceptual aspects II.2 Strategic maps for the creation of public value II.3 Principles that sustain the Strategic Maps of the Balanced Scorecard PRINCIPLES THAT SUPPORT THE DEVELOPMENT OF AN INSTITUTIONAL MODEL FOR SAIs INDICATORS IV.1 Selection principles for performance evaluation indicators IV.2 Methodological Aspects: Input product results impacts - consequences IV.3 Input Indicators IV.4 Process Indicators IV.5 Results and Impact Indicators IV.6 Indicators of cost effectiveness of ATI: V. EX POST EVALUATION VI. BIBLIOGRAPHY 2

3 METHODOLOGY FOR MEASURING AND ASSESSING THE AUDITING IMPACT ONT THE SAVINGS AND ON THE GOOD USE OF PUBLIC RESOURCES. (Process, Product, Results, Impact) I. INTRODUCTION The first task of any theory is to clarify confusing terms and concepts Only after reaching an agreement on the terms and concepts can there be hope of considering matters easily and clearly, and of sharing the same point of view I.1 GENERAL OBJETIVE To make a proposal to the Supreme Audit Institutions (SAI) regarding a methodology that allows measuring the impact, intervention and/or frequency of the control and audits performed by the SAIs on savings and on the good use of public resources. This methodology would make it possible, in general terms, to determine whether the programmed work (operating and strategic) produced the desired effects on the Stakeholder (group involved) that, among others, includes the citizenship and the audited entities themselves. The metrics determined could show if these effects are or are not attributable to the intervention of the SAIs. I.2 SPECIFIC OBJECTIVES: Specifically, the proposal is for a methodology, which by means of quantitative and qualitative metrics would make it possible to determine unforeseen consequences in the desired effects and also determine whether the improvements and benefits of the audit were possible as a consequence of it, or if they would have occurred without it. Regarding significant information for managing the results obtained by applying the method, valuable facts about the what, why and what for, could be obtained from the audit plan carried out for each SAI. I.3 JUSTIFICATION OF THE ISSUE: Organizations can not manage what they are unable to measure Likewise, what can not be described can not be measured Undoubtedly, for the Supreme Audit Institutions, the worldwide control scenario has undergone important mutations that inevitably mean rethinking the development of new models for controlling Supreme Institutions, models that must bring together particular characteristics that will allow dealing with the new requirements and demands of citizens, Public Administrations and others who make up a Stakeholder that demands a special contribution from Control Organizations regarding efficiency, effectiveness and integrity in public entities. Therefore, the proposed issue has unquestionable relevance in two significant aspects of each SAI. Firstly, regarding the valuable information that the aforementioned metrics can provide for formulating the financial plans of SAIs, and their own operating and strategic action plans, as well, since even public sector entities and non profit organizations must show how they create public value for those who form part of them and other interested parties. 3

4 On the other hand, the incorporation of this methodology could make it possible to know, even with specific quantitative volumes, about the responsibility corresponding to each SAI in aspects concerning Public Administrations as critical as the greatest efficiency and effectiveness, a culture for public ethics and the quality and productivity of work projects, which certainly have definitive responsibility in the ever closer relationship between the user and the client, concretely the citizens, that governments and Public Administrations must promote. I.4 IMPORTANCE OF THE ISSUE IN RELATION WITH THE INTOSAI STRATEGIC PLAN An organization must measure the few key parameters that represent its strategy for the creation of long-term public value. The methodology proposed for discussion could produce a connection that would notably empower the aforementioned process, in that it necessarily incorporates elements that bring together vital aspects of auditing, such as objectives, goals and indicators, which should be monitored satisfactorily, thereby producing areas of direct contact with the purposes of planning. In this way, in a Balanced Scorecard, as a management control support for measuring impact, would provide information at the macro level of OLACEFS regarding economy, efficiency, effectiveness and ethics in the SAIs and in Audited Public Administrations, as well as information about the behavior of financial indicators, citizen perception, process improvement and learning and knowledge management. II. PERFORMANCE EVALUATION BASED ON STRATEGIC MAPS A measurement system based on institutional strategy will solve the problem of how to communicate it and put it in operation II.1 Conceptual aspects The objective of an Evaluation is to obtain a technical, formal opinion on the performance or behavior of a process, product, result or impact derived from the object of investigation. It implies compiling a theoretical, methodological body on the standards, criteria and indicators that an instrument uses to evaluate the performance of the object of investigation qualitatively and quantitatively. The orientation of the evaluation is of a self-regulatory nature; it must favor correct decision making on high-priority, necessary actions capable of producing the desired results, impact and consequences. It is not a uniform concept, so it could be considered as the sum of many different factors, sometimes diverse among themselves, that make up a common element or concept. In general, the evaluation process comprises three different parts: 1. Obtaining and handling information; 2. Formulation of judgment based on criteria and indicators; and 3. Decisionmaking. In the SAIs, the type of Evaluation is differentiated by the function it fulfills: 1. Diagnostic: Function of the initial evaluation 2. Regulatory: Allows adjusting decisions on intervention based on the results and impact 3. Preventive: Facilitates estimating optional courses of action on expected results and impact 4

5 4. Feedback: Informs public entities on the progress of institutional improvement 5. Follow-up and control: Verifies the implementation and fulfillment of recommendations to the audited entities based on findings. Institutional Evaluation obliges executives and involved personnel to identify, clearly state and analyze the existing mission, outlook, policies, objectives and work processes, thereby acknowledging the status of the situation and performance attained with the organizational culture in effect. All this is transformed and oriented toward strategic planning instruments, which allows the reallocation of investments, technology and human resources aimed at new substantive purposes that favor the results and impacts of the expected development. Also, in this way formal backing and transparency is lent to the activities to which management is committed, avoiding resistance, deviations and other internal or external management errors. A strategic performance management system not only ensures results, but feedback on new strategic processes. An emerging possibility of institutional evaluation entails reengineering central processes, for the purpose of simplifying procedures and reducing bureaucracy, minimizing costs that do not add public value and favoring the fulfillment of requirements for products and/or services in the time periods required to generate the desired results and impacts. All this is possible with a Balanced Scorecard that aids in aligning the array of organizational resources and the hierarchy of objectives and focuses attention on the substantive variables capable of adding public value internally and to the group involved (Stakeholder) in the scope of the operation. Dynamics of organizational resources = human capital + information capital + organizational capital = Processes that create value Outstanding performance = Objectives + Indicators +Strategically focused organization Public management is the process that strategically links the actions of an entity to its mission and objectives, in accordance with the priorities established in the development plans, for the purpose of guaranteeing maximum transparency and harmony between the decisions resulting from planning and the actions reflected in the budget. Without a systemic description of the Strategy, executives are not able to communicate it easily, neither among themselves nor to the employees. Without shared comprehension of the strategy, executives are unable to produce sufficient alignment concerning it. Without focus, new strategies cannot be implemented. Therefore, five management principles for focusing attention on the strategic performance of SAIs are suggested: 1. Translate the strategy into operating terms. 2. Align the organization with the strategy. 3. Convert the strategy into a daily task for everyone. 4. Convert the strategy into a continuous process. 5. Mobilize the change through the executive leaders. Performance Evaluation is associated with the judgment made once the action o the intervention of any undertaking (plan, program, or project) is completed. Its purpose is to answer key questions on how the action was carried out, if the objectives (concretely, to the extent these were fulfilled), and the level of satisfaction of the target population were fulfilled, among others. Briefly, the purpose is to evaluate how good or how acceptable the performance of a certain public organization has been with the objective of taking the necessary action to perfect the administration. 5

6 The Performance evaluations most frequently mentioned by Caiden and Caiden (1999) are: inputs, activity level, results (outputs), impact (outcome), productivity, costs, user satisfaction and service quality and response time. These measurements are used to verify the economy, efficiency and effectiveness with which a public organization carries out its activities. Economy is related to the capacity to operate at the lowest cost level possible and can be estimated with the measurement of the inputs and comparisons with an established pattern, or benchmarking. Efficiency represents the relation between inputs and results: it can mean minimum use of inputs to obtain a certain result or the production of maximum results for a determined level of inputs. Effectiveness defines the success of the undertaking and focuses on the results evaluated through their impact (efficiency and effectiveness combined). In performance evaluation, Newcomer (1996) identifies inputs, operations or processes, output and outcome as the most frequent performance evaluations. Inputs are the resources available for implementing a determined undertaking (plan, program, or project); Operations or processes represent the defined procedures through which the undertaking must operate; Product and results outputs are the activities or events created by the undertaking; Impact outcomes represent the objectives of the undertaking, the scope of which is viable due to the operations and products of the undertaking. These measurements are differentiated by their purpose in three types of evaluations: Analysis based on problems; Performance evaluation and impact evaluation (Newcomer, 1996). 1. Analyses based on problems are performed if the existence of problems resulting from fraud, abuse, or incorrect management or alignment of the undertaking is suspected. In this case, audits are conducted to identify the causes and, eventually, to offer solutions. 2. Performance evaluations tend to demands for evidence on the performance of the Entity or to the need for information on the results of the teams for the undertaking or of those who work with the budget, in relation with the improvement of undertakings. 3. Impact evaluations are used when data is needed to justify the continuance of extremely expensive or politically vulnerable undertakings. They include the measurement of how much a certain undertaking progressed in the desired direction on certain target population. They present the difficulty of establishing the cause-andeffect relationship between the undertaking implemented and the change which occurred. The evaluation, with the objective of perfecting the allocation of resources among undertakings, may use information on inputs, costs, output, outcomes, quality and user satisfaction. The evaluation, with the objective of improving the undertaking, must also take into account the analysis of operations and processes. Thoenig (1999), who emphasizes the role of evaluation for improving undertakings, highlights gathering and producing information necessary for daily decision-making and for performing a self-evaluation that will have an impact on the routine work of the users themselves. 6

7 With benchmarking, he mentions the exchange of information on good management practices with an important instrument for the development of a performance culture. The products and impact of the undertaking (plan, program, project) are necessary but insufficient measurements for evaluation oriented toward transparency. According to Levy (1999), for an evaluation to produce accountability information (rendering of accounts and transparency), it is necessary to provide explanations, the meaning of which must be understood and accepted by all interested parties. Even though the control of results increases the possibilities of accountability due to the fact that the objectives and means involved in public policy are more explicit, it can result in an undemocratic instrument if the definition of Indicators and/or results is a process determined by few agents. Accountability refers to the rendering of accounts and to definition of the terms on which accounts will be rendered. This definition must be joint in order to create transparency; thereby creating trust between those who govern and those who are governed. (Levy, 1999). The taking of satisfaction surveys of public service users (public entities, in the case of SAIs), on its part, requires the development of a specific methodology. The evaluation of the quality of the services on the part of consumers is carried out by means of the comparison between the expected services and the perception of the services received. The expectations of consumers activate a fundamental role in the process of evaluation of services and are influenced by four factors: the personal needs of consumers, their previous experience, the comments they hear from other consumers and external communication (Zeithaml, Parasuraman and Berry, quoted in Pesquisa ENAP, 2001). It is almost a consensus that in the performance evaluation of public spending the Criteria for analysis are Effectiveness, Efficiency, Economy and Quality of the State s intervention, regarding Governmental Control.. The follow-up on these Criteria by SAIs is vital for issuing a professional opinion on the performance of those responsible for governmental control. Additionally, these Criteria contribute in evaluation processes, strategy formulation and changing the organizational culture of SAIs and public entities in general.. While the systems for performance evaluation presents limitations on the complexity of the action of the State and particularly of SAIs, there will always be specific areas in which it will be plausible to apply them. The key to the success of modernization processes of SAIs lies especially in the change processes of Organizational Culture aimed at results and impacts of their implementation, a clear orientation of service with added public value for its target public, user or client, and the efficiency, effectiveness and economy of the truly productive internal operations processes. In the performance evaluation, the fulfillment (or lack ) of the objectives in terms of input, product, result and impact, must be taken into account, since this provides a clear idea of the cost of achieving objectives and the effectiveness of the programming. That is: If the general and specific objectives of the intervention were fulfilled in terms of inputs involved (or productive factors). If savings were produced in the use of the inputs (economy). If an appropriate combination of inputs was used (efficiency). 7

8 What level of productivity was attained (how well the use of the factors was combined). How many products were produced as a result (effectiveness) regarding the goals defined. To what degree the general and specific goals were achieved (evaluated in terms of results and impact). Additionally, effectiveness in fulfilling public values to be attained, proposed in terms of economy, efficiency, effectiveness and quality (if they were established ex - ante) should be taken into account. Regarding the base of Criteria defined, for the purposes of synthesis and as a complement to the developments of Parry (1997), the relation between Indicators for measuring performance and those oriented toward performance evaluation could be systemized in the following manner: Fig: Indicators for Measuring and Evaluating Performance Public Expenditure Programming Imput Efficiency Productivity Product Effectiveness quality Effectiveness Result Impact Imput Efficiency Product Effectiveness quality Productivity SAIs create differentiated, sustainable public value by means of strengthening their Intangible Assets: Human capital, databases and information systems, sensitive and high quality processes, relations with clients, innovative capacity and organizational culture. This makes it possible to draw closer to the new economy of knowledge and public services. SAIs achieve success through the implementation of internal processes that have the support of their intangible assets (learning and growth). With a Balanced Scorecard, the fiduciary perspective reflects the objectives of a significant group of contributors: the funds derived from contributors and donors that provide financing to SAIs. The satisfaction of stakeholders, financiers as well as clients (audited public entities), in accordance with the Mission, create a strategic architecture of efficiency and effectiveness issues that reflect the issues of productivity and growth that creates institutional public value for them. II.2 Strategic maps for creating public value The framework for the creation of public value in SAIs and Public Sector Entities in general, is similar to the private sector framework, although with some important distinctions: 8

9 1º The maximum definition of success for SAIs is their Performance in fulfilling their Mission. 2º SAIs have a wide variety of Missions (due to delegated authority and competence); therefore, they have to define their Social Impact and their strategic Objective in a different, and specific way. Fig: Strategic Map: Supreme Audit Institutions. The Mission Fiduciary Perspective If we are successful, how will our contributors and/or donors see us? Client Perspective To achieve our Vision, how must we present ourselves to our clients? Perspective of the Internal Processes To satisfy our clients, donors and financiers, in which processes must we stand out? Learning and Growth Perspective To achieve our Vision, how must we learn and improve our organization? The Strategic Map shows how SAIs create Public Value. Therefore, it is a visual representation of the cause-and-effect relations among components of the organization s strategy. The Strategic Map adds a second layer of details in which the temporary dynamics of a strategy are illustrated; it also adds a layer of detail that increases the clarity and focus of attention. In this way, a Strategic Map provides a uniform and coherent way of describing a Strategy, so that objectives and indicators can be established; thereby providing the critical link between the formulation of a strategy and its execution. SAIs must focus on the few key internal processes that produce the proposed differentiated public value, which because they are critical, allow improving productivity and sustain the capacity of the organization. A Strategic Map also provides a regulatory checklist of the components and interrelations of a strategy. If a strategy is missing an element in the Strategic Map model, the strategy will probably be defective. Omissions of this type in a Strategic Map generally lead to disappointing results. 9

10 II.3 Principles that support Balanced Scorecard Strategic Maps Strategic Maps are based on several Principles: 1. The strategy balances contradicting forces (income growth cost reduction); (productivity sustainable growth); (short term long term). 2. The strategy is based on a public value proposal differentiated for the client: Best total cost; Leadership of products; complete solutions for clients; blocking system (defining authority and competence among entities). 3. Public value is created through internal management processes: Financial and Client perspectives define results; Internal perspective and that of learning and growth create and sustain public value. Internal processes are classified in four groups: a) Operating management b) Client relations; c) Innovation; d) Regulatory and social processes. 4. The strategy is composed of simultaneous and complementary issues (processes): Operating processes; client relations; innovation; regulatory and social processes. The strategies must be balanced and must incorporate at least one strategic issue from each one of the four internal groups. 5. The strategic alignment determines the public value of intangible assets. Public value is obtained by the synergetic linking of the same by means of three approaches: a) Families of strategic positions; b) Strategic portfolio of Information Technology c) Agenda for organizational change The perspective of Learning and Growth of the balanced scorecard classifies them in three categories: a) Human capital (abilities, skills, knowledge of employees); b) Information capital (Data bases, information systems, networks and technology infrastructure); c) Organizational capital (culture, leadership, alignment of officers, teamwork, knowledge management). SAIs achieve a high level of preparation for creating public value when: The ability of human capital in the families of strategic positions is closely aligned with the issues or strategic perspectives. Information capital provides the vital infrastructure and technological applications and fosters extraordinary performance. Culture, leadership, alignment and teamwork are elements which reinforce changes in the organizational atmosphere necessary for the strategy. Tangible physical and financial assets differ from the public value generated by intangible assets, since in the case of the latter: 10

11 The creation of public value is indirect. The public value is contextual. The public value is potential The assets are in related groups. Fig. 2.- Continuous process that describes what Value is and how it is created Mission: Why we exist Values: What is important to us Vision: What we want to be Strategy: Our game plan Strategic Map: To translate the strategy; Measure and Focus. Goals and initiatives: What we need to do Personal objectives: What I need to do STRATEGIC RESULTS: Cooperative, motivated, and well-trained officers Effective and productive line operating processes Clients who are taken care of and satisfied (public entities). Fiduciaries: satisfied donors and financiers III. PRINCIPLES THAT SUPPORT THE DEVELOPMENT OF AN INSTITUTIONAL MODEL FOR SAIs In accordance with their authority and competence, Supreme Audit Institutions have diverse purposes and interests in the evaluation of their activities; for example, some emphasize institutional planning, technological processing, organization and the production process. Others are oriented toward granting certificates to private audit firms, to governmental and internal auditors; others focus more attention on training and transfers of public administration competences. There are others interested in investing in regulatory projects such as governmental control organizations; additionally, they give priority to controlling fortunes by means of sworn declarations of the property and income of public officers, among other interests. Regardless of the focus or diversity of objectives of SAIs, in all cases it is important to recognize a set of evaluation principles that sustain the development of a model for institutional evaluation based on results and impacts, as follows: 1. An extensive, overall diagnostic evaluation of the results and impacts is sought to facilitate making decisions on governmental control in SAIs. 2. Public administration of SAIs is influenced by variations in the social-political-economic system that exist in the microenvironment. 3. Effective performance of SAIs can have a diminished impact of their interventions due to: informality, rotation and irregularities of high authorities in audited public entities. 11

12 4. There will always be a subjective area in the evaluation and interpretation of the findings and data obtained. For this reason, the structured and systematic treatment of the quantitative and qualitative information is imperative. 5. Based on their authority and competences, SAIs have multiple objectives; for this reason, a multiple set of Indicators is required for the evaluation of results and impacts. 6. Public administration of SAIs is developed long term; so their results and impacts must also be measured over relatively long periods of time. 7. In addition to the financial perspective, SAIs require other non-financial perspectives to evaluate their results and impacts. 8. The results and impacts of SAIs public administration is expressed by diverse groups involved (stakeholders) that, include, among others: citizens, audited entities, donors, financiers, etc. Not all these groups express the impact received directly and reciprocally. The measurement of performance in results and impacts and its subsequent evaluation should not be limited to being just a source of gathered data, but should provide as well critical elements that are useful for decision making, especially for better allocation of public resources. In this sense, the measurement of performance must be oriented toward those key elements indicated in the strategic objectives of the organization and the effects of this process must be thoroughly discussed and communicated among all the actors in the process (public organizations, the citizens, etc.), for transparency and accountability. The information related to the objectives and the progress toward achieving them must be kept open, through modern communication systems such as the Internet or private Intranets, bulletins, annual reports, etc. IV. INDICATORS Be careful what you wish for; you might get it An Indicator is an operating enunciation (facts, representations, expenses) on an aspect of life and its actors, which makes it possible to identify its quality and quantity within an established time. It is a set of variables to be measured or monitored. They are quantitative and qualitative indexes (figures or enunciations that make the situation known at a given time) that make it possible: to identify the fulfillment or achievement of objectives sought; to identify if there are deviations between what was programmed and what was executed; and to deduce what the current situation is concerning the achievement of objectives. (Carrasco, G. 2002). Among the main purposes for the design and development of Indicators, the following can be pointed out: That institutional management is based on data, facts and situation. To create a culture of continuous measurement and improvement. To add public value to the decision making process in the organization. To verify the determining factors (key factors) for success. To establish criteria for comparison with other similar entities. To generate information to analyze effectiveness, efficiency, economy and quality. To analyze the fulfillment of plans, objectives, goals and results. To fulfill the objectives for which it was created, a good Indicator must have three parts: 12

13 a) Quality describes what and sometimes how to measure; b) Quantity, how much and at what value something has been created or acquired; c) Finally, when the quality and quantity of the item measured are attained; the time. In another sense, to achieve the strategic course, it is necessary to establish Indicators that make it possible to monitor and follow up on predetermined objectives and based on this, to analyze the behavior of the organization in the short, medium and long term. These Indicators must be formulated, among other aspects, within the framework of the questions: What is going to be done? How is it going to be done? When will it be done? Who will do it?, among other aspects. Management based on Indicators places emphasis on the importance of the entity s having plans with measuring instruments that allow measurements to be taken for follow-up, evaluation and governmental control purposes that goes from strategic levels to operating levels. The purpose is to establish concrete objectives that can be measured over time to determine strategies, allocate resources, assign responsibilities, start plans, programs and projects, and establish the controls to determine to what extent the proposed objectives for each level are fulfilled. To measure management, instruments must be developed, such as: Indicators, Balanced Scorecards, statistical analyses, self-diagnosis, and perception of satisfaction, among others, that integrate systematization activities, from obtaining data at the operating level to the information for high-level decision making. There is a great variety of classifications and types of indicators for which, in the case of SAIs, the classification of the following Indicators is proposed: Input; Process; Product; Result, Impact. Performance is usually measured using Indicators (quantitative and qualitative) that reflect or represent the status of a situation within a scale of pre-established nominal value. The application of Indicators clarifies whether an undertaking (strategy, program, project) has been successful and to what degree, within a specific scale. Evaluation is a more extensive and complex process than simple measuring or qualification. In contrast to Evaluation (which is always a process, is extensive, involves interpreting data, and covers everything) Measuring is punctual, restricted, involves obtaining data, and is partial. Indicators should be focused on what is most important for the organization: its strategy. Be careful what you wish for; you might get it. Measuring is a powerful motivator. Executives and other employees strive to obtain good results in any Indicator that is selected; especially if the Indicators are in connection with a compensation through incentive plans. Therefore, before deciding what to measure, questions should be asked. What is to be obtained? What will the objectives be? (How are success, results and impact defined?). So institutional objectives should be linked to cause-and-effect relations, which implies elaborating a Strategic Map in which the objectives are interrelated by means of a basic pattern of causes (actions) and effect (results and impact). 13

14 IV.1. Selection principles for performance evaluation indicators To identify and select the Indicators that best express the performance levels in any undertaking (strategy, program, project), it is imperative that a minimum set of Principles for selecting Indicators be pre-established. (U.S. National Center for Public Productivity (1997). Committee on the Modernization of Public Administration in Chile (1996)). Principles Relevance Independence Cost Reliability Simplicity Timeliness Non redundant Focused on controllable areas Inclusive Description of selection of indicators Address essential processes and products that the entity or undertaking develops. Not conditioned to external factors, such as related activities of third parties (other public or private entities). Reasonable cost when obtaining information to elaborate indicators. Consistent, independently of who carries out the measurement. Easy to understand, free of complexities. Within the time necessary for its timely measurement, reporting and action. Unique and non repetitive or similar. Centered on areas that can be corrected with the new performance, generating clear direct responsibility of those involved. Include all relevant actors, ensuring legitimacy and commitment and unifying purpose of objectives and indicators. Additionally, the Indicators for products, results and impacts must be credible (changes are an effect of the activities) and independent; a product and a result as an effect are not the same. Finally, it is recommended that the Indicators at any level of the chain of public value generated have the following characteristics: Friendly: easy to understand Measurable: quantifiable in a way that is useful for follow-up and evaluation purposes. Verifiable: Sufficient and accessible evidence in order to verify the object being measured. Possible: Possible to obtain lower costs in relation to the undertaking under investigation. Timely: Sufficient timeliness for follow-up and evaluation purposes. IV.2 Methodological Aspects: Inputs product results impacts - consequences For an effective performance evaluation system to exist, measurement criteria must be established in order to allow for an objective evaluation basis, eliminating any subjective criteria. The measurement criteria are applied to any evaluation, whereas procedures may change, depending on the nature of the object being evaluated and the scope of the review. The key questions asked when taking a measurement are: What is to be measured? How will data be obtained? Who will take the measurement? Which measurement mechanisms will be used? How will they be processed? Which allowances for variances will be determined? How will they be interpreted? Who is interested in the results of the measurement? What will be done with the results? 14

15 As mentioned previously, in general the Indicators for measuring Performance are grouped as Input, Product, Results, and Impact Indicators (IPRI). All are instruments for measuring the main variables associated with complying with the Objectives of the Entity or undertaking to be evaluated. Therefore, they constitute a concrete qualitative or quantitative expression of what needs to be attained with a specific established objective. Indicators are used for the follow-up and evaluation of SAIs institutional administration. Each executing Entity must enter information on operations, objectives, Indicators and goals into the follow-up and evaluation system for its institutional administration. Each Indicator will be used to compare the progress or fulfillment of operations, products, results and impacts based on its planned goals. What do SAIs generate (outputs)? Rules, regulations, certifications, audit reports, verifications, follow-ups, requirements, custodies, registers, fulfillment of macro and micro social plans, and others, such as input-products); and statistics on products, investments and resources used as results of processes and management. For whom are they produced (outcomes)?: For public entity clients and individuals or legal entities involved, as impacts effects that must be derived from them. The public value as an expectation of the State, citizen users and beneficiaries of the information and regulations. How can Indicators be identified and elaborated? By identifying products of the SAIs that can be observed directly and cases in which indicators must be determined because the impacts and effects are only observable through indicators. Practical matters: How to establish conceptually what is meant to be observed and how to identify what can be observed in practice, that is, the viable indicators, description of the indicator, identification of the method for calculation or observation, quantitative or qualitative treatment, classification of the indicator according to the types of effects to which it contributes, in-depth indicators and quick indicators. Methodological matters: What determines the validity of the indicators and up to what point is it possible to identify the diverse factors that affect the indicator but are not a result of the action of the organization or undertaking. How is the indicators bank of the SAI put together? Considering: the bank of indicators is an instrument of participation, transparency and negotiation among the actors involved in SAIs. Participatory, interdisciplinary elaboration: contributions of technicians, operations personnel, executives, advisors and experts; sensitive attention to the problem of conflicting interests and the manipulation of indicators in favor of one or another; the legitimacy of indicators and their utilization for the administration of incentives, and the evaluation and taking of strategic decisions. How are the SAI indicators interpreted? Once the indicators are established, it is advisable to design an Information System. To the extent possible, this system must have characteristics that guarantee its quality in terms of: timeliness, relevance, homogeneity and consistency; which allow creating a solid base line over which it is possible to elaborate a Strategic Map (Balanced Scorecard) of the indicators system. It is important to generate and recognize flexible criteria for the design of information systems that will allow lending support to the managerial authority, integrating the systematization activities of the Entity, from gathering data at the operations level to the information for decision making at management level. 15

16 For an information system to be effective, it must have sufficient flexibility to respond continuously to changes in information and technology. VARIABLES Operating Cost Process times Product quality KEY FACTORS FOR SUCCESS Resources Processes Products User Clients ECONOMY EFICIENCY EFECTIVENESS What should be considered when making an analysis of the Indicators? An Indicator itself is data. Therefore, for this to serve as a management instrument, the following criteria for analysis of Indicators must be taken into account: The result obtained must be analyzed in an integral manner, according to the strategic planning system, determining the extent to which the result indicates fulfillment of the strategic and operating objectives. The management maps of the balanced scorecard are analyzed with a deductive approach, from the general to the specific: attending the whole by centralized and decentralized levels to investigate the cause of the result of a specific indicator. Some indicators focus on the limit between one process and another; therefore, the result of one affects the next, connecting the SAI s chain of public value. For analysis, in addition to comparing the result with pre-established ranges, comparisons should be made with previous periods to see its evolution over time. If data on the same activity from other SAIs are available, the results can be compared for reference. Fundamentally, the institutional evaluation shall apply indicators to substantive areas (related to the mission) of the SAI, with respect to support areas, that is, in cases where a result is affected by lack of training or technological or financial resources, among others. It should not be forgotten that there are many internal and external factors (unrelated to those responsible for strategic management) that can cause the results and impact on public entities to vary. 16

17 IV. 3. Input Indicators Input Indicators measure the quantity and quality of the inputs or resources planned and supplied for an activity (results of activities). Normally they quantify physical, human and financial resources used in the development of actions. Usually they are expressed in amount of costs allocated, number of members in a working team, or the number of hours or days worked. They allow a systematic follow-up on the activities that the institution carries out. By themselves they do not express the extent of progress in relation with objectives. However, they can be used with an ex ante evaluation criteria, and in this direction they take on a different dimension, since they acquire the possibility of evaluating the possible impact o an undertaking (plan, program, project) in advance, which makes it possible to select the option that predicts the best result and impact in the horizon foreseen for the undertaking. There are ex ante evaluation instruments, such as cost-impact analysis, the purpose of which is to sound out the plausibility (profile, pre-feasibility, feasibility) of an undertaking or decision, regardless of its structure: project, program, plan, entity, sector, government. The greater demand is in the wealth of information available, such as instruments for consistent measurement. In this sense, ex ante evaluation is a matter of selecting among the technically plausible options, the one that produces the greatest impact at the lowest cost. For this, the valuation criteria must be the Expected Impact in relation to the cost of implementation for each plausible option. To perform an ex ante evaluation, the following is required: 1. A goal of the objective of the Impact and of the results and products to be generated. 2. Impact as well as Product indicators. 3. Cost of each plausible intervention option for the undertaking. In this case, Products refers to the goods or services that the undertaking must generate for the achievement of the expected objective of the Impact, based on the proposed options. Products should not be confused with activities carried out to produce them. Each option will specify the Products expected. These may or may not be different. In case the products among options are the same, what will vary will be the quantity or the manner of delivery, the results of which will affect the Impact on the target public. Therefore, Products must be precise, measurable, realistic and possible to attain in the time horizon established and focused for the target public, in this case, audited public entities, for example. For each undertaking or intervention option, the expected Impact of its implementation must be specified. In this way these estimates become Impact Goals, which can be diverse and independent of each undertaking according to their purpose. The Impact Goal is the quantification of the Impact objective within a certain time lapse. What is important is that the Impact goal must be estimated by means of an approximation and must be confronted and validated through expert s opinion. The Impact Goat must be measurable, or at least must establish a type of nominal scale that allows comparing which of the options generate greater Impact and/or lower cost to merit its selection. The analysis of experts opinions must be based on certain evaluation criteria. The Efficiency criterion of an undertaking aims at verifying the manner in which resources should be combined to obtain the expected results. For this reason, efficiency corresponds to the allocation and 17

18 combination of the inputs to obtain products. Therefore, costs and products must be estimated for each undertaking option. Specifically, Total Annual Costs (TAC) must be estimated and the costs per Product Unit (CPU) for each undertaking, so that it is possible to determine the intervention that minimizes total costs and generates the greatest quantity of products. Costs can only be compared within the same unit of time (the year 0 is used for this purpose), because money has a different value over time. For this reason financial mathematics are used to update costs, discounted at a hurdle rate or cost of capital. The criteria for Effectiveness is the Cost per Unit of Impact (CUI). This reflects the cost incurred to generate one unit of Impact. The Indicator for this criterion is obtained by dividing the total cost per year of the undertaking option by the Impact estimated for that option multiplied by one hundred. Cost per Unit of Impact: CUI = TAC / OAI * 100 TAC = Total annual cost of the option OAI = Estimation of the Objective of Annual Impact for the undertaking option As it is possible to confirm, in the Ex ante evaluation: Cost-Impact implies favoring the option that should have greater impact at the lowest cost. This does not mean choosing the cheapest option, but the one which represents the best relation between cost and impacts that the undertaking option will generate. IV.4 Process and Product Indicators Process Indicators measure or investigate what occurred compared to what was planned (results of operations). The evaluation of processes is based on the monitoring of activities set forth in the strategic plan. Generally, data on the execution of activities must be compared with the budgeted cost in accordance with the pre-established timetable. Having a working plan that clearly indicates the actions foreseen and the persons responsible is assumed for this purpose. The Product Indicators show the quantity and quality of goods and services resulting from the execution of activities or operations (results of specific objectives). It is important to clarify that the evaluation of processes or monitoring does not allow a longterm evaluation of performance results or impact; it only provides information on whether activities were carried out and adherence to the budget. In process evaluations on the follow-up that SAIs do on audited organizations, mention can be made of instruments such as the Annual Tracked lmpact System (ATI): (Impact Tracked System: ITS), the application of which permits mid-period and final evaluations of the performance of audited public entities through timely information, analysis and comparisons, and the use of Indicators. These instruments allow focusing attention on the status of audited public entities compliance with findings and recommendations, providing necessary information and suppressing elements that are difficult to monitor annually, clarifying definitions and improving the database. In this way it would be possible to increase the number of indicators to respond to the interests of target publics (Congress, Government, Donors, organized civil society). In this case, with the ATI Evaluation (Annual Tracked Impact System: Impact Tracked System: ITS), the number of public entities assisted or audited is monitored, classifying them according 18