Comments on Your Government

Transcription

1 Comments on Your Government A SPECIAL PUBLICATION OF THE RHODE ISLAND PUBLIC EXPENDITURE COUNCIL Performance Management in State Government This RIPEC report discusses performance management as a general method for assessing government effectiveness, particularly as it relates to the regulation of banking; explores bankingrelated performance management practices in northeastern U.S. states; and applies a performance management framework to the Rhode Island Department of Business Regulations (DBR) through a RIPEC-developed Banking Regulation Index. The specifics of this DBR Index and implications for its use are also discussed. Introduction As the demand for global resources increases, governments are under increased pressure to improve the efficiency and effectiveness of public services. Performance management in government is becoming a widely recognized method for measuring outputs and outcomes of government initiatives and programs and for ensuring that government expenditures lead to the desired results. Performance management in government is distinctly more challenging than in the private sector. Outputs and efficiencies are generally considered to be sufficient to gain an understanding of the performance of actors in the private sector. Their impact to society can be determined by individual consumption decisions, which can be aggregated and expressed in terms of revenue and profit. However, these measures are not applicable in the public sector, because Figure 1 Performance Management Framework environment final outcomes 1 relevance needs Agency intermediate outcomes 5 effectiveness objectives inputs activities outputs 2 utility and sustainability 3 efficiency 4 cost effectiveness SOURCE: Van Dooren, Bouckaert & Halligan, 2010 May 2014

2 outcomes of public services aim to have a collective and often intangible impact on society. They are thus not taken into account and assigned an economic value in individual consumption decisions. A government agency s performance can be measured either by its outputs or by its outcomes. Outputs consist of the immediate product of an agency s actions, such as cases completed or applications processed. Outcomes are more difficult to quantify, as they consist of the changes brought by the agency s actions. This report will describe a framework for public sector performance management and then the process by which an index for outcome measurement was developed for the Banking Division of the Rhode Island Department of Business Regulation (DBR). Performance Management Framework A performance management framework adapted from Performance Management in the Public Sector by Van Dooren, Bouckaert, and Halligan is provided in Figure 1. According to this framework, needs of society, as defined by the political process, lead to the formulation of objectives for an organization or a program. The relation between needs and objectives is referred to as relevance (number 1 in Figure 1) whether the organization is defining its objectives in line with its needs. These objectives inform the inputs, which are used to perform activities. These activities, in turn, result in outputs. These outputs are used to determine efficiency (3), which is defined as a ratio of outputs to inputs. Effectiveness (5) goes a step further up the causal chain and takes into account the outcome that results from the output. It is defined as a ratio of output to outcome. Cost effectiveness (4) is defined as a ratio of input to outcomes. Finally, the comparison of needs and outcomes enables assessment of the utility and sustainability (2) of the organization or program being evaluated. Since DBR commissioned this performance management study and the development of a performance measure for DBR s Banking Type of Measure Output measures Outcome measures Measure CT ME MD MA NH NJ VT VA Total # of examinations x - x - - x # or list of enforcement activities x - x x - x - x 5 # of licensees/applications x x x - x - - x 5 Community Reinvestment Act ratings x - - x # of complaints x x x Aggregate industry metrics (e.g. total deposits, total premiums) x x - - x x - x 5 Aggregate industry key performance indicators x x Efficiency indicator Timeliness - - x - - x - x 3 SOURCE: Various state agency websites Table 1 Publically Reported Performance Measures by Bank Regulators by State (2013) 2

3 Division, this study applies government performance management to state bank regulators. An analysis of performance reporting practices by states in New England and along the eastern seaboard revealed that agencies currently report output and outcome measures. Three of the examined states reported timeliness measures, which could be interpreted as efficiency indicators. None of the states reported effectiveness indicators. The results of this analysis are summarized in Table 1. Application of the Performance Framework to DBR DBR s mission is to assist, educate, and protect the public through the implementation and enforcement of state laws mandating regulation and licensing of designated businesses, professions, occupations, and other specific activities while recognizing the need to foster a sound business environment. More specifically, the Banking Division provides regulatory oversight of state-chartered banks, credit unions and licensees. The agency uses inputs such as personnel and other resources to conduct activities such as examinations and reviews to determine compliance with state banking laws, financial solvency and safety and soundness of operations. The State of Rhode Island publishes performance measures online and as part of the annual budget process. In this context, the Banking Division so far uses output measures, such as the average amount of days it takes to issue a license, the average amount of days it takes to conduct an examination or the amount of time it takes to resolve a complaint. Its current practices are thus very much in line with other states reporting practices. These measures are known as efficiency indicators (3) as they are determined essentially by a ratio of outputs (e.g. a license) to inputs (e.g. days). In the case of the Banking Division, the desired outcome is a safe, healthy and solvent banking sector in the state. The fact that the financial services sector already uses a range of quantifiable indices and performance indicators facilitates the task of defining effectiveness indicators for the Banking Division. Effectiveness measures are challenging to develop because most agencies strive to address complex issues in society. Thus, most outcome measures can be indicators of performance at best. Frequently, decision makers must consider a tradeoff between the rigor of scientific analysis and the practicality of data collection and analysis. The Banking Regulation Index, developed by RIPEC, goes one step further than the aforementioned framework by aggregating various effectiveness indicators into one index. Process of Establishing Measures The formulation of outcome measures starts with the mission statement and the objectives of the agency, or organization. Outcome measures should attempt to measure the extent to which the agency is reaching its objectives (though this tenet assumes that objectives are formulated in terms of outcomes, not outputs). When using objectives or goals of an agency for performance measurement, one must consider two major factors: (1) are the objectives measurable both in theory and practice and (2) are these objectives within the reasonable realm of influence of the agency? The first of these factors addresses measurability. Though most questions, including subjective ones, can be answered quantitatively, through various methods of data collection including surveys, stakeholders should determine whether it makes logical sense to operationalize a subjective issue in terms of numbers. If it is deemed sensible and necessary, the second consideration is whether a performance measure is practically feasible. Conducting surveys and collecting data as forms of measurement may be 3

4 cumbersome and expensive, so the benefit derived from having certainty over the performance data should be weighed with the various costs of collecting and analyzing that data. The second factor considers the influence of external factors on outcomes. As Figure 1 acknowledges, the external environment may have a significant influence on outcomes, especially final ones. It is thus important to either choose outcome measures that are within the locus of control of the agency or attempt to control for the influence of the environment to the extent possible. The Banking Regulation Index as an Outcome Measure The Components of the Banking Regulation Index The Banking Regulation Index was developed on the previously described basis. The three pillars of the Index are: Health of the industry (the desired outcome, subject to multiple influencing factors); Proficiency of DBR (intrinsic ability of the regulating entity to conduct its business); and Status of the environment (controlling for external factors likely to influence the outcome). The seven components of the index are listed here and described in more detail later: 1. A weighted average of the rating score assigned to RI banks and credit unions (CAMELS rating) that assesses the institutions capital adequacy, assets, management capability, earnings, liquidity and sensitivity to market risk; 2. The weighted average score assigned to RI banks and credit unions by external credit rating agencies; 3. The overall economic growth in the state as compared to the past ten years performance; 4. The accreditation scores assigned to the DBR by the national associations for bank regulators and credit union regulators; 5. Substantive enforcement actions and repeated enforcement actions against banks and credit unions; 6. The complaints brought forward against banks and credit unions, expressed in a deposits to complaints ratio; and 7. The timeliness of CAMELS ratings when institutions are in particularly poor condition. 1. CAMELS Rating The CAMELS rating system is used by regulators of banks and credit unions to classify an institution s overall condition based on the six components of the rating listed above. This rating is based on a scale of 1 to 5, with 1 indicating strong performance and risk management practices and 5 indicating significant deficiencies with high probability of failure. It includes an adjustment for the regional context, where a strong average CAMELS rating in the state compared to the average of other states in the region leads to a higher score. This weighted rating thus directly reflects the health of the regulated industry, with a small adjustment accounting for the health relative to other states in the region. This component is clearly influenced very strongly by external factors, such as the institution s management or the overall economic condition. This is the reason for the regional context adjustor, which is supposed to control for external effects that apply to the region as a whole. This is weighted as the second strongest component, making up 18 percent of the overall index, because conducting these ratings is a key output of the agency. At the same time they are influenced strongly by external factors. 4

5 2. External Rating Agencies The ratings assigned by external rating agencies are similar in their meaning to the CAMELS ratings, but are less rigorous in their underlying analysis. However, the key added value of these ratings is that they are conducted by an independent third party. However, since this component provides little additional information, it receives a relatively weak overall weight and makes up 9 percent of the Banking Regulation Index. 3. Economic Indicator Adjustor Since economic growth is likely to be a major factor that influences the performance of the regulated industry, this component is intended to control for this effect that is beyond the control of the DBR. It is based on the current quarter s annualized GDP growth as compared to the annual GDP growth over the previous ten years. Below average economic growth will positively influence the index because this weak growth is likely to adversely affect the regulated industry. As an adjustor, this component is not formally weighted. However, since the component can add another 10 percentage points to the overall score, the rest of the index is weighted with only 90 percent. This way a perfect score across all aspects of the Index will result in a score of Accreditation Scores The most important component of the Banking Regulation Index consists of the Conference on State Bank Supervisors (CSBS) and National Association of State Credit Union Supervisors (NASCUS) accreditation scores. There is a comprehensive set of criteria that must be fulfilled for a state bank regulator to receive the CSBS or NASCUS accreditation. These categories include: Department administration and finances, Personnel, Training, Examination policies and procedures, Supervision, and Legislative environment. This component is the centerpiece of the Banking Regulation Index with a weight of 45 percent. It is the component that most comprehensively measures factors that are internal to the agency and thus almost entirely in the locus of control of the agency itself (with the exception of the legislative environment, among others). In addition, it is based on a self-assessment by the agency that is audited by an independent third party. 5. Enforcement Actions Enforcement actions are included in this index because they are an indicator for the performance of the industry and the effectiveness of the agency. Fewer substantive enforcement actions indicate that the regulated industry is more in compliance with the law and fewer repeat infractions indicate that the agency s enforcement actions are being acted upon by the regulated industry. A drawback of this measurement methodology is that it may create an incentive for inaction and thus poor regulation on the part of the agency. However, negligence on enforcement actions may have adverse effects on other components of the index, thus offsetting this potentially negative incentive. The enforcement actions component makes up 9 percent of the total index. 6. Complaints Consumer complaints against banks and credit unions may be an indicator for poor performance of the regulated industry and thus the regulator. This component measures the performance of the industry as a proxy for agency performance, recognizing that there are many external factors other than the quality of the regulator that influence number of complaints. One of the most significant drivers of consumer complaints would likely be the number of customers. This is controlled for by using the deposit to complaints ratio, which uses total deposits as an easy, publicly available proxy for the number of customers. 5

6 7. Timeliness Timeliness of examinations is a significant factor influencing accreditation scores. Thus, it is treated separately here only to a very limited extent. When an institution receives a CAMELS rating of a 3, 4 or 5 and is thus deemed to be troubled, the rating must be reviewed within a year. This aspect of timeliness is treated by this component of the index, which pulls down the score if the DBR is behind schedule on these critical examinations. Challenges in Creating a Numerical Index Creating an index expressed in a single number has the advantage of being easily comprehensible. However, this comes at the cost of reducing a highly complex, multidimensional issue into a single number and thus losing important contextual information that other performance measures might provide. Maintaining the individual components of the index counteract this issue by providing some context and explanation on how the index comes to be. One mathematical challenge to creating a single numerical index is posed by the way an average is calculated. The Banking Regulation Index is basically a weighted average of its individual components. Since the weight of the component, not the absolute value of the number (e.g. number of complaints vs. CAMELS rating), is supposed to determine the mathematical impact of the component on the index, the components have to be normalized. In this case, all weighted components of the index were adjusted to a common scale between 0 and 1. This normalization is achieved by calculating percentages (as in the case of the CAMELS scores) or by applying a logistics function to the distribution, as detailed in the appendix. This index consists of two types of components: weighted components and unweighted components or adjustors. The weighted components are multiplied by a weight, which is previously defined depending on the relative importance of that factor. The unweighted components are not formally assigned a weight because they are not normalized. They are calculated such that they adjust the index by a certain amount and are thus implicitly assigned a weight or mathematical impact on the index as a whole. The Index is designed such that the highest possible score is 100. Uses of the Index for DBR and RI State Government The Banking Regulation Index, like any metric, provides limited intrinsic information. Metrics like this develop meaning only in context. Such context can be provided by a historical development over time, by comparing the number to other entities (benchmarking) or by setting a goal or target value. Since this index is not used anywhere else, it does not lend itself to benchmarking. Since it is new, there is no historical data to compare it against and it would thus also be too early to set a target. Thus, this index will have to develop meaning over time as more contextual information becomes available. Such context can be provided by identifying and isolating trends over time. General trends can be broken down into the components of the Banking Regulation Index to provide additional detail. This can help pinpoint specific issues within the Department and help direct resources or policy to proactively remedy the issue. Finally, the Index can be used to underpin budget, regulatory and procedural policy with concrete, quantifiable data. Initially, the Department plans to calculate and use the Index in a pilot phase for internal use only. It then intends to vet the Index with the regulated industry and considers publishing it by During the pilot phase, the Index will be calculated on a quarterly basis. Upon successful implementation, the Department plans to use it as a template for the Insurance and Securities Divisions. 6

7 Appendix: Summary Table of Components of the Banking Regulation Index Explanation Rationale Mathematical range Likely range Weight Total Index CAMELS index Medium (0.18) 1.1 CAMELS RI Medium 1.2 CAMELS Regional Context Adjustor 2. External Rating Agencies 3. Economic Indicator Adjustor 4. Accreditation Score 5. Enforcement actions 5.1 Substantive enforcement actions 5.2 Repeat infractions One minus weighted average CAMELS rating as percentage of highest possible score (5) Weighted average RI CAMELS rating as percentage of regional weighted average normalized score onto a scale of using a logistic function Weighted average rating as percentage of highest possible rating Standardizes GDP growth rate based on 10-year history and then normalizes this score onto a scale of using a logistic function. Attained score as percentage of highest possible score for NASCUS and CSBS, weighted by deposits in credit unions and banks Standardizes enforcement actions based on 10-year history and then normalizes this score onto a scale of 0 1 using a logistic function. Standardizes enforcement actions based on 10-year history and then normalizes this score onto a scale of 0 1 using a logistic function. Higher rating scores lead to a higher index score. The RI CAMELS rating being higher than the regional average should raise the indicator score Higher rating scores lead to a higher index score. Slow economic growth leads to lower performance of the regulated entities. This lower rating should be offset by this adjustment. Higher accreditation score leads to higher index score. Fewer actions (relative to 10-year average) lead to higher score. Fewer actions (relative to 10-year average) lead to higher score None Low (0.09) N/A High (0.45) Low (0.1) Low (0.04) Low (0.06) May 2014

8 6. Deposits to Complaints Ratio 7. Efficiency/ Timeliness Standardizes deposits to complaints ratio (total deposits divided by sum of all valid complaints) based on 10-year history and then normalizes this score onto a scale of 0 1 using a logistic function. Every institution with CAMELS rating 3 or lower not rated in last 13 months reduces overall index by 5 percent. Higher ratio (fewer complaints) leads to higher score The Banking Regulation Index should be pulled down when DBR is behind schedule on rating troubled institutions. When this is not the case, the index should not be affected Low (0.09) N/A N/A 8