Development Discussion Papers

Development Discussion Papers Incorporating Provincial Opinion in the Evaluation of Local Government Fiscal Capacity in Indonesia Blane D. Lewis Development Discussion Paper No. 472 October 1993 Copyright 1993 Blane D. Lewis and President and Fellows of Harvard College Harvard Institute for International Development HARVARD UNIVERSITY

HIID Development Discussion Paper no. 472 Incorporating Provincial Opinion in the Evaluation of Local Government Fiscal Capacity in Indonesia Blane D. Lewis Abstract The government of Indonesia has recently adopted regulations which direct central authorities to release a significant amount of control over fiscal operations to the local level. The regulations note that such decentralization should be gradual and selective in nature and that it should be informed by sound evaluation of local government capability. This paper argues that any assessment of the fiscal capacity of local governments should incorporate the expert views of provincial level officials. In particular, the paper offers empirical evidence against the common contentions that provincial officials are unwilling and unable to assess the relative abilities of local governments and that, in any case, expert views are unnecessary in evaluating local authority performance. Blane D. Lewis is a Project Associate for the Urban Development Policy and Finance Project in Indonesia.

HIID Development Discussion Paper no. 472 Incorporating Provincial Opinion in the Evaluation of Local Government Fiscal Capacity in Indonesia Blane D. Lewis INTRODUCTION Recently the government of Indonesia has enacted legislation which directs central authorities to devolve a considerable amount of the fiscal responsibility for providing public services to the local 1 level. The new regulation notes that the decentralization of control over local service delivery should proceed in a gradual and selective manner, beginning with the most capable local governments first. In addition, the official decree admits of the need for a sound methodology which can be used to select those particular local governments that are most able to assume more responsibility in providing and financing public services. This paper argues that any methodology developed to rate the relative capabilities of local 2 governments should incorporate the informed views of provincial level officials. The main rationale for soliciting provincial judgments in the evaluation of local government capacity is straightforward: provincial administrators are charged with the oversight of many local government activities and are therefore uniquely well-positioned to offer insight into local authority capacity for assuming increased responsibilities. The paper proceeds as follows. First, a methodology for assessing provincial officials' views regarding the capabilities of local governments under their purview is briefly described and some preliminary results from a trial implementation of the evaluation technique are presented. Second, a numerical index of local government capacity to assume more responsibility over fiscal operations is constructed, based on information captured from the structured inquiry of provincial officials' opinions. Third, the index is employed to address several important criticisms of the socalled expert assessment. Finally, the paper concludes with a summary of the main points. THE EXPERT ASSESSMENT OF LOCAL GOVERNMENT CAPACITY As noted, an evaluation methodology based on the judgement of provincial experts has been designed for rating local government capacity. This expert assessment consists of structured interviews with provincial officials regarding the abilities of local governments within their respective provinces. Local governments are evaluated in four substantive areas: planning and programming, resource mobilization, general management and budgeting, and project design and 1 Republic of Indonesia, "Government Regulation Number 45 of 1992 Concerning the Implementation of Regional Autonomy with Emphasis on Second Level Regions". 2 Regional government in Indonesia is comprised of provincial (Tingkat I) and local (Tingkat II) governments. The former number 27 and the latter some 300. 1

implementation. The choice of substantive areas is not arbitrary. Each area corresponds to the general policy domain of a particular central government agency: The National Planning Agency (BAPPENAS), Ministry of Finance, Home Affairs, and Public Works, respectively. These agencies are those most directly responsible for guiding the decentralization process in Indonesia. 3 Provincial-level administrators are asked to rate local governments in their particular areas of expertise according to a series of predetermined questions. The so-called experts include representatives from local offices of central technical ministries (i.e. from Kanwil) as well as officials from various provincial government agencies (i.e. from Dinas) whose work closely corresponds to the substantive areas mentioned above. The questionnaire, implemented in face-toface meetings with the officials, focuses on a variety of aspects important to local government operations in Indonesia. The inquiry attempts to evaluate the skill with which local governments carry out their duties, for example vis-a-vis: collection and use of germane information, 4 cooperation with pertinent agencies, adoption of appropriate analytical methodologies, etc. Several experts are queried in each substantive area. Ratings by officials are based on ordinal-level scores: 3 for "good" performance by a local government on a particular criterion, 2 for "average" performance, and 1 for "poor" performance. Scores are summed for a particular local government across the various criteria within a substantive area and then averaged across all experts within the category. The result is an average total score for each local government in each of the four areas; the maximum possible score for a local government for any particular substantive category is 21 and the minimum is 7. 5 The evaluation tool has been implemented on a trial basis in three provinces: Nusa Tenggara Barat, North Sulawesi, and Jawa Timur. These particular provinces were selected, in part, to reflect the vast diversity in geographic condition and scale that exist in Indonesia. 6 Summary results from the expert assessment of local government capacity carried out in Jawa Timur are presented in Table 1. The table gives the average score for all local governments within each of the four substantive areas of inquiry, as well as the standard deviation and the number of experts questioned in each area. As can be seen from the table, experts in each of the four substantive areas rate local governments in Jawa Timur, in general, as slightly better than 3 Personnel from each of these agencies helped to develop the local government rating mechanism described here. 4 The precise criteria used to evaluate local governments are available from the author. 5 There are seven criteria within each of the substantive areas of inquiry. A score of 3 given by all experts in a substantive area for all criteria would result in an average total score of 21. Similarly, if all experts in a given substantive area were to give a particular local government a score of 1 then the local government's average total score would be 7. 6 Jawa Timur is the largest province in Indonesia, incorporating 37 local governments. Nusa Tenggara Barat and North Sulawesi possess six and seven local authorities, respectively. 2

7 "average". Local governments receive the highest average score in the area of planning and programming, followed by resource mobilization, management and budgeting, and finally project design and implementation. Table 1: Results of Expert Assessment in Jawa Timur QSPP QSRM QSMB QSPD Avg. 15.51 15.03 14.42 14.06 Std. 1.75 2.33 0.97 1.08 Dev. Number 8 7 7 8 Note: QSPP, QSRM, QSMB, and QSPD are the "qualitative scores" for local government ability in planning and programming, resource mobilization, management and budgeting, and project design and implementation, respectively. Next, information provided by the expert assessment described above is used to construct a simple index of local government capacity. AN EXPERT ASSESSMENT INDEX FOR RATING LOCAL GOVERNMENTS The construction of indexes raises several general theoretical and empirical concerns. These 8 considerations relate primarily to the choice of individual variables to be included in an index. Both conceptual and empirical questions in index construction are addressed below. The most important theoretical concern in index construction is logical validity. Logical validity concerns the power of a particular variable to operationalize the intended concept. Obviously, a close match between concept and operational variable is highly desirable. The degree of correspondence between concept and proxy is at least partly a matter of judgement, however, and in practice the issue is often subject to much debate. Construction of an index based on provincial officials' assessment of local government capacity poses no real conceptual problems regarding logical validity. Within each of the four substantive areas, information has been sought which attempts to guarantee the logical validity of the individual measures. That is, questions were devised in the first instance which indicate, at face value at least, the capacity of local governments in each of the four respective areas. If no conceptual problems exist with the construction of an index based on the expert assessment then attention can be turned to empirical matters. The salient empirical considerations 7 The "average" score would be 14. 8 See Earl R. Babbie (1979) The Practice of Social Research, Chapter 15, for a clear and comprehensive discussion of the issues involved in index construction. 3

in index construction concern the bivariate relationships among individual pairs of variables, the multivariate associations among all variables, and the bivariate relationships between individual variables included in the index and the index itself. 9 Individual pairs of variables included in an index should be associated with one another empirically. If two variables were both good proxies for the particular phenomenon in question, as suggested on grounds of logical validity, then it is reasonable to presume that the variables themselves would be empirically related to one another. At the same time, any two variables should not be empirically correlated to a very high degree. If two variables were very closely associated with one another then there would be no need to include both in the construction of the index. Similarly, the multivariate relationships among all variables should demonstrate that any individual variable is not explained to a very great extent by all others combined. That is, a multivariate analysis should show that each individual variable would contribute something to the index that is not contributed by the others taken together. Finally, each individual variable included in the index should be empirically associated with the index itself. That is, if the index has been well constructed it should be internally valid. The results of the expert assessment carried out in Jawa Timur provide data with which to employ the above empirical criteria in the construction of an index to rate local government capacity. Consider first the associations between pairs of expert assessment variables. Table 2 below shows the Pearson correlation coefficients (r s) among all such pairs. As can be seen from the table, the relationships are all reasonably strong, ranging from 0.494 to 0.657, and all associations are statistically significant. The results suggest that all qualitative variables are indeed measures of the same general phenomenon. The relationships are not "too strong", however, as the table shows. This implies that no single variable would usefully serve as a proxy for any other in an index. 9 In addition, the bivariate relationship between the constructed index and other independent or external indexes attempting to measure the same thing should be considered. No comparable independent indexes of local government capacity in Indonesia exist and so the issue of external validity is ignored here. 4

Table 2: Bivariate Relationships Among Expert Assessment Variables Variables QSPP QSRM QSMB QSPD QSPP 1.000 0.494** 0.668** 0.655** QSRM 0.494** 1.000 0.626* 0.657** QSMB 0.668** 0.626** 1.000 0.554** QSPD 0.655** 0.657** 0.554** 1.000 Number of cases: 37 1-tailed significance level: * = 0.01; ** = 0.001 Consider next the multivariate relationships among expert assessment variables as summarized in Table 3 below. As the table shows, the proportion of the variance of any individual variable that is accounted for by the others combined ranges from only 0.494 to 0.538. Apparently, each individual variable is not explained to a very great extent by the other variables taken together. This suggests that each variable would indeed contribute something extra to the measurement of local capacity and therefore should be included in an expert assessment index of local government capability. Table 3: Multivariate Relationships Among Expert Assessment Variable 2 Dependent Variance Explained by Other Variables (Adj R ) Variable QSPP 0.530 QSRM 0.494 QSMB 0.523 QSPD 0.538 The results of the bivariate and multivariate analyses above suggest that the four expert assessment variables might reasonably be combined to form a single index of local government capacity. All variables are measured in the same units and the combination is easily achieved by taking the simple unweighted sum of the individual scores. The resultant variable is named QSALL in the analysis that follows. Given the construction of the expert assessment index, QSALL, an analysis of its internal validity can be carried out. Table 4 shows the extent to which each component variable of the index is related to the index itself, as measured by the Pearson correlation coefficient. As can be seen from the table, the bivariate relationships between the index and the respective component 5

variables are all strong and statistically significant. The results demonstrate that the index possesses strong internal validity. Table 4: Internal Validity of Expert Assessment Index Variable QSPP 0.824** QSRM 0.874** QSMB 0.812** QSPD 0.832** Association with QSALL Number of cases: 37 1-tailed significance level: * = 0.01; ** = 0.001 Next, the index QSALL is employed to address several important criticisms of the attempt to incorporate provincial views in the assessment of local government capacity. EXPERT ASSESSMENT CRITICISM Critics of the expert assessment of local government capacity in the central government of Indonesia and elsewhere generally express three related concerns. First, it is often asserted that provincial administrators would be unwilling, in the first instance, to pass judgement on the capabilities of local governments. That is, it is maintained that cultural considerations constrain Indonesian officials from making relative evaluations about individual members of the collection of local governments. Second, it is frequently claimed that, even if provincial officials were willing, they are simply unable to correctly assess the capacity of local governments. In this regard, critics assert that provincial officials are either not skilled enough to make such judgements or that they are unduly influenced by inappropriate criteria (political, cultural etc.) or perhaps both. The above problems render the expert assessment invalid, it is argued. Third, and partly as a result of the second, it is sometimes contended that any local government evaluation methodology should necessarily be based to a greater extent on objective, quantifiable variables. At the limit, observers maintain that an analysis of quantitative indicators would, in fact, be sufficient to rate capacity of local authorities. This latter complaint, if justified, makes the expert assessment unnecessary. Each of these concerns is taken up, in turn, below. Are Provincial Officials Unwilling? In the event, it appears that provincial experts are quite willing to make relative evaluations of local government capacity, given assurances of confidentiality regarding their views and the appropriate authorization. In the present case, the latter consisted of an official request from the 6

provincial governor's office. In Jawa Timur, thirty out of thirty-three questionnaires were satisfactorily completed. It is not entirely clear as to why the remaining three questionnaires were not, in fact, concluded, but the relevant officials all had seemingly unobjectionable reasons for failing to carry out the request. In any case, a 90 percent response rate on the implementation of a survey questionnaire would normally be considered quite acceptable. Are Provincial Officials Unable? As demonstrated above, the various expert assessments of local government capacity are all rather strongly associated with one another. The fact that the assorted expert judgements are so closely related suggests that they measure the same general phenomenon. Consistent opinions are not necessarily valid ones, however and it is possible, of course, that the expert evaluations measure a phenomenon different from local government capacity. Consistent, but consistently wrong, as it were. One way to test the assumption that the provincial official opinions evaluate capacity and not something else would be to compare the results of the expert assessment with logically valid quantitative measurements of local government capacity. Such an analysis is carried out next. Quantitative criteria are considered in each of the four substantive areas noted above: planning and programming, resource mobilization, general management and budgeting, and project design and implementation. A brief discussion regarding the selection of the quantitative variables in each category follows. 10 Planning and Programming: Effective local governments should use resources at their disposal to make investments in local infrastructure which provide services demanded by the local population. Admittedly, this abstraction is rather difficult to capture in a single quantitative measure. Nevertheless, in this paper, the adequacy of local government development expenditures is taken as a first approximation for ability in infrastructure investment. More specifically, the variable chosen is the (natural logarithm of) development expenditures per capita (LDEPOP). It is expected that local government capacity would vary directly with LDEPOP. Resource Mobilization: Competent local governments should be able to effectively raise resources to share in the cost of providing services to their populations. In this regard, local fiscal effort, own-source revenue adequacy, and self-reliance in meeting current budgets, inter alia, are all important features of effectiveness in mobilizing resources. Here, the (natural logarithm of) 10 These are the actual variables that were chosen by central government officials as their preferred indicators of local government capacity. Data to estimate the variables described below come from the Lembaga Administrasi Negara data base on local governments. The estimates are average values over the five year period 1985/86 to 1989/90. Figures are in nominal terms; price indices at the local government level do not exist. 7

11 the ratio of own-source revenues to gross regional development product (LOSGRDP), a standard proxy for fiscal effort, is chosen to represent local government capability in mobilizing resources. Local government capacity should be directly associated with LOSGRDP. General Management and Budgeting: Effective local governments should be able to accurately budget and manage the resources at their disposal. Local governments have most control over PAD revenues and so it is reasonable to assume that competent local governments should be able to accurately budget the level of those particular resources to which they will have access. The difference between realized and planned PAD revenues is therefore taken as a proxy for ability in budgeting resources. More precisely, the chosen variable is the (natural logarithm of the) squared deviation from one of the ratio of realized PAD revenues to planned PAD revenues (LRPDPPD). It is expected that local government capacity would vary inversely with LRPDPPD. Project Design and Implementation: Finally, capable local governments should be able to design and implement local development projects that meet appropriate technical standards. Currently, local governments in Indonesia are most responsible for water supply, solid waste collection, and the construction of local roads. Because data on the water supply and solid waste collection are limited, adequacy of road infrastructure is selected as a proxy for capability in designing and implementing projects. In particular, (the natural logarithm) of the number of kilometers of local roads divided by gross regional domestic product (LRDGRDP) is selected as the relevant variable. It is assumed that local government capacity should be positively associated with LRDGRDP. Table 5 below shows the association between the expert assessment index and each of the four quantitative variables enumerated above. As can be seen from the table, the index is significantly related to each of the four variables. One of the variables, the proxy for planning and programming shows quite a strong and statistically significant relationship with the constructed index. The proxies for own-source revenue mobilization, management and budgeting, and project design and implementation are slightly less strongly associated with the index, although the relationships are still statistically significant ones. 11 Own-source revenue includes all tax and charge revenue under the direct control of local government (PAD) as well as the local authority's share of property tax revenues (PBB), as per the standard International Monetary Fund (IMF) definition. 8

Table 5: Association Between Expert Assessment Index and Selected Quantitative Variables Quantitative Variable Association with QSALL LDEPOP 0.522** LOSGRDP 0.358* LRPDPPD -0.341* LRDGRDP 0.309* Number of cases: 37 1-tailed significance level: * = 0.01; ** = 0.001 The existence of statistically significant correlations between the constructed index and the various quantitative proxies above would seem to suggest that the expert assessment offers quite a good measure of local government capacity. In other words, contrary to the views of many, local officials are indeed capable of evaluating the capacity of local governments. Consistent and consistently right, one might say. 12 Is Provincial Input Unnecessary? The third criticism of the expert assessment of local governments is that quantitative variables are necessary and sufficient in any rating of local governments. This paper makes no argument against incorporating quantitative criteria into an evaluation of local government capacity. Provided reliable data exist on valid indicators of government capacity, such information should, of course, be included in assessing local authorities. The quarrel here is with the notion that such quantitative criteria are sufficient. Put simply: standard quantitative indicators cannot substitute for local expert knowledge. One way to test the sufficiency of quantitative variables in rating local governments would be to determine the extent to which the quantitative proxies account for variation in the qualitative index of capacity. Were a good deal of the variation in the expert assessment index to be explained 12 The validity of such a statement would be weakened if the quantitative variables themselves were judged to be unfit as indicators of local government capacity, of course. Indeed, it is not difficult to find at least some fault with each of the variables regarding its ability to serve as a measure of capacity. On the other hand, each of the quantitative proxies were found to be significantly associated with the others. (The associations ranged in value from 0.308 to 0.600.) The correlation among variables supports the claim that such variables are indicators of a common concept. The argument here is that the concept is local government capacity. 9

by the quantitative variables then perhaps the latter might suffice in evaluating local government capacity. If, on the other hand, the explanatory power of the quantitative proxies were found to be weak then the sufficiency of the "objective" indicators variables would be difficult to argue, at least on empirical grounds. This above test is easily conducted by regressing the expert assessment index, QSALL, against the set of quantitative indicators (in natural logarithm form) specified above. Ordinary least 2 squares estimation results in an adjusted R of 0.256. The outcome indicates that just over onequarter of the variance in the expert assessment index is accounted for by the four quantitative variables taken together. Clearly then, these quantitative proxies could not be expected to adequately substitute for provincial expert input in an evaluation of local government performance. That is, the results argue against the sufficiency of quantitative variables in measuring local government capacity. 13 SUMMARY AND CONCLUSIONS: THE NEED FOR LOCAL INPUT IN EVALUATION This paper began by noting the need for a sensible methodology for determining those local governments that are most capable of assuming more responsibility in providing and financing public services. As described above, an evaluation instrument has been developed which solicits the views of experts at the provincial level regarding the relative abilities of local authorities. The evaluation methodology has been tested in three different provinces and results from the trial implementation in Jawa Timur have been used to test several criticisms of the expert assessment mechanism. The results from the research indicate that provincial administrators are not, as some would claim, reluctant to express their views about the relative abilities of local governments, given approval for doing so by their superiors. In addition, the exercise demonstrates that the judgments of provincial officials regarding the relative abilities of local governments are remarkably consistent with one another. Experts in the areas of planning and programming, resource mobilization, management and budgeting, and project design and implementation exhibit an exceptional congruence of opinion regarding the relative capabilities of local governments. Moreover, it was shown that the expert opinions are quite compatible with evaluations based on so-called objective, quantitative measurements of local government capacity. Expert 13 One might reasonably expect to explain a greater percentage of the variance in QSALL by adding other quantitative proxies of local government capacity to the right-hand-side of the equation. Experimentation along these lines was attempted and, in 2 fact, higher adjusted R s were occasionally obtained. In no case, however, did the set of quantitative variables explain more than 50 percent of the variance in QSALL. Different functional forms were also tried in an attempt to increase explanatory power of the various regressions. These experiments did not produce appreciably different results from those obtained using the original specification. The conclusion that quantitative proxies should not be used as a substitute for the expert assessment of local government capacity seems a rather robust one. 10

views are not, however, explained to a very great extent by quantitative proxies. These latter two facts strongly suggest that, contrary to the beliefs of some, provincial experts are indeed capable of evaluating local government capacity and furthermore that local expert knowledge cannot be substituted for by standard quantitative variables. In sum, the incorporation of provincial input into the evaluation of local government capacity is both feasible and indispensable. 11

REFERENCES Babbie, Earl R. (1979), The Practice of Social Research. Wadsworth Publishing Company, Belmont, CA. Republic of Indonesia (1992), "Government Regulation Number 45 of 1992 Concerning the Implementation of Regional Autonomy with Emphasis on Second Level Regions," Jakarta. 12