Organisation de Coopération et de Développement Économiques Organisation for Economic Co-operation and Development

Transcription

1 Unclassified GOV/RPC/MRP(2016)2/REV1 GOV/RPC/MRP(2016)2/REV1 Unclassified Organisation de Coopération et de Développement Économiques Organisation for Economic Co-operation and Development 13-Jun-2016 English - Or. English PUBLIC GOVERNANCE AND TERRITORIAL DEVELOPMENT DIRECTORATE REGULATORY POLICY COMMITTEE Building regulatory policy systems in OECD countries Draft analytical paper 8th OECD Conference on Measuring Regulatory Performance June 2016 Sydney, Australia This paper provides a revised version of [GOV/RPC/MRP(2016)2], taking into account the comments received from delegates. The original paper was presented at the meeting of the Steering Group on Measuring Regulatory Performance on 12 April 2016 and circulated for comments to all RPC delegates in April Pending comments from RPC delegates by 11 July 2016, the Secretariat will publish the paper in the series of OECD Working Papers on Regulatory Policy. Christiane Arndt (Christiane.Arndt@oecd.org), Céline Kauffmann (Celine.Kauffmann@oecd.org), Sarah Hermanutz (Sarah.Hermanutz@oecd.org), Rebecca Schultz (Rebecca.Schultz@oecd.org) English - Or. English JT Complete document available on OLIS in its original format This document and any map included herein are without prejudice to the status of or sovereignty over any territory, to the delimitation of international frontiers and boundaries and to the name of any territory, city or area.

2 I. INTRODUCTION 1. Delegates to the OECD Regulatory Policy Committee (RPC) have consistently made the case that countries require better information about where investments in programmes to improve regulations should be focused to pay growth and welfare dividends. This information is necessary to target scarce resources for reform efforts, to communicate progress and to generate the political support needed for implementing regulatory policy reforms. 2. The 2012 Recommendation of the Council on Regulatory Policy and Reform (Recommendation) provided the normative framework and some strategic orientations to countries on the key pillars of regulatory policy. The Framework for Regulatory Policy Evaluation (OECD, 2014) was a first step to provide an analytical framework to assist countries in evaluating their regulatory policy programmes. The development of Indicators of Regulatory Policy and Governance (ireg, see Box 1) provided a means to countries to situate their efforts against three of the 12 principles set out in the 2012 Recommendation. Results and first analysis of the indicators have been published in the Regulatory Policy Outlook (OECD, 2015). Box Indicators of Regulatory Policy and Governance (ireg) The 2015 Indicators of Regulatory Policy and Governance (ireg) present up-to-date evidence of OECD member countries and the European Commission s regulatory policy and governance practices advocated in the 2012 Recommendation of the Council on Regulatory Policy and Governance. They cover in detail three principles of the 2012 Recommendation: stakeholder engagement, Regulatory Impact Assessment (RIA) and ex post evaluation, and provide a baseline measurement to track countries progress over time and identify areas for reform. The Indicators present information for all 34 OECD member countries and the European Commission as of 31 December The 2015 Indicators draw upon responses to the 2014 Regulatory Indicators survey. Answers were provided by delegates to RPC and central government officials. Compared to previous surveys, the 2014 survey puts a stronger focus on evidence and examples to support country responses, as well as on insights into how different countries approach similar regulatory policy requirements. The survey questionnaire has been developed in close co-operation with RPC delegates and members of the OECD Steering Group on Measuring Regulatory Performance. Survey answers underwent a verification process carried out by the OECD Secretariat in co-operation with delegates to the RPC in order to enhance data quality and ensure comparability of answers across countries and over time. The survey focuses on RIA and stakeholder engagement processes for developing regulations (both primary laws and subordinate regulations) that are carried out by the executive branch of the national government and that apply to all policy areas. Questions regarding ex post evaluation cover all national regulations regardless of whether they were initiated by parliament or the executive. Based on available information, most national regulations are covered by survey answers, with some variation across countries. Most countries in the sample have parliamentary systems. The majority of their national primary laws therefore largely originate from initiatives of the executive. This is not the case, however, for the United States where no primary laws are initiated by the executive, or, to a lesser extent, for Mexico and Korea where the share of primary laws initiated by the executive is low compared to other OECD member countries (4% over the period and 30% in 2013 in Mexico and 16% in Korea over the period ). Based on the information collected through the 2014 survey, the OECD has constructed three composite indicators on RIA, stakeholder engagement for developing regulations, and ex post evaluation of regulations in order to help present the information collected in an easily expressible format. Each composite indicator is composed of four equally weighted categories: systematic adoption, methodology, transparency, and oversight and quality control. 2

3 While composite indicators are useful in their ability to integrate large amounts of information into an easily understood format (Freudenberg, 2003), they cannot be context specific and cannot fully capture the complex realities of the quality, use and impact of regulatory policy. In-depth OECD country peer reviews are therefore required to complement the indicators and provide readers with an in-depth assessment of the quality of a country s regulatory policy, taking into account the specific governance structures, administrative cultures and institutional and constitutional settings to provide context-specific recommendations. Moreover, the results of the ireg indicators, as those of all composite indicators, are sensitive to methodological choices, unless country answers are homogeneous across all practices. It is therefore not advisable to make statements about the relative performance of countries with similar scores. Please note that while the implementation of the measures assessed by the indicators aim to deliver better regulations, the indicators should not be interpreted as a measurement of the quality of regulation itself. All underlying data and scores for the composite indicators are available at Further information on the indicator design and methodology, as well as the full list of survey questions covered by the indicators can be found in: Arndt, C., A. Custance Baker, T. Querbach and R. Schultz (2015), 2015 Indicators of Regulatory Policy and Governance: Design, Methodology and Key Results, OECD Regulatory Policy Working Papers, No. 1, OECD Publishing, Paris. Results and first analysis of the indicators are available in the OECD Regulatory Policy Outlook Source: 3. The rich dataset provided by ireg is now publicly available, but has yet to be fully exploited. In particular, the ireg provides detailed insights into the practices of countries in carrying out Regulatory Impact Assessments (RIA), stakeholder engagement when developing regulations and ex post evaluation. This detailed knowledge provides new insights into the way regulatory policy is carried out by countries, notably whether practices in the three areas tend to be developed simultaneously or not and what priorities countries put on the respective areas of methodology, systematic adoption, transparency and oversight. This paper builds on this detailed knowledge to stimulate discussion on reform priorities and to help countries plan their next steps in improving their regulatory policy system. 4. While OECD countries aim to build a solid regulatory policy system comprising all the elements of the OECD Recommendation, no country has yet fully implemented the OECD Recommendation. Countries are at different stages of implementing the Recommendation and, under constraints of resources and absorption, need to make decisions on which practices to adopt first and which practices might be introduced at a later stage. These choices are not without risks implementing some practices of the Recommendation without other essential complementary elements may limit the effectiveness of the practices. This paper, in the first section, analyses the choices made by countries with respect to the uptake of practices of RIA, stakeholder engagement in the development of regulations and ex post evaluation of regulations to explore whether these are seen as complementary in OECD countries or whether countries focus on one tool in particular. It identifies ings of countries around the emphasis they put on each of these tools. To stimulate discussion on how to stage the improvement of each of these tools, the second section analyses for each of these tools what policy choices distinguish relative ings of countries (advanced, middle and starting reforms). 3

4 Stakeholder engagement Ex post evaluation Ex post evaluation GOV/RPC/MRP(2016)2/REV1 II. INVESTMENT IN KEY REGULATORY POLICY TOOLS 5. The ireg show countries progress in implementing the Recommendation in three key areas: ex ante analysis of the impact of planned regulations through regulatory impact assessment, ex post analysis of existing regulations and the engagement of stakeholders in the design of regulations. While the Regulatory Policy Outlook provides a descriptive analysis of the results against each of the three composite indicators, much more can be inferred from a combined analysis of these policy tools. 6. The combined analysis of these three composite indicators shows that countries that invested in RIA also invested in stakeholder engagement systems (see Figure 1A). There is also a clear link between investments in ex ante and ex post evaluation systems (see Figure 1B). In fact, the correlation coefficient between RIA and ex post evaluation is as high as the correlation coefficient between RIA and stakeholder engagement (see Table 1). However, the nature of the relationship differs: while scores on RIA and stakeholder engagement increase proportionally (see Figure 1A), the relationship between RIA and ex post evaluation is non-linear (see Figure 1B). The correlation between RIA and ex post evaluation is weak for countries that score below average on the RIA indicator. Only for countries above the OECD average on the composite indicator for RIA, the scores for RIA and ex post evaluation correlate strongly. Figure 1. Two-dimensional graphs of policy tools 4 A 4 B 4 C 3 y = 0.6x y = 0.4x 2-0.9x y = 0.5x Regulatory impact assessment Regulatory impact assessment Stakeholder engagement Note: Scores represent the average of primary laws and subordinate regulations, except for the scores of Regulatory Impact Assessment and Stakeholder Engagement for the United States, for which only the results for subordinate regulations are used. Scores are based on composite indicators constructed on the basis of qualitative data. For details on the construction of the composite indicators, see Box 1. In the scatterplots, each dot represents a country and its position on the graph is determined by its scores on the two respective policy tools. A line of best fit was added that best represents the trend of the data. Please note that correlation does not imply causation. Source: OECD, 2015 Indicators of Regulatory Policy and Governance. Online available at: 7. The lowest correlation can be found between the scores on ex post evaluation and stakeholder engagement, i.e. a number of OECD countries have strong stakeholder engagement systems but only rudimentary ex post evaluation of regulations in place (see Figure 1C and Table 1). This is not true for the reverse; no countries that are strong on ex post evaluation are in the starting for stakeholder engagement. To some extent, this result reflects the fact that stakeholder engagement is a widespread 4

5 discipline of regulatory policy. By contrast, ex post evaluation is a practice that only a limited number of OECD countries apply systematically. Table 1. Correlation results between composite indicators Note: The underlying scores represent the average of primary laws and subordinate regulations, except for the scores of Regulatory Impact Assessment and Stakeholder Engagement for the United States, for which only the results for subordinate regulations are used. Scores are based on composite indicators constructed on the basis of qualitative data. For details on the construction of the composite indicators, see Box 1. A correlation coefficient is a measure of the degree to which two variables tend to move together. The coefficient has a value between plus and minus 1, which indicates the strength and direction of an association. Please note that correlation does not imply causation. Source: OECD, 2015 Indicators of Regulatory Policy and Governance. Online available at: 8. The analysis of country ings according to their results on the composite indicators on RIA, ex post evaluation and stakeholder engagement confirms the strong correlation between the three tools of regulatory policy (See Figure 2. For detailed information on the country ing, see Box 2). Countries with strong RIA practices also tend to have invested in open stakeholder engagement systems and in ex post evaluation. No countries that are in the advanced on RIA are in the starting for stakeholder engagement or ex post evaluation. 9. Overall, two relatively homogeneous clusters of countries emerge. A first cluster of countries res 10 countries with advanced RIA practices. These countries also have relatively advanced stakeholder engagement and ex post evaluation practices. A second cluster res 11 countries with starting RIA practices. These countries also feature practices for stakeholder engagement and ex post evaluation that are either at the starting or middle level. A third of countries gathers the remaining 14 OECD jurisdictions. This displays middle level practices in relation to RIA and a very heterogeneous performance with respect to stakeholder engagement and ex post evaluation. 10. Most countries with a common law system (Australia, Canada, New Zealand, the United Kingdom and the United States 1 ) are in the first cluster, with a strong emphasis on regulatory impact assessment and stakeholder engagement. Three countries from this are in the advanced for all three regulatory policy tools. However, there are also countries with a civil law system (Estonia, the Czech Republic, Germany and Mexico) and the European Union in the first cluster, having advanced RIA practices and relatively advanced stakeholder engagement and ex post evaluation practices in place. Nordic countries (Denmark, Finland, Iceland, Norway and Sweden) also seem to have similar systems in place. They are mostly in the middle for stakeholder engagement and ex post evaluation, and in the starting for RIA with the exception of Sweden. Central and Eastern European countries (Czech Republic, Estonia, Hungary, Poland, Slovak Republic, Slovenia) are mostly in the middle for RIA and in the advanced or middle for stakeholder engagement with the exception of Hungary where engaging with stakeholders is not a policy priority. 1 Further OECD countries with a common law system are Ireland and Israel. 5

6 11. Countries that were particularly affected by the financial and economic crisis (Greece, Iceland, Ireland, Italy, Japan, Portugal and Spain) seem to put less emphasis on regulatory policy and governance than other OECD countries. Comparing ireg with information collected before the crisis, it seems that this was already the case before the crisis. The exception is Ireland where the regulatory policy system was more advanced before the crisis and is now less of a priority because of austerity measures. Figure 2. Grouping of countries across policy tools RIA advanced RIA intermediate RIA starting ex post advanced ex post intermediate ex post starting ex post advanced ex post intermediate ex post starting ex post advanced ex post intermediate ex post starting stakeholder advanced AUS CAN GBR MEX CHE EST (USA)* SVN CZE SVK stakeholder intermediate DEU EU NZL BEL KOR POL SWE AUT HUN ITA LUX FRA DNK ISL NOR FIN GRC TUR stakeholder starting ESP IRL ISR NLD PRT CHL JPN Note: *The underlying scores are an average of the scores for primary laws and subordinate regulations, except for stakeholder engagement and regulatory impact assessment for the United States, for which only the results for subordinate regulations were used. The ings are based on countries that scored 0.5 above or below the mean score. The Grouping of countries is based on Regulatory Policy Outlook, p. 29. Source: OECD, 2015 Indicators of Regulatory Policy and Governance. Online available at: Box 2. Description of country ing The ing of countries in each policy tool is derived from the OECD Regulatory Policy Outlook (OECD, 2015, p. 29) and based on the mean of countries scores on each policy tool. Countries that score 0.5 above the mean belong to the advanced, countries that score 0.5 below the mean belong to the starting and the remaining countries form the middle. Countries average score on stakeholder engagement is 2.09 and on RIA For these two policy tools, countries average score is close to the mean of the indicator, i.e. 2. However, for ex post evaluation, countries average score is at 1.54 and thus considerably below the mean of the indicator. Hence, for ex post evaluation it has to be kept in mind that countries scoring above 1.04 already belong to the middle and countries scoring above 2.04 already belong to the advanced. Nine countries would move from the middle to the starting if the average of possible scores, 2, were to be used instead of the actual average. Which average is used makes no difference for countries in the advanced as the country with the lowest score in the advanced has a score of 2.51, scoring clearly above both the mean of the indicators and the mean of OECD countries. This method results in the following distribution of countries across the three s: For stakeholder engagement, 11 countries are in the starting, 14 countries in the middle and 10 countries in the advanced. For RIA, 11 countries are in the starting, 14 countries are in the middle and 10 countries are in the advanced. For ex post evaluation, 10 countries are in the starting, 18 countries are in the middle and 7 countries are in the advanced. In order to explore the sensitivity of the country ing, we extended and reduced the range of the middle by 10% (0.525 above and below and above and below the respective indicator mean instead of 0.5). With respect to the stakeholder engagement indicator, if the range of the middle was extended by 10%, Spain would 6

7 move up from the starting to the middle and the Czech Republic would move down from the advanced to the middle, If the range of the middle for stakeholder engagement was reduced by 10%, Poland would move up from the middle to the advanced. With respect to the RIA indicator, the Czech Republic would move up from the middle to the advanced if the range of the middle was reduced by 10%. With respect to the ex post evaluation indicator, extending or reducing the range of the middle does not lead to any changes in the ing. Consequently, the country ing is not particularly sensitive to small changes in the limits. 12. A crucial finding of the Regulatory Policy Outlook (OECD, 2015) is that ex post evaluation tools are much less systematically applied across countries than ex ante evaluation. Ex post evaluation seems to be the last step in the regulatory policy reform agenda. This is confirmed in the country ing, with less countries scoring clearly above average on ex post evaluation than on RIA and stakeholder engagement and this despite the fact that the average is significantly lower for ex post evaluation than for the other two tools (see Box 2). 13. A number of political and technical challenges may explain the lack of emphasis on systematic ex post evaluation (OECD, 2013): Governments may focus on implementing their election promises and on solving daily crises instead of reviewing what was regulated some time ago. Second, systematic ex post evaluation requires resources which are scarce. Third, reviews can also lead to review fatigue if reviews are not targeted and their benefits are not clear. Fourth, regulations that are costly to an economy can also provide benefits to some s or be perceived to be more widely beneficial, so that reform may accordingly be resisted. And in some cases governments may wish to implement systematic ex post evaluation but face important challenges to move the culture in the administration towards open and evidence-based policy-making. Successful ex post evaluation does require a culture of acknowledging and allowing for failure and ministries do not want to be blamed for unsuccessful policy measures. Given these challenges the institutional setting including a standing capacity with reference to regularly undertake reviews in key policy areas and political leadership to communicate the potential gains of ex post evaluation and overcome resistance to reform are essential. 14. Countries scores on the three indicators are a continuum and small changes in practices can therefore lead to changes in their position relative to other countries, in particular for the stakeholder engagement indicator and to a lesser extent for the RIA indicator (see Annex I). Consequently, a few countries at the boundaries of the three s would move up to the next higher following some modest reforms but a few countries may also easily move down a in case of stagnation (see Box 2). This is much less the case for the ex post evaluation indicator where the gaps between the three s are large. III. POLICY CHOICES TO IMPROVE EVALUATION AND STAKEHOLDER ENGAGEMENT SYSTEMS 15. Countries wishing to improve their regulatory policy systems may look at countries that are one step ahead for guidance. This section analyses the key policy differences that distinguish countries in one from the next. It draws for this purpose on the subcategories that constitute the composite indicators for stakeholder engagement, RIA and ex post evaluation. 7

8 Figure 3. Overview of categories for the three composite indicators 16. Each of the three composite indicators is composed of the same four categories, which provide critical information on the building blocks of regulatory policy: Systematic adoption records formal requirements and how often these requirements are conducted in practice. Methodology gathers information on the methods used in each area, e.g. the type of impacts assessed or how frequently different forms of consultation are used. Transparency records information from the questions that relate to the principles of open government, e.g. whether government decisions are made publically available. Oversight and quality control records the role of oversight bodies and publically available evaluations. The maximum score that can be achieved on each category is A number of countries that are in the starting for a regulatory policy tool might already have put in place quite advanced practices in one or more of the four categories systematic adoption, methodology, transparency and oversight and quality control. To further look into this, we calculated, separately for starting, intermediate and advanced countries (see Box 2), the percentage of countries that have advanced practices in place for each of the four categories, scoring above the category mean (i.e. > 0.5) This analysis helps to understand at what stage countries introduce requirements to systematically use a tool, when transparency measures are taken, at what stage an advanced methodology is developed 2 Individual country scores on the four categories are online available at: 8

9 and when oversight and quality control is put in place. The results are reported below separately for stakeholder engagement, RIA and ex post evaluation. Stakeholder engagement 19. A process of communication, consultation and engagement which allows for public participation in the regulation making process as well as in the revision of regulations can help governments understand citizens, business and other stakeholders needs. In addition, administrations may be pursuing public consultation in an effort to regain trust in governments (OECD, 2015). Also, it can help governments collect more information and resources, increase compliance and reduce uninformed opposition. It may enhance transparency and accountability as interested parties gain access to detailed information on potential effects of regulation on them (OECD, 2012). 20. As reflected in the Regulatory Policy Outlook, most OECD countries have put in place the legal requirements for stakeholder engagement. This shows clearly in the fact that all s of countries reach a very high level for "systematic adoption" (Figure 4). Already in the starting, more than half of the countries have taken the necessary policy measures to systematically conduct stakeholder engagement, scoring above average in the category systematic adoption (Figure 4). In particular most countries in the starting systematically conduct stakeholder engagement on draft laws. By contrast, methodology, transparency and oversight and quality control are much less developed. Figure 4. Analysis of categories across s: Stakeholder engagement 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Systematic adoption Methodology Transparency Oversight and quality control Starting Intermediate Advanced Note: Each pillar represents the percentage of countries in the respective that score above 0.5 (with the highest possible score being 1) for the respective category (systematic adoption, methodology, transparency, oversight and quality control). The underlying scores are an average of the scores for primary laws and subordinate regulations, except for the United States, for which only the results for subordinate regulations are used. The Grouping of countries is based on Regulatory Policy Outlook, p. 29. For stakeholder engagement, 11 countries are in the starting, 14 countries in the middle and 10 countries in the high. 9

10 Source: OECD, 2015 Indicators of Regulatory Policy and Governance. Online available at: What differentiates critically the middle from the starting is the further development of the methodology for stakeholder engagement. Two-thirds of OECD countries score above the average on methodology in the middle compared to less than 10% in the starting. Key practices in this respect include early stage consultation to identify options to address a policy problem, the use of interactive websites for consultation and methodological guidance for stakeholder engagement. Minimum periods for consultation are also less common among the countries in the starting, although even some countries in the middle and advanced do not have minimum periods for consultation in place. Beyond the development of the methodology, some countries of the middle have enhanced transparency in their engagement with stakeholders, yet two third of countries in this still score below average on transparency. 22. By contrast, countries of the advanced have systematically invested in transparency in addition to systematic adoption and methodology. In most countries in this any member of the public can choose to participate in a consultation and in more than half of countries there is a requirement to publish a response to consultation comments online for all or major primary laws and subordinate regulations. "Oversight and quality control remains the most challenging area, even for the most advanced countries. Only 60% of countries in this score above average, having established some institutional responsibility for stakeholder oversight and publishing some information on the functioning of their stakeholder system. Furthermore, even in the advanced, a number of countries never conduct consultations open to the general public at early stage, when a decision to regulate has not yet been made, which affects their score on methodology. Regulatory impact assessment 23. RIA is both a tool and a decision process for informing political decision makers on whether and how to regulate to achieve public policy goals. Improving the evidence base for regulation through a prospective impact assessment of new regulations is one of the most important regulatory tools available to governments. The aim is to improve the design of regulations by assisting policy makers to identify and consider the most efficient and effective regulatory approaches, including the non-regulatory alternatives before they make a decision (OECD, 2012). 24. One-third of countries that are in the beginning phase to introduce RIA have systematically adopted RIA practices and one-fourth has developed an advanced methodology (Figure 5). In particular, most countries in the starting systematically assess budget and public sector impacts as part of their methodology and half systematically require quantification of costs of regulation, in particular administrative burdens and government administration and enforcement costs for primary laws. As for stakeholder engagement, transparency and oversight of RIA only come next in the reform process no country in the starting scores above average in these categories. Only one country in the starting has established an oversight body for RIA for both primary laws and subordinate regulations and four countries have done so for primary laws only. 25. More than two-thirds of OECD countries in the middle systematically conduct RIA and have developed a more refined methodology. In particular, most countries in this conduct impact assessments proportionate to the significance of the regulation, assess distributional effects, impacts on competition and environment and systematically quantify administrative burdens and government administration and enforcement costs. About half of countries in the middle also systematically require the quantification of substantive compliance costs. Furthermore, half of the countries in the middle require regulators to identify a process of assessing progress in achieving regulations goals and to 10

11 consider issues of compliance and enforcement when developing new laws and regulations, which is not the case in the starting. 26. In contrast to the starting, about half of the countries in the middle have significantly invested in the transparency of the RIA system. In most countries in this, RIAs for primary laws are made publicly available online in a central registry and in one-third they are also released at least in some cases for public consultation. Furthermore, practices for primary laws seem to be extended to subordinate regulations when moving from the starting to the middle : While in the starting only two countries publish all RIAs on subordinate regulations online, six countries do so in the middle. A key difference between countries in the starting and in the middle is the establishment of a government body responsible for reviewing the quality of RIA for both primary laws and subordinate regulations. Still, only one-fifth of countries in the middle score above the average in the category oversight and quality control, which is mostly linked to the lack of publically available performance evaluation of RIA and, in some countries, lack of power of the oversight body. Figure 5. Analysis of categories across s: Regulatory impact assessment 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Starting Intermediate Advanced Systematic adoption Methodology Transparency Oversight and quality control Note: Each pillar represents the percentage of countries in the respective that score above 0.5 (with the highest possible score being 1) for the respective category (systematic adoption, methodology, transparency, oversight and quality control). The underlying scores are an average of the scores for primary laws and subordinate regulations, except for the United States, for which only the results for subordinate regulations are used. The Grouping of countries is based on Regulatory Policy Outlook, p. 29. For RIA, 11 countries are in the starting, 14 countries are in the middle and 10 countries are in the advanced. Source: OECD, 2015 Indicators of Regulatory Policy and Governance. Online available at: In the advanced all countries have systematically adopted RIA and developed a strong methodology. In particular, requirements are extended to subordinate regulations, cover a variety of different impacts and regulators in most countries need to demonstrate that the benefits of new regulation justify the costs. However, even in the advanced systematic quantification of benefits is limited to half of the countries in the. All countries in the advanced RIA put emphasis on transparency of RIA, scoring above average in this category (Figure 5). In almost all countries in this RIAs are made publicly available online and the public learns in advance that a RIA is due to take place. 11

12 Furthermore, contrary to the starting and middle, in most countries in the advanced the public is informed if a decision has been made not to conduct a RIA and there is a body responsible for reviewing the decision made by officials whether a RIA is required. In half of the countries in the the general public can contest the decision not to conduct a RIA. However, even in the advanced only two countries always release RIAs for public consultation. 28. In contrast to the middle and starting, about two-thirds of countries in the advanced have introduced institutions and processes for strong oversight and quality control. All countries in this have an oversight body to promote the quality of RIA and the oversight body can return the impact assessment for revision where it is deemed inadequate with the exception of one country. Most countries in this also publish performance reports on RIA; half of the countries in this do so regularly and also provide information on the percentage of RIAs that comply with formal requirements. Ex post evaluation of regulations 29. The evaluation of existing policies through ex post impact analysis is necessary to ensure that regulations are effective and efficient. Furthermore, in the absence of a process of renewal, the volume of red tape and regulatory costs tends to accrete over time (OECD, 2012). This complicates the daily life of citizens and impedes the efficient functioning of business. 30. However, as already highlighted in the Regulatory Policy Outlook, systematically assessing the actual functioning of laws and regulations once they are in place is not well developed in most OECD countries. In contrast to ex ante evaluation and stakeholder engagement no countries in the starting score high in this category - nor does any country in this score above average in any other category (Figure 6). The data confirms the strong focus on administrative burden reduction in OECD countries. More than half of countries in the starting have done ad-hoc reviews focusing on administrative burdens, but they do not evaluate the achievement of objectives of regulations with a few exceptions. Figure 6. Analysis of categories across s: Ex post evaluation 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Starting Intermediate Advanced Systematic adoption Methodology Transparency Oversight and quality control Note: Each pillar represents the percentage of countries in the respective that score above 0.5 (with the highest possible score being 1) for the respective category (systematic adoption, methodology, transparency, oversight and quality control). The underlying scores are an average of the scores for primary laws and subordinate regulations. The Grouping of countries is based on Regulatory 12

13 Policy Outlook, p. 29. For ex post evaluation, 10 countries are in the starting, 18 countries are in the middle and 7 countries are in the high. Source: OECD, 2015 Indicators of Regulatory Policy and Governance. Online available at: Interestingly and in contrast to practices in RIA and stakeholder engagement, even countries in the middle do not score strongly on systematic adoption, meaning that they do not conduct ex post evaluation systematically. Most countries have not either developed a refined methodology: In two-thirds of countries, most ex post evaluations that are carried out do not contain an assessment of whether the underlying policy goals of regulation have been achieved and there are no standardised evaluation techniques. Strikingly, countries in the middle put a strong emphasis on transparency for the reviews they conduct, with two-thirds of countries in this scoring above average on transparency. Most countries in this engage stakeholders in ex post evaluation of both primary laws and subordinate regulations and half of countries in this make evaluations publicly available over the internet. The low priority given to oversight and quality control in terms of reform priorities is confirmed even more strongly for ex post evaluation, with no country in the middle scoring above average. 32. Countries in the advanced depart strongly from the other s in all areas. They systematically conduct ex post evaluations and have a permanent body that conducts in-depth reviews of regulations in specific sectors or policy areas. A key difference between the middle and the advanced are improvements in the methodology of ex post evaluations. In most countries in the advanced, all ex post evaluations contain by default an assessment of the achievement of goals and more than half of the countries systematically assess consistency with other regulations and take steps to address areas of overlap, duplication and inconsistency. All countries in this have developed guidelines for conducting ex post evaluations, yet only about half have developed standardised evaluation techniques. Countries in the advanced have also further enhanced transparency of ex post evaluations, with all countries making evaluations of existing regulations publicly available. Only in the advanced are countries that put emphasis on oversight and quality control for ex post evaluation, having established a quality control system for ex post evaluations and publishing performance information on the functioning of ex post evaluation. Areas for improvement for a number of countries in this include establishing a method to ensure that a required type of assessment is completed and publishing information and indicators on the functioning of the ex post evaluation system. Overview across the three tools 33. Results from the combined analysis of the three policy tools tend to show that countries have a staged approach to regulatory policy, with more countries having well-developed stakeholder engagement and RIA systems than ex post evaluation systems. This can be further analysed by aggregating the four sub-categories described above across the three regulatory policy tools (RIA, stakeholder engagement and ex post evaluation). Figure 7 illustrates that on average countries seem to have opted for similar reform choices for building their systems for stakeholder engagement and RIA: Formal requirements and a standard methodology are in place for many countries for stakeholder engagement and RIA, but less so for ex post evaluation. Practices of RIA and stakeholder engagement seem to be less transparent than those for ex post evaluation. Clearly, the area of oversight and quality control is the least developed for all three indicators. This area is nevertheless essential to provide credibility and the capacity to monitor the effectiveness of regulatory policy. This result emphasises the findings of the Outlook that countries need to go the extra mile to ensure that regulatory policy is not only perceived and carried out as a procedural exercise, but exert the necessary pressure that will ensure regulatory quality in fine. 13

14 Figure 7. Average category scores separated by policy tools Ex post evaluation Regulatory impact assessment Stakeholder engagement Systematic adoption Methodology Transparency Oversight and quality control Note: Scores represent the average of primary laws and subordinate regulations, except for the scores of Regulatory Impact Assessment and Stakeholder Engagement for the United States, for which only the results for subordinate regulations are used. Source: OECD, 2015 Indicators of Regulatory Policy and Governance. Online available at: Extending the scope of practices 34. When building their regulatory policy systems, do countries equally invest in practices for primary laws and subordinate regulations or do they tend to focus on one type of regulation before they extend the practice? The latter seems to be the case for RIA and ex post evaluation where the overall average is higher for primary laws than for subordinate regulations (see Table 2). Table 2. Averages for primary laws and subordinate regulations for the three policy tools Average for Primary Laws Average for Subordinate Regulations Stakeholder engagement Regulatory impact assessment Ex post evaluation Source: OECD, 2015 Indicators of Regulatory Policy and Governance. Online available at: With respect to RIA, countries in the starting conduct RIAs more systematically for primary laws than for subordinate regulations, confirming the hypothesis that countries tend to focus on primary laws before they extend the practice. In contrast, most countries in the middle and advanced have similar practices in place for primary laws and subordinate regulations with the exception of 14

15 transparency of RIA: in all three s, many countries have more transparent practices in place for primary laws than for subordinate regulations. 36. For ex post evaluation, countries in the starting tend to focus more on evaluating primary laws, while countries in the advanced tend to have stronger practices in place for subordinate regulations than for primary laws. In particular the US and Canada seem to have a stronger focus on evaluating subordinate regulations. In these countries subordinate regulations do play a very important role in the regulatory framework which may explain the particular importance attached to regulatory policy practices in this area. Contrary to RIA, there are several countries in all s that have different practices in place for primary and subordinate regulations. 37. For stakeholder engagement, contrary to the other two tools, the overall average for primary laws and subordinate regulations is the same and there is no clear difference between the starting, middle and advanced in terms of having stronger practices in place for either primary laws or subordinate regulations. This does not mean that countries have the same practices in place for primary laws and subordinate regulations. In all three s there are countries that have adopted practices more systematically for one type of regulation than for another. In particular Canada, France and Japan have stronger practices in place for subordinate regulations and Denmark, Ireland, Israel, Korea and Switzerland seem to focus more on primary laws. Policy complementarity 38. The Recommendation clearly emphasises the complementary nature of its 12 principles, including the importance of combining open government to ensure that regulation serves the public interest and is informed by the legitimate needs of those interested in and affected by regulation (principle 2) and oversight mechanisms to support and implement regulatory policy (principle 3). Oversight bodies, transparency measures along the lines of sunshine disinfects and consultation with stakeholders can all contribute to control the quality of RIAs and ex post evaluations. Oversight bodies systematically provide technical expertise and ensure RIAs are actually conducted and are of sufficient quality. Consultation can help to provide the views of those affected by regulation to improve RIAs and provide, together with transparency measures, some checks and balance on the government. 39. However, given resource constraints, countries may not always be able to put all these measures in place at the same time to ensure high-quality RIAs and ex post evaluations. A legitimate question may then be whether, in practice, countries may substitute or complement quality control by oversight bodies by consulting with external stakeholders and making information publicly available? Can substitution lead to a higher risk of poor implementation of regulatory policy or regulatory capture? With respect to RIA, the evidence does not support a substitution effect between public consultation and transparency on the one hand, and oversight and quality control on the other. Many countries have oversight bodies for RIA in place and those that put emphasis on transparency and public consultation usually also have an oversight body. However, consultation on RIA is a rare practice even for countries with advanced systems. 40. The situation looks different for ex post evaluation. Most countries do not have a body overseeing ex post evaluations, but do engage with stakeholders and about half of these countries also publish ex post evaluations online. So there seems to be a priority given to stakeholder engagement and, to a lesser extent, transparency, and much less focus on government institutions to ensure ex post evaluations are conducted and are of high quality. There is no substitution effect the other way round most countries with a body in charge of oversight of ex post evaluations also engage with stakeholders and publish ex post evaluations online. 15

16 41. The question arises whether transparency and the possibility for stakeholders to engage on its own provide sufficient quality control. This requires a public that is both interested, actively engaged and has the expertise to understand evaluations to provide a systematic quality control on its own. As Etzioni (2014, p.687) explains Most people are busy making a living and maintaining a family and a social life, and they have very limited time and energy to devote to following public affairs. And, as studies reviewed in Thinking, Fast and Slow by Nobel laureate Daniel Kahneman (2011, p ) show, people do not have the training necessary to evaluate the relevant data. This of course does not mean that transparency is not important on the contrary data and information can be picked up by academics, journalists, civil society and business organisations and feed into a public debate. The question is just if this is sufficient on its own for systematic quality control and if those actors that do engage are sufficiently representative to avoid regulatory capture. IV. CONCLUSION 42. Regulatory policy should be seen as a systemic and whole of government approach to regulatory quality. The key principles of regulatory policy (as embodied in the 2012 Recommendation of the Council on Regulatory Policy and Governance) demonstrate the importance of pursuing this systemic approach and adopting all 12 Principles. However, the Recommendation does not provide for a roadmap of regulatory policy reforms, nor does it clarify the steps to build, in a resource-constrained environment, the elements of a regulatory policy that will ensure regulatory quality. This is precisely the question that this paper aims to address by reviewing the country evidence collected through the 2014 Survey of regulatory policy and governance and analysing in combination the three composite indicators of stakeholder engagement, regulatory impact assessment and ex post evaluation. 43. The paper found that countries that have invested in regulatory impact analysis also generally put emphasis on stakeholder engagement and ex post evaluation. The link between stakeholder engagement and ex post evaluation is less strong. Some countries have made significant investments in stakeholder engagement but only started to develop ex post evaluation practices. Moreover, the in-depth analysis revealed some regional and institutional country clusters, with similar practices among countries with common law systems, among Nordic countries and among Eastern European countries. This finding might indicate that regional or institutional proximity facilitates the diffusion of regulatory policy practices leading to similar reform choices as regards the development of the three regulatory policy tools. Of course, regional and institutional proximity of countries is just one factor among many others potentially influencing the development of regulatory policy systems. 44. The analysis of progress of countries in developing each policy tool is highly instructive. For stakeholder engagement and regulatory impact assessment, the pattern is similar. Countries seem to start with the systematic adoption of requirements, then they refine the methodology, enhance transparency and at last, they establish mechanisms for oversight and quality control. For ex post evaluation, in contrast to the other two policy tools, only the few countries in the advanced conduct ex post evaluations systematically including key elements such as the evaluation of the achievement of objectives of regulations. The starting conducts mostly ad-hoc reviews with a focus on administrative burdens and the middle has already invested in transparency and stakeholder engagement. The low priority given to oversight and quality control is confirmed even more strongly for ex post evaluation, with room for significant progress even for advanced countries. 16

17 45. While it is straightforward that countries start by introducing formal requirements and developing a standard methodology, it might be worth discussing why strong oversight and quality control only seems a priority for advanced countries. This begs the question of where to put the emphasis for countries that have established the legal conditions for regulatory policy: in refining and extending the methodology for regulatory policy tools or in ensuring that formal requirements are implemented through a credible oversight framework. A key finding of the Regulatory Policy Outlook and the 7th OECD Conference on Measuring Regulatory Performance in Iceland in 2015 identified a high potential for improvements to the institutional framework for regulatory policy in OECD countries to bridge the gap between formal requirements and implementation, leading to higher impact of regulatory policy on economic and social outcomes. This will be at the core of discussions at the next MRP workshop in Australia on Realising impact: The role of institutional frameworks in regulatory policy. 46. Oversight bodies, transparency measures along the lines of sunshine disinfects and stakeholder engagement can all contribute to control the quality of RIAs and ex post evaluations and are complementary. For RIA, many countries have oversight bodies in place and those that put emphasis on transparency and stakeholder engagement usually also have an oversight body. However, stakeholder engagement on RIA is a rare practice even for countries with advanced systems. It seems that the application of this practice requires a protracted cultural change towards a more transparent public administration and a political coalition willing to drive this change. For ex post evaluation, there seems to be some substitution the other way around: Stakeholder engagement and to a lesser extent transparency are much more common than oversight bodies. It does not seem straightforward to replace one measure by the other as they all contribute in different ways to the quality control of evaluations. 47. Finally, many countries starting to build their RIA and ex post evaluation systems focus first on primary laws while countries at the next stages tend to have more similar practices in place for both types of regulation. This might be an option for gradually scaling up the regulatory policy system in a country and at the same time may lead to some lack of coherence. 17

18 V. BIBLIOGRAPHY Arndt, C., Antonia Custance Baker, Tobias Querbach and Rebecca Schultz, (2015), "2015 Indicators of Regulatory Policy and Governance: Design, Methodology and Key Results", OECD Regulatory Policy Working Papers, No. 1, OECD Publishing, Paris. Etzioni, A. (2014), The Limits of Transparency, Public Administration Review, Vol. 74/6, pp Available at SSRN: Freudenberg, M. (2003), Composite Indicators of Country Performance: A Critical Assessment, OECD Science, Technology and Industry Working Papers, No. 2003/16, OECD Publishing, Paris, Kahneman, D. (2011), Thinking, Fast and Slow, Farrar, Straus and Giroux, New York. OECD (2015), OECD Regulatory Policy Outlook 2015, OECD Publishing, Paris, OECD (2014), OECD Framework for Regulatory Policy Evaluation, OECD Publishing, Paris. OECD (2013), Assessing Progress in the Implementation of the 2012 Recommendation of the OECD Council on Regulatory Policy and Governance, Policy Findings and Workshop Proceedings, 5th Expert Meeting on Measuring Regulatory Performance, OECD, Paris, OECD (2012), Recommendation of the Council on Regulatory Policy and Governance, OECD Publishing, Paris, 18

19 ANNEX I Composite indicators: Stakeholder engagement for developing regulations Austria Chile Hungary Ireland Israel Italy Japan Luxembourg Netherlands Portugal Spain Belgium Denmark EU Finland France Germany Greece Iceland Korea New Zealand Norway Poland Sweden Turkey Australia Canada Czech Republic Estonia Mexico Slovak Republic Slovenia Switzerland UK (US) 0 Note: The vertical axis represents the total aggregate score across the four separate categories of the composite indicators. The maximum score for each category is one, and the maximum aggregate score for the composite indicator is four. The scores are an average of the scores for primary laws and subordinate regulations, except for the United States, for which only the results for subordinate regulations are presented. The ings are based on countries that scored 0.5 above or below the mean score. Source: OECD Regulatory Policy Outlook (OECD, 2015), p

20 Composite indicators: Regulatory Impact Assessment for developing regulations Chile Denmark Finland Greece Iceland Israel Japan Netherlands Norway Portugal Turkey Austria Belgium Czech Republic France Hungary Ireland Italy Korea Luxembourg Poland Slovak Republic Slovenia Spain Sweden Australia Canada Estonia EU Germany Mexico New Zealand Switzerland UK (US) 0 Note: The vertical axis represents the total aggregate score across the four separate categories of the composite indicators. The maximum score for each category is one, and the maximum aggregate score for the composite indicator is four. The scores are an average of the scores for primary laws and subordinate regulations, except for the United States, for which only the results for subordinate regulations are presented. The ings are based on countries that scored 0.5 above or below the mean score. Source: OECD Regulatory Policy Outlook (OECD, 2015), p