RAISING EFFECTIVENESS AND QUALITY IN PUBLIC ADMINISTRATION THROUGH APPLICATION OF THE COMMON ASSESSMENT FRAMEWORK (CAF MODEL): TATARSTAN CASE

Transcription

1 RAISING EFFECTIVENESS AND QUALITY IN PUBLIC ADMINISTRATION THROUGH APPLICATION OF THE COMMON ASSESSMENT FRAMEWORK (CAF MODEL): TATARSTAN CASE Liliya NURGATINA Director of School of Public Administration of Kazan (Volga region) Federal University Kazan, Russia Dmitry MASLOV Head of the Research Centre for Effectiveness and Quality in Public Administration School of Public Administration, Kazan (Volga region) Federal University Kazan, Russia ABSTRACTS The paper presents recent Tatarstan initiative in field of effectiveness and quality implementation of the European Common Assessment Framework (CAF model) which since 2000 offers an easy to use tool to assist public sector organisations across Europe to develop quality management and continues improvement philosophy. The CAF Model is a comprehensive system of assessment, analysis and improvement on the basis of employee engagement (through the mechanism of self-assessment), measuring the satisfaction of various stakeholders (in particular citizens/customers), bench learning and best practice exchange between CAF users, and implementing the process approach. Tatarstan Government with methodological and organisational support of the School of Public Administration of Kazan Federal University and European Institute of Public Administration (EIPA) started its quality journey with CAF Model as a pilot region in Russia. First findings of adapting CAF to Russian conditions could consider that model as a methodological framework for the development of an integrated system of performance measurement and quality improvement for Public Administration in Russia. Paper is focused on municipal level and methods of internal organisational self-assessment. Authors propose a Quality Management System Index (QMS-Index) as a "dashboard" for decision making for local governments, which allows managers to monitor their activities and diagnose areas for improvement. POINTS FOR PRACTITIONERS More than 3000 public organisations have registered to use the CAF Model since its launch and thousands more across and outside Europe use it for their own specific development purposes. Despite CAF originally focused on the Member States of the European Union, other countries actively use CAF in order to support people working in public administrations in their day-to-day journey of delivering a quality service. The School of Public Administration of Kazan Federal University, being a national partner of the EIPA CAF Resource centre in Russia, develops CAF in this area and supports Public Administrations in Russia and CIS countries to implement CAF model and exchange best practices in field of quality and effectiveness in Public Sector. QMS-Index is based upon CAF criteria and methodology of self-assessment. It can be implemented as a diagnostic technique and a total quality management (TQM) instrument. Attached questionnaire will help practitioners from public sector to conduct self-assessment and start to implement CAF model in their organisations. KEY WORDS Effectiveness, Measurement, Self-Assessment, Performance, Quality 1. INTRODUCTION The Russian public administration sphere, in an effort to achieve higher administrative efficiency, is receiving a massive injection of western management techniques, like management by results, performancebased budgeting, e-government, and many others. Quality and Effectiveness issues look very important in reform

2 agenda. But largely recognized common methodology for measuring, analyzing and improving effectiveness in public administrations, efficiency of civil servants and quality of public services still does not exist both at the regional and federal level of the Russian Federation. Tatarstan Republic is one of the leading regions of Russia in terms of effectiveness of public administration. Tatarstan government undertakes various initiatives in order to enhance its efficiency improve performance results and raise satisfaction of citizens/customers. 2. CAF MODEL CAF in brief Since 2000 in European Public Sector is widely used The Common Assessment Framework a TQM Model inspired by the EFQM Excellence Model. The CAF is a result of co-operation among the EU Ministers responsible for Public Administration. The CAF Resource Centre at the European Institute of Public Administration is in charge of further development of CAF methodology and coordination of the network of CAF users across Europe. The CAF is offered as an easy to use tool to assist public sector organisations to introduce quality management thought the comprehensive self-assessment framework that is conceptually similar to the major TQM models, EFQM in particular, but is specially conceived for the public sector. Among CAF users there are organisations from such field like Education, Social Services, Healthcare, Transport and Infrastructure, Police and even Churches. But the most CAF-claimed area is Federal Governmental Bodies as well as Local and Regional Administrations. Being a generic tool CAF includes the 9 criteria, 28 sub criteria and the scoring system. The structure of the CAF Model is illustrated on fig. 1. Further description of the CAF Model is given by CAF-2013 brochure 1, which is available online on the CAF website (as well as Russian language version 2 ). Figure 1. The CAF Model The nine-box structure identifies the main aspects requiring consideration in any organisational analysis. Criteria 1-5 deal with the managerial practices of an organisation: the so-called Enablers. These determine what the organisation does and how it approaches its tasks to achieve the desired results. In criteria 6-9, results achieved in the fields of citizens/customers, people, social responsibility and key performance are measured by perception and performance measurements. Each criterion is further broken down into a list of sub-criteria. The 1 The Common Assessment Framework. Improving Public Organisations through Self-Assessment. CAF 2013 / CAF Resource Centre. EIPA P Общая схема оценки Common Assessment Framework (CAF). Совершенствование организаций публичного сектора через самооценку / перевод с англ. под ред. И.Р. Гафурова, Л.А. Нургатиной, Д.В. Маслова, // European Institute of Public Administration (EIPA) CAF Resource Centre. 2012, Высшая школа государственного и муниципального управления КФУ, Казань: Издательство Казанского университета, с.

3 28 sub-criteria identify the main issues that need to be considered when assessing an organisation. They are illustrated by examples that explain the content of the sub-criteria in more detail and suggest possible areas to address, in order to explore how the administration fulfills the requirements expressed in the sub-criterion. These examples represent a lot of good practices from all over Europe. Not all of them are relevant for every organisation, but many can be considered as points of attention during self-assessment. Integrating the conclusions from the assessment of the enablers and results criteria into the managerial practices constitute the continuous innovation and learning cycle that accompanies organisations on their way towards excellence. The CAF aims to be a catalyst for a full improvement process within the organisation and has five main purposes: 1. to introduce public administrations into the culture of excellence and the principles of TQM; 2. to guide them progressively to a fully-fledged PDCA (PLAN, DO, CHECK, ACT) cycle; 3. to facilitate the self-assessment of a public organisation in order to obtain a diagnosis and a definition of improvement actions; 4. to act as a bridge across the various models used in quality management, both in public and private sectors; 5. to facilitate bench learning between public sector organisations. Since its launch, nearly 3000 public sector organisations across and outside Europe have used the model, and the number of CAF users continues to grow. Tatarstan CAF-context The national effectiveness assessment framework was introduced in Russia by the Presidential Decree in These system is resulted in annual rank of all subjects of federations. Tatarstan is always on top of this rating. One of the reasons is searching for new methods and techniques to raise quality and efficiency of public administration. In the end of 2012 Kazan (Volga region) Federal University (KFU), supported by the government of Tatarstan Republic, signed a bilateral agreement with the European Institute of Public Administration (EIPA) and became National partner in field of distributing CAF Model in Russia (with focus on Tatarstan and Volga region). In 2013 in the framework of this agreement KFU School of Public Administration have implemented three activities: 1. Training and Site visits to European CAF user in Germany, Netherlands and Belgium for civil servants of Tatarstan Republic in April; 2. CAF training in Kazan in August for 150 civil servants delivered by EIPA experts; 3. First International Quality Conference in Kazan in December for 200 participant with contributes from National Ministries and Agencies, Administration of the President of Russian Federation, EIPA experts, CAF practitioners from Norway, Lithuania, Slovenia. In 2013 in KFU School of Public Administration a new research project Methodology of analysis and evaluation of effectiveness and quality in public administration (Volga region case) was launched. The CAF model is considered to be a theoretical framework of developing methodology, and experience of CAF practitioners from EU countries are to form practical basis for further research and bench-learning. 2. QMS-INDEX Ideology Quality Management System Index (QMS-Index) is a set of indicators characterizing the level of quality management system in organisation. Proposed method of calculating the QMS-Index was designed especially for municipal level of public administration in the Republic of Tatarstan. This technique can also be scaled to the level of a region, or used to assess the quality management system in the national context. QMS-Index is calculated on the basis of self-assessment helps to identify good practices and areas for improvement in the municipality and increase the level of quality of municipal services provided to citizenscustomers. One of the main QMS-Index feature is involving employees in the process of diagnostic self-assessment which covers various aspects of the organisation in the framework of desired and achieved level of quality management system in the municipality. QMS-Index is harmonized with the criteria of the CAF Model and the EFQM Excellence Model. Proposed system is not considering as a "punitive" tool or a mechanism of dismissal. In contrast, the QMS-Index aims to establish channels of communication between management and employees for more accurate diagnosis and better adjustment of the management system.

4 Methodology QMS-Index structure consists of 9 indicators divided into two groups: Enables and Results. Each group has equal ratio - 50/50. Indicators within groups have weights, shown in Table 1: Table 1. Wight coefficient of QMS-Index criteria coefficient, k score Enables Criteria 0,50 50 Criterion 1: Leadership 0,10 10 Criterion 2: Strategy and Planning 0,08 8 Criterion 3: People 0,09 9 Criterion 4: Partnerships and Resources 0,09 9 Criterion 5: Processes 0,14 14 Results Criteria 0,50 50 Criterion 6: Citizen/Customer-oriented Results 0,20 20 Criterion 7: People Results 0,09 9 Criterion 8: Social Responsibility Results 0,06 6 Criterion 9: Key Performance Results 0,15 15 Self-assessment is conducted by non-personalized survey. Respondents fill in paper questionnaire or online form, where the first part includes questions regarding importance of different areas of organisation, and the second part achieved level on the same areas (questionnaire is attached - Annex 1). The e-survey can be created using an open application (e.g. Google Drive or SurveyMonkey) or developed an app upon special terms and conditions. Link to the survey has to be sent individually to the work address of each respondent. To ensure the validity of research results with a statistical error of less than 5% it is necessary to provide participation in the survey from 50 to 80 % of employees, depending on their total population in the organisation. It is recommended to conduct self-assessment quarterly (or at least annually) to measure the dynamic of improvements. QMS Dashboard based on consolidated and interrelated results of two part in the questionnaire, harmonized with each other. In the first part, respondents evaluate to what extent this or that area of organisation important for its further development it will be desired profile of management system. In the second part, respondents form actual profile of management system evaluating the same areas in terms of achieved results. Each line in the questionnaire is harmonized with QMS-Index criteria as shown in the table 2. Table 2. Questionnaire and QMS-Index Criteria Relationship QMS-Index Criteria Question No Part 1 Part 2 Enables Criteria 1: Leadership 1; 2; 3; 4; 5; 7 50; 51; 52; 53; 54; 56 2: Strategy and Planning 2; 6; 8; 9; 10; 11; 12; 13; 14 51; 55; 57; 58; 59; 60; 61; 62; 63 3: People 11; 15; 16; 17; 18; 19; 20; 21; 22 60; 64; 65; 66; 67; 68; 69; 70; 71 4: Partnerships and Resources 13; 23; 24; 25; 26; 27; 28 62; 72; 73; 74; 75; 76; 77 5: Processes 10; 13; 24; 27; 28; 29; 30; 31; 32 59; 62; 73; 76; 77; 78; 79; 80; 81 Results Criteria 6: Citizen/Customer-oriented Results 31; 32; 33; 34; 35; 36; 37; 38 80; 81; 82; 83; 84; 85; 86; 87 7: People Results 30; 32; 39; 40; 41 79; 81; 88; 89; 90 8: Social Responsibility Results 35; 36; 42; 43; 44; 46 84; 85; 91; 92; 93; 95 9: Key Performance Results 45; 46; 47; 48; 49 94; 95; 96; 97; 98 Consolidated QMS-Index reflecting the current state of the quality management system is calculated based on the evaluation result of the second part of the questionnaire by the following formulas:

5 where i Criterion No n Question No k Weight coefficient (Table 1) q n Score (0 5 points): q max Maximum score 5 points m Total amount of respondents Q Enables Achieved score in Enables Q Results Achieved score in Results Q 1 9 Achieved score on criterion 1 (achieved score on criteria 2-9 is calculated with regard to the relevant coefficients and questions indicated in Tables 1 and 2). In order to evaluate effectiveness of organisation QMS-Effect could be calculated: Both indicators: achieved QMS-Index and QMS-Effect are reflected at the top of the QMS Dashboard (Figure 2). Following the same algorithm desired level indicators of QMS-Index are calculated. Proposed selfassessment technique allows to define gaps between desired and achieved level of performance of organisation through the prism of 9 criteria, as well as within groups of Enables and Results. In addition, the gap between desired and achieved levels of consolidated QMS-Index can be shown (middle part of the Dashboard in Figure 2). To define a level of maturity for each criterion it is suggested to use the scale similar to CAF methodology where the PDCA (Plan-Do-Check-Act) cycle is integrated into scoring system. If an organisation performs no data or only has plans in this or that area, that means only beginning phase of continues improvement. If organisation performs plan, do, check and act (review and improve), that means high level of maturity. If organisation has fully integrated PDCA cycle it can be consider as a best practice. PDCA scale is applicable to Enable group. Maturity of Results criteria is characterized by trends shown by arrows. For in-depth analysis of the achieved level (results of the second part of the questionnaire) it is suggested to divide respondents into two groups managers (or decision makers) and employees; and calculate all the results separately for each group. The gap in score of managers related to score of employees (overestimation/underestimation) is determined for each specific area (subject) to identify areas of critical differences in the estimates over 25 % (low part of the Dashboard in Figure 2). If necessary, separate analysis by gender and/or age can be done. In order to present all the results of self-assessment it is suggested to use QMS-Dashboard presented in Figure1. It is simple and easy to understand info-graphics of organisational performance that can be used by decision makers in Local Government Sector and replicate in higher levels of Public Administration in Russia and in Europe.

6 Figure 2. QMS-Dashboard 4. CONCLUSION QMS-Index methodology is still developing. There is a strong interest from the Government of Tatarstan Republic to introduce it at municipal level as a pilot project till the end of QMS-Dashboard could be interesting for all CAF users or organisations starting quality journey. It is easy to use express instrument allows to get quick wins after first self-assessment, to understand better organisational performance and communication gaps between decision makers (managers/executives) and decision doers (employers/workers).

7 REFERENCES Common Assessment Framework Good Practice Book Regional Centre for Public Administration Reform of UNDP Bratislava Regional Centre, June. Löffler, E., Vintar, M. ed Improving the Quality of East and West European Public Services. Ashgate, UK. Maslov, D From CAF to EPUS: A New Quality Approach for Russian Public Administration. NISPAcee Journal of Public Administration and Policy. vol. 3. P Maslov, D Internal Contradictions in Russian Reforms. Russian Analytical Digest. 118: Russia's Evolving Political System. P Staes, P., Thijs, N., Stoffels, A., Geldof, S Five Years of CAF 2006: From Adolescence to Maturity What next? A study on the use, the support and the future of the Common Assessment Framework. EIPA. The Common Assessment Framework. Improving Public Organisations through Self-Assessment. CAF CAF Resource Centre. EIPA. ANNEX. QUESTIONNAIRE Your role in organisation: manager/decision maker employee/worker Gender: man woman Age: under over 60 Part 1. Rank the following statements in terms of its contribute to organisational excellence. Use proposed scale 0 No answer 1 Not important 2 Rather not important 3 Rather not important 4 Important 5 Very important n Statement Score 1 Leaders are citizen- and society-oriented in strategic planning 2 Leaders report overall development goals to each employee in the organisation 3 Leaders demonstrate their loyalty to employees 4 Leaders support initiatives from employees 5 Leaders provide all necessary resource to carry out all set organisational functions 6 Organisation has long-term development strategy 7 Former leaders demonstrate their efficiency as informal leaders too 8 Organisation is focused on maximal satisfaction of needs of each citizen and society in whole 8 Staff are well informed about mission and strategy of organisation 10 Intra-and inter-organisational relations function well and provide optimal processes

8 n Statement Score 11 Personal goals of employees correspond with the mission and strategy of organisation 12 Implementation of the mission and strategy are under regular monitoring 13 Organisation systematically gather, analyse and employ information external sources 14 Organisation analyses information about performance of other organisations 15 Organisation encourages knowledge and good practices exchange among employees 16 Organisation develops staff rewarding system 17 Organisation improve working conditions for staff 18 Organization carries out social support, pays much attention to healthcare, safety and recreation 19 Organisation set communication channels for claims and offers from employees up to leaders 20 Organisation has an effective staff appraisal system 21 Employees have opportunities to get further education and professional development 22 Equal opportunity of getting job for men and women, representatives of different religions and nationalities 23 Organisation has sufficient financial security 24 All resources are planned and managed 25 Saving energy and material resources, waste management 26 Information about organisation are available for all stakeholders 27 Organisation is in continues search for new efficient management approaches 28 Finance resource are under control 29 Systematic analysis and improvement of all organisational processes 30 Leaders delegates power to responsible employees in their professional field 31 Regular monitoring of citizens and other stakeholders satisfaction with quality of delivered public services 32 Organisation use ideas on improvement received from different stakeholders 33 Organization analyzes how citizens, society and other stakeholders perceive its activities 34 Organisation analyses needs and expectations of citizens, society, other stakeholders related to delivered public services 35 Organisation cares about its image (media coverage, number of awards, etc.) 36 Organisation analyses what benefit does it bring to citizens, society, other stakeholders 37 Organisation develops additional services to their stakeholders 38 Organisation evaluates the level of confidence of citizens, society, other stakeholders 39 Organisation motivates employees to be more involved in organisational activities 40 Organisation measures people satisfaction 41 Organisation pays attention to achievements of employees (training courses, awards, etc.) 42 Organisation contributes to local society 43 Organisation monitors opinions and complaints from citizens, society and other stakeholders 44 Organisation cooperates with private companies and NGOs in systematic way 45 Organisation analyses results of audits and inspections 46 Organisation analyses its partnerships 47 Organisation conducts cost analysis and optimises its expenses 48 Organisation has solid financial management system 49 Organisation manages its material assets and equipment in effective

9 Part 2. Rank the same statements in terms of achieve results of your organisation. Use proposed scale 3. 0 We are not active in this field, we have no information or very anecdotal. No results are measured and/or no information is available. 1 We have a plan to do this. Results are measured and show negative trends and/or results do not meet relevant targets. 2 We are implementing / doing this. Results show flat trends and/or some relevant targets are met. 3 We check / review if we do the right things in the right way. Results show improving trends and/or most of the relevant targets are met. 4 On the basis of checking / reviews we adjust if necessary. Results show substantial progress and/or all the relevant targets are met. 5 Everything we do, we plan, implement, check and adjust regularly and we learn from others. We are in a continuous improvement cycle on this issue. Excellent and sustained results are achieved. All the relevant targets are met. Positive comparisons with relevant organisations for all the key results are made. n Утверждение Оценка 50 Leaders are citizen- and society-oriented in strategic planning 51 Leaders report overall development goals to each employee in the organisation 52 Leaders demonstrate their loyalty to employees 53 Leaders support initiatives from employees 54 Leaders provide all necessary resource to carry out all set organisational functions 55 Organisation has long-term development strategy 56 Former leaders demonstrate their efficiency as informal leaders too 57 Organisation is focused on maximal satisfaction of needs of each citizen and society in whole 58 Staff are well informed about mission and strategy of organisation 59 Intra-and inter-organisational relations function well and provide optimal processes 60 Personal goals of employees correspond with the mission and strategy of organisation 61 Implementation of the mission and strategy are under regular monitoring 62 Organisation systematically gather, analyse and employ information external sources 63 Organisation analyses information about performance of other organisations 64 Organisation encourages knowledge and good practices exchange among employees 65 Organisation develops staff rewarding system 66 Organisation improve working conditions for staff 67 Organization carries out social support, pays much attention to healthcare, safety and recreation 68 Organisation set communication channels for claims and offers from employees up to leaders 69 Organisation has an effective staff appraisal system 70 Employees have opportunities to get further education and professional development 71 Equal opportunity of getting job for men and women, representatives of different religions and nationalities 72 Organisation has sufficient financial security 73 All resources are planned and managed 74 Saving energy and material resources, waste management 75 Information about organisation are available for all stakeholders 76 Organisation is in continues search for new efficient management approaches 77 Finance resource are under control 3 Assessment panels of CAF-2013 Classical Scoring are used.

10 n Утверждение Оценка 78 Systematic analysis and improvement of all organisational processes 79 Leaders delegates power to responsible employees in their professional field 80 Regular monitoring of citizens and other stakeholders satisfaction with quality of delivered public services 81 Organisation use ideas on improvement received from different stakeholders 82 Organization analyzes how citizens, society and other stakeholders perceive its activities 83 Organisation analyses needs and expectations of citizens, society, other stakeholders related to delivered public services 84 Organisation cares about its image (media coverage, number of awards, etc.) 85 Organisation analyses what benefit does it bring to citizens, society, other stakeholders 86 Organisation develops additional services to their stakeholders 87 Organisation evaluates the level of confidence of citizens, society, other stakeholders 88 Organisation motivates employees to be more involved in organisational activities 89 Organisation measures people satisfaction 90 Organisation pays attention to achievements of employees (training courses, awards, etc.) 91 Organisation contributes to local society 92 Organisation monitors opinions and complaints from citizens, society and other stakeholders 93 Organisation cooperates with private companies and NGOs in systematic way 94 Organisation analyses results of audits and inspections 95 Organisation analyses its partnerships 96 Organisation conducts cost analysis and optimises its expenses 97 Organisation has solid financial management system 98 Organisation manages its material assets and equipment in effective