This is the third and final article in a series on developing

Performance measurement in Canadian government informatics Bryan Shane and Gary Callaghan A balanced performance measurement system requires that certain principles be followed to define the scope and provide a philosophy of operations. This is the third and final article in a series on developing and implementing a performance measurement system (PMS) for an informatics function in a public service environment. 1 The first article presented a conceptual approach to performance measurement based upon the Balanced Scorecard 2 approach, while the second article described the steps to developing and implementing a PMS. This third article describes the realization of a PMS in a Canadian government informatics organization, the Application Management Service (AMS) of Public Works and Government Services Canada (PWGSC). Specifically, this article describes the mission, services and clients of AMS; the rationale for embarking on the performance measurement project; and the development and implementation process. Finally, it provides an assessment of the benefits and risks associated with implementing such an endeavour. Bryan Shane is a senior partner with BPC Management Consultants. For more than 20 years, he has provided consulting services in the areas of strategic management, information technology and performance consulting, to a wide variety of public and private sector organizations. Mr. Shane has a BA in Political Science from Carleton University and a BEd and MEd from the University of Ottawa. He has also completed graduate studies in statistics and evaluation. AMS description AMS is a sector within the Government Telecommunications and Informatics Services (GTIS) branch of PWGSC. Its vision is to develop information management and information technology (IM/IT) partnerships with the department s other business lines and with Special Operating Agencies (SOAs) to maximize PWGSC s value as a Common Services Organization. As one of the department s eight major business lines, GTIS is recognized by the federal government community as the principal agent for the management of common IM/IT infrastructure and services. GTIS provides a wide range of IM/IT services to PWGSC and other departments of the Canadian government. AMS has 418 employees and is augmented by a similar number of consultants. It has an annual budget allocation Gary Callaghan is the Manager of the Application Management Service (AMS) Program Office for the Government Telecommunications and Informatics Services Branch of Public Works and Government Services Canada. He has over 20 years of experience in the information management and information technology field, all with the Canadian federal government. Mr. Callaghan has been a project manager for the past seven years and has spent the last two years developing and implementing the AMS performance measurement framework. 16 Optimum, The Journal of Public Sector Management Vol. 29, No. 1 (16-24)

of $42 million and an in-house supported technology inventory valued at approximately $300 million. Its mission is to provide development and support services in respect of administrative processes and education to government IM/IT professionals. It delivers value by working closely with clients to reduce business risk, while increasing flexibility and performance. To achieve its mission, AMS provides a number of services to its clients, including: Application support includes operational-support and functional-enhancement services primarily for PWGSC applications. 27% officers and/or line managers 43% middle managers FIGURE 1 AMS s client base 7% clerical employees 23% senior managers and/or directors Application development includes major functional enhancements, architectural re-engineering services and IT project management. Application environment includes architecture and environment services, such as standard development tools and techniques and quality management, primarily for AMS. Other AMS services include business process re-engineering, Internet support, data warehousing and software exchange. Figure 1 illustrates the client base of AMS. The majority of AMS clients (87 percent) work within PWGSC, with the remainder coming from outside the department. Most clients are using more than one AMS service, with 71 percent using at least three services. The most widely used services include: application support (84 percent), application development (75 percent), application environment (41 percent), Internet (39 percent), data warehousing (26 percent), and software exchange (24 percent). 3 Rationale PWGSC and GTIS are committed to the review and renewal of federal programs and services to ensure that they are affordable, accessible and responsive to the needs of Canadians. This new environment created a number of challenges and opportunities in the delivery of these programs and services. There is a need to deliver high-quality programs and services across the country in an environment of ongoing fiscal restraint. Staff reductions and costsaving initiatives have created an environment in which employees often have difficulty managing their workloads, with a resulting negative effect on morale. There are opportunities for more effective leadership and simplified management structures that permit long-term, viable alternative approaches to delivering programs and services at reduced costs, based upon new forms of collaboration across branch and department lines. There is a need for a means to facilitate the sharing of best practices in the delivery of programs and services. These challenges and opportunities, which are directly related to the effective delivery of programs and services to the Canadian public, were further compounded by the need for effective performance measurement practices in AMS. Until the Balanced Scorecard approach was implemented, the performance of programs and services was not measured in a holistic way. At present, the established standard in the federal government for reporting performance is solely based on a financial perspective. This one-dimensional approach fails Optimum, The Journal of Public Sector Management Vol. 29, No. 1 17

to provide performance information related to client and employee satisfaction, quality of service or continuous improvement. Many performance measurement systems do not provide the information needed to determine whether objectives and strategies are being effectively implemented. Much performance information is reactive rather than proactive. Moreover, it fails to recognize outstanding group or individual performance, which tends to undermine motivation and morale within the organization. This situation is further compounded by the fact that while approximately 10 percent of all informatics organizations have formal performance measurement programs, only one organization in six that starts such a program achieves a successful implementation. Development and implementation approach To address these problems, AMS undertook a holistic approach to the development and implementation of a PMS. The objective was to develop and implement a balanced and systematic PMS to assess the effectiveness of AMS s operations from various points of view: financial quality of programs and services client satisfaction employee satisfaction Performance measurement requires time, effort, skills, expertise and, perhaps most importantly, the active support of senior management. Performance measurement must be organized around the department s planning and budgeting cycles. Performance measurement business processes for finances, quality of service, employee satisfaction, client satisfaction and continuous improvement must be created or adapted to collect and analyze the data. An Office of Performance Measurement (OPM), in this case the AMS Program Office, must be established and held responsible for the planning, implementation and ongoing operation of the PMS. The OPM must report directly to, and be supported by, senior management. The PMS should be developed and implemented in phases. Simple and existing information sources should be used. There is no need to pursue new, expensive and/or time-consuming approaches to data generation. The PMS must be designed so that multiple lines of evidence are generated for each perspective at different organizational levels and at different times. The information generated from one perspective must be confirmed by another so that there is a convergence of information to support the diagnosis and action on any issue. continuous improvement. The PMS was designed to provide feedback at all three levels: strategic, tactical and operational. It was also designed to evaluate the effectiveness of strategies and plans, to improve decision making, enable proactive problem correction and promote continuous improvement within the organization. Principles Developing and implementing a balanced PMS requires that a number of principles be followed to define the scope of the project and provide a philosophy of operations. AMS was no different. These principles indicate: Steps in developing and implementing a performance measurement system As noted above, the PMS for AMS was implemented in phases. Phase 1 was a pilot project used in three core divisions, two offering development services and one providing support services. In Phase 2, the project was extended to include all high-profile projects, those directly involving clients and some internal to AMS. This expansion covered most directorates within the sector. Phase 3, now underway, will extend the project to the two remaining directorates. The steps used in developing and implementing each phase of the PMS are briefly described below. 18 Optimum, The Journal of Public Sector Management Vol. 29, No. 1

Step 1: Project orientation The first step was to provide an orientation to management and staff on the balanced approach to performance measurement. This orientation covered the roles and responsibilities of participants, the stages of development and implementation, the time frames or milestones, the key principles of performance measurement, and anticipated and unforeseen issues. Step 2: Readiness assessment The second step was to assess the readiness of the AMS directorates and/or divisions to accept the balanced approach to performance measurement. This assessment would demonstrate how the organization s mandate and functions supported or reinforced the business plans of AMS, GTIS and the department. Those responsible for developing the PMS were looking for evidence of management and staff commitment, as well as an assurance that the necessary resources to carry out the project would be made available. Following a positive analysis of the organization s readiness, it was necessary to select core functional areas in which to begin a pilot. Step 3: Performance Measurement Architecture (PMA) The third step was the development of the Performance Measurement Architecture (PMA) to provide the design, content and structure of the PMS. In developing the PMA, business objectives were developed first, then informatics objectives were designed to support the business directions. Performance measures and performance indicators were then derived for the informatics function. These measures and associated indicators are objective, qualitative, quantitative and output oriented, but are also quantifiable in nature (see Information quality below for details). The PMA was an evolving document, changing as measures were implemented and feedback was received. Once the PMA was approved, it became the sanctioned PMS architecture for AMS (see Table 1) and it was important to place it under change control, i.e., proposed changes were scrutinized and approvals were required. Step 4: Implementation As noted above, Phase 1 began on a pilot basis in three core areas. This approach was used to demonstrate the value of performance measurement and to build the competencies of the performance measurement team. After a successful three-month pilot, Phase 2 was started. The second phase lasted six months and concentrated on high-priority informatics projects involving AMS. Phase 3, currently underway, is addressing the remaining parts of the AMS organization. The activities required to implement the AMS Performance Measurement System included: Performance Measurement Profile (PMP) A PMP was developed from the PMA to assess the degree to which existing operational processes needed to be developed or adapted in order to implement the PMS. The profile also identified areas where information and processes concerning the performance measures and related performance indicators existed, and where information and processes needed to be adapted or created (see Appendix). Further, it identified the availability of baseline and benchmark information. Implementation Strategy An implementation strategy was developed for each of the performance measurement perspectives: financial quality of service client satisfaction employee satisfaction continuous improvement. These strategies provide the direction to guide the implementation effort by defining the roles and responsibilities of participants, the frequency of reporting and the data collection methods. They also outline the methods for analyzing, reporting and interpreting the information, as well as for identifying any issues that could impede progress for each measurement perspective. Business Processes Various day-to-day, month-tomonth business processes were developed or adapted to support data gathering for each of the performance measurement perspectives. Optimum, The Journal of Public Sector Management Vol. 29, No. 1 19

TABLE 1 AMS Performance Measurement Architecture VISION To be the supplier of choice to government departments for developing, supporting and integrating common administrative applications and infrastructure, and to be acknowledged by partners and clients as an employer of competent professionals who deliver measurable efficiencies through the management of information and technology. Financial Perspective Quality Service Perspective Customer Satisfaction Perspective Innovation and Learning Perspective Objectives Ensure cost effectiveness of application services On time delivery of quality application management services at lowest cost to client Achieve high level of customer satisfaction with application management services Create an organization of highly competent and dedicated staff Establish a corporate culture of continuous learning Performance Measures Revenues Expenditures IT investment Contracting Funding ratios Project Management Index Client Satisfaction Index Employee Satisfaction Index Continuous Improvement Index Performance Indicators Actual vs. planned revenues Actual vs. planned expenditures Variance between revenues and expenditures IT investment % of revenue IT investment per employee Competitive vs. noncompetitive contracts Ratio of O&M to FTE * expenditures Ratio of salaries to overtime Project Management Index: projects on time projects on budget projects within scope functional quality technical quality issue management Client Satisfaction Index client opinions on: products/services staff/consultants communication overall satisfaction Employee Satisfaction Index employee opinions on: goals/objectives role clarity decision making/ communication team building staff utilization rewards/recognition overall evaluation Continuous Improvement Index: staffing professional development suggestions for improvement staff rewards/ recognition * FTE: Full-time equivalent. 20 Optimum, The Journal of Public Sector Management Vol. 29, No. 1

Information Capture Once business processes were adapted or created for each measurement perspective, the information generated from these processes was captured according to established reporting formats for each of the performance measurement perspectives (e.g., monthly status reports, procurement activity reports and action logs). the methodology for developing performance measurement architectures; the means to establish baseline information to compare future performance; methods for collecting, analyzing and interpreting information; Interpretation This performance measurement information was interpreted to ensure internal validity and to provide an comparative external view. The interpretation of the information was accomplished in several ways: rating systems, baseline information, benchmarks, experience within the organization and the use of multiple lines of evidence. Reporting The monthly performance information relates to financial performance and the quality of services. A full performance measurement report, the Dashboard, is published quarterly. Both monthly and quarterly reports discuss the organizational mission from various points of view. Cumulatively, this information provides a causal link between the goals of the mission and any strategic, tactical and operational issues interferring with their attainment. When issues are raised, a briefing note is prepared and presented to management in advance for discussion at the monthly review (see Appendix). Communication Communicating results to senior management and staff is imperative so that changes can be made to keep the organization on course towards its mission, to recognize and reward the efforts of individuals and teams, and to encourage continued positive results. Benefits The benefits of the AMS performance measurement project must be assessed from several points of view. A discussion of the salient perspectives appears below. Knowledge transfer There was a considerable transfer of knowledge from the subject matter experts to AMS staff. Some key areas of knowledge transfer included: ways of developing or adapting business processes to generate performance information related to all measurement perspectives; and the use of facilitation techniques and Straw Dog Models to increase individual and organizational support for the PMS. As a result, AMS is relatively self-sufficient in knowledge and is self-sustaining in its ability to operate and update the PMS. Organizational culture AMS was successful in developing an organizational culture that values and supports balanced and comprehensive feedback as an essential element in examining issues and providing the information necessary for effective decision making. By improving communication, a common language and understanding developed in AMS concerning priorities, constraints and opportunities, as well as problems related to finances, quality of programs and services, client and employee satisfaction and continuous improvement. The PMS fostered a higher level of motivation and morale among staff through a greater appreciation of the issues and a greater involvement in formulating strategies and plans to resolve them. It is now viewed as a necessary and valued component of the management regime in AMS and has brought discipline to many of the sector s business processes. Decision making By providing AMS managers with multi-dimensional sources of information, the PMS provides a framework for decision making on cost effectiveness; improving, modifying or continuing programs or services; and improving client and employee satisfaction. It also provides information on program and service enhancements arising from Optimum, The Journal of Public Sector Management Vol. 29, No. 1 21

changing client, employee or technology needs or innovations. Moreover, it provides a means of translating the AMS mission into concrete objectives, strategies and plans, which can be monitored and adjusted in response to ongoing environmental changes. Lastly, it provides a frame of reference for management and staff, working together, to sustain and enhance excellence in program and service delivery to their clients. Information quality The use of indices magnifies the complexity of information generated about any of the measurement perspectives in the PMS. The PMA for AMS includes four indices: Project Management Client Satisfaction Employee Satisfaction Continuous Improvement. The use of these indices allows qualitative and quantitative information to be combined, yet provides the ability to quantify both so that all measures become output oriented. For example, the Project Management Index provides an assessment of qualitative and quantitative measures of high-priority projects used to deliver services to clients. Quantitative measures used include whether the project is on time, within budget, within scope, and whether it meets all functional and technical quality requirements. Qualitative measures include the effective use of estimation, risk management, methodologies and tools, the quality of user involvement, and the effective use of staff and consultants. These dimensions of project management were expanded or decreased in response to changing conditions or experience. By using an index, these diverse measures were reduced to a common overall score representing all dimensions of project management. The use of indices in AMS provides a powerful and flexible tool to provide comprehensive information on the areas mentioned. Interpretation of the information Several techniques are used to increase the rigour with which the information provided by the AMS Performance Measurement System is interpreted. Rating systems, baseline information, benchmarks, and organizational knowledge and information from multiple lines of evidence are used to ensure that the issues revealed by the PMS are interpreted to ensure internal validity with a comparative external perspective. Identification of best practices The use of the PMS has also resulted in the identification of a number of best practices at both the project and sector level. By sharing these best practices, AMS has given employees the knowledge and skills necessary to deliver programs and services in a constantly changing IM/IT environment. Risks/Threats There were a number of risks inherent in the development and implementation of a PMS in AMS. Leadership Performance measurement requires the active support of management in communicating the rationale and benefits of the system, to help breakdown individual and organizational resistance to what may be perceived as a threatening project. Without the active and ongoing support of management in promoting the PMS and using the information to improve the functioning of the organization, the exercise would have been viewed as another form of unnecessary overhead. Even though there is strong support by the AMS executive for ongoing performance measurement, there is always the risk that insufficient time will be devoted to discussing and reviewing information at monthly and quarterly presentations. Resources Developing and implementing a successful PMS requires time, effort and money. It took just over a year to build the processes and framework for AMS. For most of that time, the assistance of consulting expertise was 22 Optimum, The Journal of Public Sector Management Vol. 29, No. 1

needed until the process became self-sustaining. In particular, the process required the support of an already overburdened staff. Organizational context The PMS was built at the sector level but fits within the planning and reporting framework (Performance Reporting and Accountability Structure) for GTIS and PWGSC. Many of the financial, human resource and other information systems did not produce the type of information required at the proper time or to the required level of quality. There was also a reliance on staff members outside the sector, operating in a virtual organization, to produce certain types of performance information. Together, these elements tended to restrict the ability of those implementing the PMS to produce reports with the frequency and quality desired to meet departmental business and financial cycles. Exposure By publishing monthly and quarterly performance measurement reports, AMS provides a complete picture of the financial, quality-of-service, client/employee-satisfaction and continuous-improvement issues that it has dealt with during that period. By providing greater exposure to the organization s strengths and weaknesses, AMS is identifying the issues and actions required to resolve them. While such exposure is beneficial to a proactive organization such as AMS, it could be detrimental when no action is taken, or when issues can only be resolved at the branch or departmental level. continues to provide insights into issues needed to steer AMS towards its mission. It also provides insights that can be applied to other organizations to assist in the smooth development and implementation of a balanced PMS, which is portable and highly applicable to other informatics organizations. To date, the insights and lessons learned indicate: The development of a balanced PMS must be tailored to fit the unique requirements of each organization in terms of specific measures, timing, sequence of activities and knowledge transfer. The use of pilots is necessary to provide evidence of the utility of performance measurement, to build acceptance and support, and to provide the experience needed to mould the PMS to the needs of the organization and its sub-organizations. The implementation of a balanced PMS requires the full support of management in terms of providing the leadership to communicate the necessity and importance of the approach and to supply adequate resources. The development of a PMA must begin at the top of the organization and be implemented downwards. The development and implementation of a PMS must avoid expensive data gathering and implementation approaches. Use a Just Do It mentality with existing sources of information, work around strategies, and available methods for interpreting and maintaining progress. Once established, the performance measurement process tends to be self-correcting and self-sustaining. Conclusion The AMS performance measurement system is now well launched and is developing at an acceptable pace. It Most importantly, the performance information, along with the associated analysis, must be used to take corrective actions in the form of strategies and plans to deal with issues interfering with the achievement of an organization s mission. Endnotes 1. The first article in this series, Improved performance measurement: a prerequisite for better service delivery, appeared in Optimum, Vol. 27, No. 4, pp. 1-5. The second article, Implementing a performance measurement system in a public service informatics function, appeared in Optimum, Vol. 28, No. 3, pp. 36-44. 2. R.S. Kaplan and D.P. Norton. The Balanced Scorecard: Translating Strategy into Action (Boston: Harvard Business School Press, 1996). 3. These numbers represent the total where multiple selections were available and are not intended to be a cumulative total. Optimum, The Journal of Public Sector Management Vol. 29, No. 1 23

Appendix: AMS Performance Measurement Profile and AMS Issue Briefing Note 24 Optimum, The Journal of Public Sector Management Vol. 29, No. 1