Fifth Conference of the Latin America and the Caribbean Monitoring and Evaluation Network

Transcription

1 United States M&E Trends: Using Performance Information to Improve Results Fifth Conference of the Latin America and the Caribbean Monitoring and Evaluation Network John R. Pfeiffer U.S. Office of Management and Budget Bogota, Colombia, November 6,

2 A President Committed to Results The question we ask today is not whether our government is too big or too small, but whether it works whether it helps families find jobs at a decent wage, care they can afford, a retirement that is dignified. Where the answer is yes, we intend to move forward. Where the answer is no, programs will end. And those of us who manage the public s dollars will be held to account to spend wisely, reform bad habits, and do our business in the light of day because only then can we restore the vital trust between a people and their government. President Obama January 20,

3 Building on the past Congress and previous Administrations laid important groundwork for government-wide performance improvement, including GPRA In 1993, Congress passed the Government Performance and Results Act. GPRA requires every federal agency to set strategic and annual goals (dealing with societal outcomes, not just agency activities), measure performance, and report to Congress, OMB, and the public on progress relative to selected goals. PART The Bush Administration created the Program Assessment Rating Tool (PART ) to give true effect to the spirit as well as the letter of GPRA. PART shifted the focus of goal-setting and measurement from the agency level to the program level. 3

4 Obama Administration critique of GPRA and PART: OMB Deputy Director for Management Jeff Zients believes that too much emphasis was placed on producing performance reports for their own sake, and too little attention paid to analyzing and acting on this information Zients wants to use Federal performance information: to communicate priorities, progress, and raise issues; to illuminate what works that should be continued and what does not work that needs attention; to motivate the best from our workforce and our service delivery partners; and to allocate scarce resources wisely. 4

5 Shelley Metzenbaum, OMB s new Associate Director for Performance Management, in a study prepared before the last election found that GPRA and PART offered no comprehensive way to see how the federal government is performing and what agency goals and program targets are. Despite reams of performance GPRA and PART material, it is difficult to find meaningful government performance information performance levels, performance trends, and even targets because: too little attention has been paid to communicating targets and trends and too much to communicating the percentage of targets met as the primary indicator of overall performance. 5

6 Too little attention was paid to understanding the size and characteristics of problems and why performance levels changed. Few agencies and programs routinely analyze their performance and other data, as businesses do, to generate insights needed to improve performance. The program review process has been overly subjective, creating unfair inconsistencies and frustrating disputes about what constitutes appropriate measures, targets, and evaluation methods. Too little attention has been paid to identifying key audiences for goals and performance data, to determining their performance information needs, and to delivering information where and when it is needed in a format the target audiences can understand. As a consequence, key audiences, including Congress, field offices, delivery partners, and others, have not gotten the performance information they need.. 6

7 Too much attention has been given to program review, assessment, and control, and too little to providing expert advice and to stimulating innovation, discovery, cooperation, and assistance 7

8 Guiding Principles to Improve Federal Performance Management Metzenbaum concluded that the key performance management challenge is to use not just produce performance goals and measures. Four guiding principles should undergird changes to current federal performance management efforts: Communicate performance trends and targets, not target attainment and ratings Encourage performance improvement with increased diagnostic analysis, practical experiments, and knowledge sharing Present information in ways that meet the needs of specific audiences Structure accountability mechanisms to encourage and inspire, not embarrass, reprimand, or punish 8

9 Recommendations for the President 1. Clearly Identify Presidential Priority Targets. Meet at least quarterly with each responsible Cabinet secretary to keep agencies focused on the targets. 2. Appoint a Chief Performance Officer and create a White House Performance Unit working closely with OMB to advance progress on Presidential priorities. 3. Run Goal-Focused, Data-Driven Meetings pertaining to the priority targets. 4. Increase Analysis of performance and other relevant data pertaining to presidential, cross-agency, agency, and program targets. 9

10 5. Appoint Experienced Performance Managers to key government management positions, and recruit experts to provide counsel to Cabinet secretaries. 6. Identify and Manage Cross-Agency Targets and Measures where performance improvement needs cross-agency attention and cooperation. 7. Adjust Accountability Expectations. The President should hold agencies accountable for the persistent application of evidence, intelligence, and effort to achieve continual performance gains. 10

11 Recommendations for the Office of Management and Budget 8. Direct agencies and programs to communicate targets and performance trends for key indicators 9. Create a Federal Performance Portal with easy to find performance targets, trends, and other related information. 10. Encourage outside expertise and multiple perspectives on selection of targets, performance measures, and strategies to improve performance. 11. Build communities of practice to facilitate cross-agency learning. 12. Increase training in performance management practice and analysis. 11

12 13. Revise, but continue PART, shifting the emphasis from program rating to performance improvement. Remove the centrality of rating and assessment, and shift attention to performance improvement and communication, not target attainment for its own sake. Eliminate barriers that complicate audience-focused delivery of performance reports and information. 14. Continue the President s Management Council as a forum for senior agency deputies to meet regularly to discuss progress toward performance and management priorities and to reduce management risks. 12

13 State and local government models OMB s performance managers are particularly impressed by successful efforts at the State and local government level to use performance information to monitor and evaluate government programs. Local governments,including New York City, New York; Charlotte, North Carolina; Baltimore, Maryland; King County, Washington; and Austin, Texas have demonstrated how performance information can be used to improve outcomes, reduce crime, increase housing starts, and drive down costs. 13

14 Planning for the FY 2011 Budget and Performance Plans OMB intends to develop an improved performance management framework aligning high priority performance goals, GPRA performance reporting, and many of program-level PART measures. It will focus on outcomes, allow comparisons across programs and agencies, and show trends over time. It will focus on information supporting long-term decisions about targets and organizational strategies, as well as key decision-makers in the service delivery chain including those in Federal field offices, State and local partners, non-profit grantees, and contractors. It will use new information technologies to make this more feasible, less cumbersome, and far more useful than past alignment efforts. In addition, the Administration is proposing historic investments in comparative effectiveness research and evaluations, integrated with performance management efforts. 14

15 Increased Emphasis on Program Evaluations In June, OMB Director Peter Orszag wrote in his regular blog on the OMB website that we haven t been making the right investments to build a new foundation for economic prosperity and we need smarter investments in education, health care, and social services. Much of the time, it s hard to say whether a program is working well or not. Many initiatives drive funds to the local level, but don t track how they are spent; others spend dollars bit by bit, so the results are hard to see Too often rigorous evaluations don t happen. This has to change, and I am trying to put much more emphasis on evidence-based policy decisions here at OMB. Wherever possible, we should design new initiatives to build rigorous data about what works and then act on evidence that emerges expanding the approaches that work best, fine-tuning the ones that get mixed results, and shutting down those that are failing. 15

16 Beyond investing more in evaluations, we are also designing new initiatives with evaluation standards built in using a two-tiered approach providing more money to programs that generate results backed up by strong evidence. That s the top tier. Then, for programs, with some supportive evidence but not as much, we ve said: Let s try those too, but rigorously evaluate them and see whether they work. If not, we ll redirect their funds to, more promising efforts. This two-tiered structure will provide objective criteria to inform our decisions on which models to invest in. It will also create the right incentives for the future. Organizations will know that to be considered for funding, they must provide credible evaluation results that show promise, and be ready to subject their models to analysis. As more models move into the top tier, it will create pressure on all the top-tier models to improve their effectiveness, so they continue to receive support. By instilling a culture of learning into federal programs, we can build knowledge so that spending decisions are based not only on good intentions, but also on strong evidence that carefully targeted investments will produce results. 16

17 New evaluation initiative Most agencies review ongoing programs, but not robustly enough to provide meaningful data about outcomes and progress towards the objective. Many agencies lack the capacity to carry out a rigorous, strategic research agenda, and policy priorities are established without evidence to back them up. Programs are continued year after year, without a hard, objective look at their effectiveness. On October 7, OMB asked agencies to volunteer for an initiative designed to strengthen rigorous, objective assessments of existing federal activities to improve results and better inform funding decisions. Ongoing program evaluation research will be published online, and an interagency task force will identify and help to shape evaluations of programs that cross over several agencies. Participants will serve as demonstration projects through which we can test approaches to improve program effectiveness and efficiency, share best practices, and further improve performance. After assessing the initiative in FY2011, the Administration will implement government-wide evaluation metrics. 17

18 Additional features: On-line information about existing evaluations. OMB will work with agencies to make information readily available online about all Federal evaluations focused on program impacts that are planned or already underway. Inter-agency working group. Together with other White House offices, OMB will establish a new inter-agency working group to promote stronger evaluation across the Federal government. New evaluation funding. OMB will allocate a limited amount of funding for agencies which volunteer to: Show how their Fiscal Year 2011 funding priorities are evidencebased or otherwise subject to rigorous evaluation; Assess their own capacity to support evaluation and suggest pathways for strengthening that capacity; Propose new evaluations that could improve government programs in the future; and Identify impediments to rigorous program evaluation in their statutes or regulations. 18

19 Initial focus This initiative focuses on impact evaluations, or evaluations aimed at determining the causal effects of programs. Although the Administration is committed to improving a wide range of evaluation and performance measurement activities, this initiative will initially focus on social, educational, economic, and similar programs whose expenditures are aimed at improving life outcomes (such as improving health or increasing productivity) for individuals. While OMB would consider on a case-by-case basis the inclusion of evaluation efforts in other spheres, most activities related to procurement, construction, taxation, and national defense are beyond the initial scope of this initiative. In addition, because drug and clinical medical evaluations have independently received extensive discussions, they are also excluded. 19

20 Conclusion This is an exciting time for those interested in improving program evaluation in the Federal government and in seeing the results of those studies have a more direct impact on Federal decision-making. For the first time in memory, staff have been told that our annual budget reviews with the Director for all parts of the Federal budget, not just social programs, must give prominent attention to quantitative evidence of program performance and results. As another of our top managers said in a meeting on the program evaluation initiative this week, this is a very fluid and dynamic process, more developments are sure to come. 20