Evaluating Your Technology Integration Initiative: Pointers for Success

Transcription

1 Intel Education Research Global research that supports education transformation Evaluating Your Technology Integration Initiative: Pointers for Success This memo is intended as a resource for leaders of large scale technology integration initiatives. It offers rationale and guidelines for conducting evaluations of such initiatives, providing examples of the types of evidence that can be used to improve the initiative and demonstrate its value and success. Why Conduct Evaluation? A well-conducted evaluation can help determine if your technology initiative is achieving its goals and ensure that it remains on track to do so. Evaluation is a systematic process of collecting, analyzing, and using information to answer basic questions about a program; it is the application of evidence to determine if an effort is on track and producing the results that policy-makers, funders, developers, or other stakeholders envision for the program. Everyone involved in a new program or initiative has a stake in it. Policy makers and government officials want to serve the good of the public. Program designers and champions want to show the value of their efforts. Funders want to know if their investments are worthwhile. Participants wonder about the effect an initiative will have on them, their peers, or their communities. All stakeholders want to see evidence that the initiative is making a positive impact.

2 2 GUIDING elearning FROM VISION TO PRACTICE For technology integration initiatives, evaluation can support positive outcomes in at least three important ways: 1. Defining clear goals for the technology integration. Evaluation requires clear and explicit statement of the goals for a technology integration initiative, encouraging discussion that moves beyond general aspirations to specify concrete outcomes. While any technology integration can have multiple outcomes, evaluation helps prioritize and target those that are the most desirable. 2. Identifying important milestones and elements for success. Evaluation helps stakeholders better understand how to get to intended outcomes by specifying elements needed along the way and identifying indicators of short-, medium-, and long-term success. If improving local capacity for entrepreneurship is a high priority, for example, an evaluation might focus on whether appropriate software tools are made available to participants, how these tools get used, and what new businesses have begun to use them. 3. Supporting ongoing improvements to the initiative. Perhaps the most important use of evaluation is systematically learning from what has been done in order to refine, strengthen, and plan for next phases of the initiative. Evaluation is the backbone of an ongoing process of examination that gives project leaders the data, analysis, and other information they need to make decisions for the future. In sum, using specialized research processes, evaluation provides the informational resources necessary for determining the effectiveness of efforts to date and for making the future of that initiative the best it can be for all the stakeholders involved. Applied to technology integration initiatives, evaluation can be essential for safeguarding the value of an elearning investment. GOALS OR PURPOSE OF YOUR INITIATIVE Input or Resources Activities Outputs Impacts What resources will help achieve your initiative goals? What will you do with your resources to achieve your goals? What are the direct results of those activities? What are the effects, results, or outcomes of having taken action short-term? mid-term? long-term? CONTEXT OR CONDITIONS OF YOUR INTIATIVE 2

3 GUIDING elearning FROM VISION TO PRACTICE 3 Engaging in the Evaluation Process A good evaluation is objective, replicable, and evidence-based. It requires careful design and execution, using data collection instruments that have been field-tested and rigorous methods of analysis. Because of these standards, many evaluations rely on the qualifications of a professional evaluator researchers specially trained in designing and conducting studies focused on particular programs or initiatives. Even though they are independent and strive to be objective, evaluators work in close collaboration with program development and implementation leaders, understanding the nuances of what program leaders are seeking to accomplish and how they are trying to do so. The evaluation process entails modeling the logic or change theory of an initiative: what are the initial goals or needs, what resources will be applied, what activities will be undertaken, and what will be the outputs and impacts of the effort. Evaluation can happen at different stages in the development of an initiative and generally can be categorized into two main types: Formative evaluation: Focuses on processes or activities associated with program development and implementation. Addresses such issues as how well a program model is implemented, whether an intended target population is being reached, and what are an initiative s major challenges and successful strategies. Summative evaluation: Focuses on determining outcomes and overall program effectiveness. Measures if and to what extent the expected changes occur and whether these changes can be attributed to the initiative. Both formative and summative evaluations provide essential information about the characteristics and qualities of an initiative, depending on the phase of its development and the purposes intended. Formative evaluation is generally oriented towards immediate improvement as the program is evolving. The technology deployment and integration research supported by the Intel Guide to Monitoring elearning Programs can be viewed as a precursor to formative evaluation because this research focuses on documenting the conditions in which technology deployments take place and deriving from this documentation a set of key considerations for success. The considerations identified constitute a framework that can be useful for helping project leaders and other stakeholders know in broad strokes what needs to be in place for their technology integration to succeed across phases for the visioning, planning, implementation, and re-informing vision of the initiative. While the Common Framework points the direction for technology integration, an evaluation focuses on the actual path an integration effort follows. Evaluations measure indicators of progress towards short-, medium-, and long-term implementation milestones and program outcomes. The evidence collected leads to judgments about the how the initiative unfolds, recommendations regarding course correction, and assessments of a program s quality or worth.

4 4 GUIDING elearning FROM VISION TO PRACTICE Educational Technology Deployments Educational technology deployments vary on a number of characteristics, including goals, geographic scope, leadership and participation models, and initiative components. The goals for technology deployments fall into three broad categories: Educational improvement, including the development of 21st century skills Social equity, focused on increasing digital access and participation Economic development, facilitated by new technological capacity and infrastructure Some initiatives cite goals in all three areas, while other focus more directly on one strategic area or another. One regional initiative has set explicit goals that equally emphasize educational improvement, social equity, and economic advancement. Other initiatives have goals in all three areas, but emphasize social equity most strongly for example, through reducing the digital literacy gap and involving more families in education. In other cases, initiatives focus most heavily on economic development, citing goals for teaching and learning as medium-term benefits that can serve to promote workforce readiness and integration into the 21st century global economy. Another regional program is targeted at ubiquitous learning, where students can engage in individualized learning, anytime and anywhere, using personal mobile devices. The goals of each initiative have specific implications for its implementation plan, the types of activities it will involve, and the short-, medium-, and long-term indicators that would be useful for evaluating the program. In each case, any metrics of success developed for the evaluation should match the goals and activities of the program and the local implementation context. A program focused on digital inclusion might include family broadband connections and Internet use among their indicators for success. A ubiquitous learning initiative might include indicators pertaining to the development of the public educational service platform and its use by administrators, teachers, and students. The following section provides suggested indicators for elearning initiatives to consider across a variety of domains (educational adoption, implementation, and change; economic change; and social change) for each of three time-points (short-, medium-, and long-term). The indicators listed here are framed in general terms, making them fairly applicable for a wide range of technology integration deployments. Program leaders would need to take the indicators a step further, defining the specific metrics and qualities for success for their own particular initiative.

5 GUIDING elearning FROM VISION TO PRACTICE 5 Suggested Indicators for Evaluating Technology Integration Initiatives At the heart of evaluation research is clarity on the types of evidence that will show the progress or success of the initiative. Indicators answer questions such as: How will we know if the initiative has been successful? What does success mean for this technology integration? What will have changed, and how quickly do we expect this change to take place? Sometimes an indicator will be a countable number (e.g., teachers using technology, new jobs created) and sometimes an indicator will require inferential measurement (e.g., improvement in attitudes, knowledge, or skills). Indicators are always particular to the goals of the project, and for each type of technology integration educational, social, and economic indicators for success will vary. Although indicators should be specific to the types of initiatives being created, we provide examples here of (1) implementation or process indicators that can be applied to any type of evaluation (broken down into short-, medium-, and long-term) and (2) outcome indicators for the types of changes that would be expected for educational, social, and economic initiatives. These indicators are relatively broad and generic not being tailored to any specific technology integration but, rather, intended to be adaptable to local, regional, or national initiatives depending on their type. Thus, they provide a starting point for developing the specific indicators a technology deployment evaluation will need in order to shape a program and appropriately determine its success.

6 6 GUIDING elearning FROM VISION TO PRACTICE Implementation indicators: Short-, medium-, and long-term Because evaluative criteria for implementation quality change over the course of an initiative s development, it is helpful to consider short-, medium-, and long-term implementation indicators separately. Short-term indication that a technology initiative has started on course typically would include evidence of an alignment of vision across stakeholders, commitment of funding to the project, development of an infrastructure for deploying technology, training and professional development for participating staff, establishment of communication systems for responsible parties, and undertaking of formative evaluation for continuous improvement. Table 1. Short-Term Indicators for Program Implementation INDICATOR OPERATIONALIZATION Alignment of vision among stakeholders horizontally and vertically Alignment with other policies Commitment of funding Infrastructure and infrastructural support Professional development (PD) workshops and trainings Systems for communicating among stakeholders Formative evaluation Stated expectations by multiple stakeholders for what gains will be had by implementing technology solution. Explicit descriptions and implicit intentions associated with technology integration in relation to other educational, social, or economic goals and strategies. Legislation, other governmental or non-governmental public policy statements that allocate funding to this initiative. State and quality of infrastructure (including wiring, wireless hubs for internet access, tech support and troubleshooting capacity); plans for improving infrastructure; fiscal commitments to same. State and quality of PD, according to needs; plans for adding or increasing PD; fiscal commitments to same. Regular and varied means by which diverse stakeholders are able to discuss progress and course correct as appropriate; quality of interaction; plans for and commitment to optimizing communication. Review of process and activities; reflection on these, communication to stakeholders and shared problem solving.

7 GUIDING elearning FROM VISION TO PRACTICE 7 Mid-term indication that an initiative is staying on course includes extensions of several of the short-term indicators e.g., infrastructure becomes well established, professionals develop communities of practice, research and evaluation efforts are fully integrated into program operations and improvement. Additionally, ownership spreads beyond the initial champions to those at the forefront of the effort. Program leaders and champions may create plans for scaling it beyond its current scope. Table 2. Mid-Term Indicators for Program Implementation INDICATOR OPERATIONALIZATION Well-established infrastructure High-quality infrastructure in place; dependable in areas of deployment; routing systems for maintaining and updating infrastructure. Professional communities of practice Systems for articulating research and practice Transition of ownership to implementing agents Initial scale-up and preparation for long-term scale-up Monitoring evaluation Existence of learning community for professionals working in the deployment (e.g., teachers, community workers); wide participation, self-organizing and sustaining. Coordinated efforts between research and practitioners for ongoing study of practice and integration of findings into next iterations. Those at the forefront of implementation, such as teachers, principals, and school and district staff, champion the project, work actively to support its goals, and take responsibility for solving problems that arise. Existence of plans and actions to support scaling program with focus on essential elements and strengths of program. Ongoing attention to process and activities, reflection on these, communication to stakeholders and shared problem solving. Long-term indication that an initiative is well-implemented includes evidence that the initiative has kept up with the times, maintained its support, been reinvented or scaled up as appropriate, and met its intended implementation goals. Table 3. Long-Term Indicators for Program Implementation INDICATOR OPERATIONALIZATION Reassessing goals and reshaping approaches Keeping up with technical evolution Maintaining political will and commitment to program Broader scale-up Summative evaluation implementation Capacity and willingness to change lines of action and program structure in line with new contingencies. Upgrading and replacing hardware and infrastructural systems as appropriate for program goals. Ongoing policy support and resource allocation for technology integration efforts. Efforts to expand the program to reach greater numbers, focusing on essential elements and strengths of program. Determination of effectiveness of program in reaching its goals for implementation.

8 8 GUIDING elearning FROM VISION TO PRACTICE Outcome Indicators: Educational, social, and economic change Because technology integration initiatives differ in goals, their outcome indicators should differ as well. Educational change ultimately centers on changes in teaching and learning. Such change involves improved capacities and practices for teachers, new types of classroom activities and curricula, student achievement that reflects 21st century skills, and the kinds of societal impacts that come from the development of greater human resources. The indicators for educational change listed below cover a span of more proximal and distal goals. Table 4. Indicators for Educational Change INDICATOR OPERATIONALIZATION Teacher capacity and practice Development and implementation of classroom approaches that make high-quality use of technology in pedagogically sophisticated ways. Classroom activities and curricula Student engagement Proximate student outcomes Longitudinal student outcomes Societal impacts Summative evaluation outcomes Types of assignments and activities that students engage in, especially more projectbased approaches, open-ended problems, and tasks requiring productive use of technology. Increased interest, attention, focus, time on task, and attendance. Improved grades, test scores, and ability to complete complex assignments; production of content; capacity to collaborate, use technology, think critically. Readiness for higher education; marketable job skills; capacity to learn and act flexibly in the 21st century knowledge-economy. Effects of having systems, teacher outcomes, and student outcomes in place. Determination of effectiveness of program in reaching its goals for outcomes and impact. The role of technology integration in social change centers on the capacity of participants to use technology as a tool to serve their own purposes. Most fundamentally, a deployment supports social change by providing access to and familiarity with technological tools. The strength in such a deployment comes when participants use technology for a diversified range of applications that provide them new opportunities for managing their affairs, scheduling activities, communicating with others, advancing their education, engaging in active citizenship, and so forth. The more control participants have over technological tools, the greater their sense of empowerment. Table 5. Indicators for Social Change INDICATOR Local technology access Family technology usage Range of technology uses by family Active citizenship and community involvement Sense of empowerment Technical capacity OPERATIONALIZATION Internet/broadband subscriptions. Range of family members using technology, and frequency/extent of use and comfort-level with technology. Extent to which family members use technology for a diverse set of activities, including leisure, following news, health tasks, financial tasks, continuing education, employment-related activities, running a business, etc. Family connections to community, participation in events, involvement in civic processes, etc. Family perspectives about how the computer impacts their lives, including via access to mainstream social and economic capital, not feeling disadvantaged with regard to others, etc. Capacity to resolve technical problems and/or connections to get issues resolved.

9 GUIDING elearning FROM VISION TO PRACTICE 9 Economic indicators provide information on changes in an economy s strength relative to perceived needs or demand, an earlier different point in time, a different geographical region, or some other comparative factor. The following indicators include both economic outputs (number of jobs, increased consumption) and economic capacity (meeting consumer needs, business partnerships). Table 6. Indicators for Economic Change INDICATOR OPERATIONALIZATION Number of jobs created associated directly with the deployment; number of jobs Job creation created as an indirect result of the deployment (due to higher demand, more skilled workers, etc.). Wage differential Revenue generation Business capabilities Business linkages Improved balance of trade (export increase) Increased foreign investment due to stronger workforce ICT industry wages as compared with national average; increase in average wages associated with deployment. Increased consumption of goods and services, including technology, Internet connections, etc.; increased revenue resulting from increased investment. Expansion of capabilities provided by companies to meet consumer needs, e.g., expanded capabilities in software/content to deliver fully-loaded classmate PCs. Partnerships developed within and across businesses, both locally and internationally. Increase in exports resulting from increased capacity and business investments. Number of multinationals investing locally, monetary value of said investments. Data Collection, Analysis, and Reporting In addition to establishing a framework and identifying the key indicators, other major steps in an evaluation effort include data collection, analysis, and reporting. Data collection typically includes a combination of interviews, surveys, and observations, as well as review of policy and project documentation where available. Interviews, surveys, and observation protocols designed for the specific initiative often are more valuable than general tools. Existing paper-and-pencil assessments of attitudes or knowledge that might be affected by a program and other pre-existing questionnaires can also be an important part of a suite of evaluation instruments. A challenge that Intel and other organizations are currently addressing is the development of appropriate assessments of 21st century skills, such as collaboration, computational thinking, and problem solving. Particular types of artifacts may be used in an evaluation as well. For example, for an education-related rollout, teachers plans and assignments, student work products and attendance and participation records can be an important part of an evaluation. Artifacts that might be collected, with permission, to evaluate a social equity initiative could include records of online transactions, communication, and Internet searches that reflect the value of technology to families newly online; this type of information can be obtained more or less reliably through interviews as well. Overall, data collection strategies depend on a balance of research considerations, resources, and practicalities for gathering information that matches appropriate indicators for the initiative. The way the data are analyzed depends largely on what the status or a change in status for an indicator means for the initiative for example, what it means if adequate technological infrastructure is (or isn t) in place, if there is (or isn t) a good system of communication established among stakeholders, or if monitoring evaluation is (or isn t) undertaken. A research team typically must determine the starting state or baseline for the set of indicators being used in the evaluation, document the activities or intervention intended to produce the desired result, and measure changes that happen over time. This trajectory from starting point, through implementation, to outcomes defines the program logic model or change theory.

10 10 GUIDING elearning FROM VISION TO PRACTICE Reporting findings based on a logic model may be relatively straight-forward. For example, a community that has few small businesses may attract many adults to a technology course teaching the essentials of entrepreneurship, and these same adults may, in proportionally high numbers, launch new small businesses shortly after the end of the course using the materials and techniques taught in the course. In this case, small business initiation rate is a measurable indicator that ties directly to program activities. Other indicators may be harder to measure at baseline and over time. For example, when program leaders engage in a process of reshaping goals and approaches based on experience to date, this may be an indicator that the program is likely to be sustained. But what does it look like when program leaders reassess goals and reshape approaches for technology integration? It also may be challenging to describe what factors promote reassessment of goals and the implications of having it to greater or lesser degree. In reporting their findings, evaluators must be able to discuss the relationships of key indicators to one another and to the goals of the program. The level of detail in such discussions, the language used, and the emphases of the report will depend in large part on the audience for a report and its purposes. A report designed to help educational administrators and practitioners better understand how to overcome implementation challenges will differ from one designed to help government agencies make funding decisions for the future. A report designed to engage the academic community will be different from one designed to demonstrate the benefits of the initiative to support replication or scaling. In sum, evaluators must be able to collect and analyze appropriate data, speaking to the implications of what they find. They must be able to analyze each key indicator by assessing: The baseline status of that indicator: What is in place at the start of the initiative? Change over time from baseline status to the current status: How does what is in place now compare with what existed or what was happening previously? Implications of having or not having the key indicators in place as described: What impact does the current situation have for project participants? Correlation between a given indicator and other relevant factors: How is this outcome affected by other conditions or activities that may be taking place? Evaluators must be able to able to use these analyses to present the most relevant evidence to their audience in the way that is most appropriate for their purposes.

11 GUIDING elearning FROM VISION TO PRACTICE 11 Conclusions This intent of this memo has been to share some of the compelling reasons for evaluating your technology integration initiatives. It has also provided possible indicators for different types of initiatives and set expectations for what is involved in data collection, analysis, and reporting. The memo does not offer a design for any particular evaluation, detailed ways of specifying the precise indicators for your initiative, or a set of tools or approaches to designing data collection instruments. These types of evaluation supports involve more extensive processes and guidelines than can be represented in a short memo. This memo can be a useful resource, however, for motivating early reflection on the key milestones for your initiative, the most meaningful indicators for your context, and the relationships between the types of indicators being tracked. Intel welcomes the opportunity to discuss your evaluation plans with you and to help direct you to additional resources that can help you meet your technology integration goals. Intel has helped to implement more than 200 education programs in over 70 countries, and has invested more than USD 1 billion in the last decade to improve teaching and learning environments. Working with governments, policy makers and local vendors, Intel helps to implement elearning solutions that provide professional development to teachers; Leadership Policy support student achievement and Professional development of 21st-century skills; Development and enable access to relevant, Research & Curriculum & Evaluation Assessment Education Transformation localized digital content. The education transformation model developed by Intel helps governments improve the quality of their education systems, leading to economic and social opportunities for their citizens. Sustainable Resourcing Information Communications Technology

12 Looking for a Total Education Solution? Intel Education Solutions Intel Education Solutions delivers total education solutions that enable a comprehensive 1:1 elearning experience. Intel Education Solutions combines purpose-built hardware, software, digital content, services, and support delivered by local vendors to meet local needs and designed to work reliably together. Drawing on ethnographic and field research, Intel Education Solutions help foster 21st century skills, such as digital literacy, information synthesis, critical thinking, creativity, and problem solving. Ready to Move Forward on Your Education Technology Program? Intel Education Technology Advisor Did you know that Intel provides a free online resource to help Educators and education IT Managers make the right technology decisions for their schools? Intel Education Technology Advisor features online content, tools and personal live assistance all designed to help schools assess their technology challenges, select a solution that meets their needs, and then deploy it effectively into their school. See for yourself at Copyright 2013 Intel Corporation. All rights reserved. Intel and the Intel logo trademarks of Intel Corporation in the U.S. and other countries. *Other names and brands may be claimed as the property of others. Printed in USA 0514/LP/PDF Please Recycle