WHITE PAPER OUTSOURCING E-ASSESSMENT. Working with an outsourcing partner to deliver the full benefits of e-assessment

Transcription

1 WHITE PAPER OUTSOURCING E-ASSESSMENT Working with an outsourcing partner to deliver the full benefits of e-assessment

2 WHITE PAPER OUTSOURCING E-ASSESSMENT 02 CONTENTS Introduction 3 01 Why adopt e-assessment? 4 02 High stakes e-assessment 6 03 Why outsource? 8 Case Study 1: Financial services national awarding body 9 Case Study 2: Civil aviation national awarding body Considerations when outsourcing high stakes testing Successful outsourcing partnership Conclusions 15

3 WHITE PAPER OUTSOURCING E-ASSESSMENT 03 INTRODUCTION Implementing best practice e-assessment yields significant benefits to all stakeholders: national awarding bodies, candidates and industry organisations. The knowledge, processes and technology involved in achieving these benefits are complex and require significant investment of capital and time. Outsourcing parts of the e-assessment process gives a national awarding body the opportunity to leverage best practice and technology and achieve economies of scale. It also enables the organisation to maintain focus on its strategic goals, while retaining control of delivering high-quality assessments. Outsourcing is a well established process. Specialist vendor organisations can bring to bear what they have learnt through hard experience; this should lower the implementation risk. Cost saving has traditionally been the starting point for outsourcing. However, the emphasis in the last 10 years has shifted towards augmenting the organisation s strategic capabilities. It is important to analyse the total system cost when designing the e-assessment process. The immediate benefits to a national awarding body are most likely to be significant improvement in the quality of candidate assessment without large capital investment and increases in cost per assessment. There is no one size fits all solution. The answer lies in careful consideration of the national awarding body s objectives and strategic goals, along with crafting a solution that balances in-house and outsourced functions to meet the organisation s needs. ASPEQ has been working with national awarding bodies for 20 years developing and implementing e-assessment solutions on their behalf. The company has extensive experience of implementing both green-field e-assessments, and in implementing e-assessment of existing licences and qualifications. From this experience we have written this white paper to explain some key principles for organisations to consider when reviewing their processes and developing their future e-assessment road-map. Outsourcing leverages best practice, technology and economies of scale COPYRIGHT ASPEQ 2011 LTD 2012

4 WHITE PAPER OUTSOURCING E-ASSESSMENT WHY ADOPT E-ASSESSMENT? e-assessment has come a long way since the early 1960s. It is now a major tool for national awarding bodies to assess candidates to issue qualifications and licences to practice. Widely adopted in the USA, e-assessment has gained significant traction across the globe in the last two decades. Of the 115 recognised national awarding bodies in the UK, 38% are currently delivering some form of e-assessment. A further 20% are in the process of implementing, and only 15% think that their qualification is not transferable to an e-method 1. Driving this adoption are some widely recognised benefits to all the stakeholders in the assessment process 2 : STAKEHOLDER NATIONAL AWARDING BODY BENEFITS > > Higher quality testing: improved consistency and reliability of testing across candidate population > > Faster and more controlled test revision process > > Increased security (question randomisation, electronic transmission and encryption) > > Reduced costs through delivery, administration and scoring efficiency > > Fewer candidate response entry and comprehension errors > > Easy to adopt new and more advanced question types and future testing methodologies. CANDIDATE > > Unbiased test administration and marking > > Faster results process: immediate marking and reporting > > Improved feedback process and more detailed information > > Increased flexibility, both to test schedules and locations possible for delivery. TRAINING ORGANISATIONS (OR INDUSTRY ORGANISATIONS) > > Improved clarity around competency requirements > > Increased capacity and flexibility to schedule tests > > Increased candidate acceptance and satisfaction > > Improved training programme performance through effective testing feedback. The principal benefit for all is an improvement in the quality and reliability of assessment. It also offers the opportunity to optimise the operational cost of delivery. 1 Acceptance and usage of e-assessment for UK Awarding Bodies Thomson 2 E-assessment guidelines for the VET sector Department of Education, Employment and Workplace Relations, Australian Government

5 WHITE PAPER OUTSOURCING E-ASSESSMENT 05 A large body of academic work defines the design of best practice e-assessment A large body of academic work and best practice defines the design and delivery of successful e-assessment to realise these benefits. This work is evolving at an ever-increasing pace as technology enables the delivery of more complex tests and assessment methodologies. However, there are some enduring high-level principles that will always hold for successful e-assessment 3 : Processes that have demonstrable consistency and reliability Adherence to principles of fair assessment Assessment is configured to relevant legislation and regulatory authority s codes of practice High standards of security in conducting assessment Be able to demonstrate accountability for the quality of assessment to external agencies Leverage the flexibility offered by technology to evolve and respond to changing requirements. 3 Regulatory principles for e-assessment Joint Information and Systems Committee, UK

6 WHITE PAPER OUTSOURCING E-ASSESSMENT HIGH STAKES E-ASSESSMENT National awarding bodies face the challenge of applying these e-assessment principles for high stakes assessments. High stakes assessment is where there is a major consequence, or is the basis of a major decision and is characterised by: a single, defined assessment having a clear achievement requirement direct consequences for passing or failing (ie something is at stake). High stakes assessments are often statutory tests and/or have results collected and used for statutory reporting. In these situations the outcomes are important to the statutory body and candidates alike 4. A whole range of professional licence and certification assessments are high stakes, for example pilot licence tests where a candidate s legal ability to fly is at stake. Conceptually there are three components to a high stakes assessment system. Each component is made up of several constituents. FIGURE 1: COMPONENTS OF NATIONAL AWARDING BODY E-ASSESSMENT STATUTORY STANDARD MANAGEMENT INTERNATIONAL STANDARDS (LAWS & REGULATION) SKILLS AND JOB ANALYSIS FEEDBACK & DELIVERY NATIONAL AWARDING BODY STANDARDS ASSESSMENT GOALS Assessment Structure Learning objectives LICENCE AND QUALIFICATION REGISTRY INDUSTRY AND TRAINING ORGANISATIONS TEST ANALYSIS Learner behaviour Test results Item performance TEST CREATION & DELIVERY TEST PREPARATION Test Definition Item Authoring TEST GENERATION Candidate item selection TEST DELIVERY Controlled test centres CANDIDATE PORTAL Registration Book and pay for tests Candidate communication 4 e-assessment Glossary Joint Information and Systems Committee, UK

7 WHITE PAPER OUTSOURCING E-ASSESSMENT 07 Each of the three principal components has its own distinctive considerations and management challenges. Effectively managing each component requires different knowledge, skills, processes and technology. Statutory Standard Management At the core of this component is the understanding of the standard or syllabus itself and how it relates to any regulatory legislation or national/international framework. Also required is knowledge of the detail around professional certification or practice accreditation. For assessment purposes it is critical that the standard sets out the different levels of job or licence being assessed and this is communicated unambiguously to candidates, the industry and training organisations. The stakeholders must have an understanding of the assessment standard being implemented. Technology and how it is robustly implemented is a key enabler Test Creation and Delivery Test creation and delivery has a far more operational orientation; turning the standard or syllabus into specific assessments, authoring high quality questions and delivering assessments to candidates. Critical to these components is a detailed understanding of the relationship between assessment and pedagogy as well as how technology is integrated into the assessment processes. A framework must be developed to manage the components and their relationship with the technology to provide a successful outcome. Technology and how it is implemented is a key enabler: Efficiently co-ordinating the authoring, reviewing and testing of high quality questions by a large group of SMEs (subject matter experts) who are often not in the same place Generating large numbers of unique assessments for candidates while adhering to the principles of fair assessment and high-stakes security Managing the efficiency of test delivery through a network of supervised assessment centres. Balancing supply and demand, much as an airline booking process, to give candidates the flexibility to sit assessments where and when is convenient to them while maintaining operational effectiveness Give candidates a modern and intuitive self-service web portal that enables them to securely book, pay, and manage their assessments and all related communications (eg results notifications, assessment feedback, booking reminders or confirmations etc). Feedback and Delivery Largely reporting and analysis, this component closes the loop. It gives candidates specific feedback on their performance, and training organisations high level feedback so that they can modify their training in poor-performing areas. It also involves all aspects of continuously improving the assessment: monitoring performance for cheating, maintaining the item bank, evolving the syllabus or standard. The management challenge here is to draw on in-depth statistical review and performance analysis, and apply them to reviewing how the national awarding body s standards are implemented and trained.

8 WHITE PAPER OUTSOURCING E-ASSESSMENT WHY OUTSOURCE? With such a diverse range of activities in the endto-end national awarding body e-assessment system, it is clearly a significant management challenge for one organisation to costeffectively build all the skills, competency, tools and systems to effectively manage all the components. Outsourcing has been long established as a viable process by all types of organisations and government functions. Organisations outsource for many different reasons, the most common are: to reduce and control operating costs, improve the organisation s focus, gain access to world-class capabilities and to free internal resources for other purposes. Organisations objectives when outsourcing have evolved considerably from the 1990s where the primary focus was on cost reduction, taking advantage of labour arbitrage and benefiting from economies of scale 5. Organisations are now looking beyond cost reduction to augmenting their strategic capability. The survey results in Figure 2 shows decision makers responses when asked what their objectives were when outsourcing. Decision makers are looking at how it will enable their organisations to achieve their strategic goals. FIGURE 2: ORGANISATIONS OBJECTIVES WHEN OUTSOURCING Converting capital investment into expense Access to technology Centralisation and standardisation Unique experience Improved management focus Capabilities of speed Lower labour cost Handle capacity fluctuations Increased business discipline and transparency Increased revenue Primary Objective Secondary Objective Source: Moving beyond costs to competitive advantage Accenture Although the research shows some strong and consistent high level themes as to why organisations outsource, individual organisations objectives may be quite different, depending on their circumstances and strategic goals. The following two case studies set out why two very different New Zealand national authorising bodies outsourced parts of their high stakes e-assessment process. They also summarise how they went about it. 5 3rd generation Business process Outsourcing CapGemini

9 WHITE PAPER OUTSOURCING E-ASSESSMENT 09 CASE STUDY 1: FINANCIAL SERVICES NATIONAL AWARDING (NAB) The national authorising body is responsible for authorising all the country s financial advisors; without authorisation they cannot practice professionally. In 2010 the Government mandated a tightening of the authorisation process through implementing a new and more rigorous national competency-based system. The awarding body outsourced critical functions of the e-assessment process in order to: Build scale very quickly to meet very tight deadlines and assess very large number of candidates Take advantage of best practice e-assessment principles and technology in order to deliver reliable, high quality assessments cost-effectively. While the government was mandating the new competency-based authorisation process, individual candidates had to pay for the assessments themselves Enable limited internal resources to focus on designing the new standard and working with the industry to ensure its adoption, rather than on operational delivery of the e-assessment process. Outline of what was kept in-house and what was outsourced: COMPONENT IN-HOUSE OUTSOURCE STATUTORY STANDARD MANAGEMENT National awarding body standard > > Agreed standard with government & qualifications agency Skills Job Analysis > > Worked with industry bodies to define Assessment Goals > > Wrote and published to industry > > Integrated e-assessment best practice into new process Qualification Registry > > Integrated with national qualifications standards systems and processes TEST CREATION & DELIVERY Test Preparation > > Identified SMEs (subject matter experts) > > Employed SMEs and owned questions > > Trained SMEs to write good questions > > Managed workflow process > > Managed question quality > > Managed employment and payment of SMEs Test Generation > > Configured and managed process Test Delivery > > Exam centres, both permanent and mobile centres > > Managed registration and candidate ID process > > Operational support to manage high availability of assessments Candidate Portal > > Provided a self-service portal skinned to NAB s branding FEEDBACK > > Provided call centre help desk service Test Analysis > > Provided statistical analysis Industry Training Organisations > > Worked with industry training organisations > > Reviewed and modified standard > > Hosted feedback workshops and reports

10 WHITE PAPER OUTSOURCING E-ASSESSMENT 010 CASE STUDY 2: CIVIL AVIATION NATIONAL AWARDING BODY The Civil Aviation Authority is responsible for all the aviation safety within the country. Integral to that function is the management of professional pilot and engineer licensing. After detailed deliberation, the Authority decided to move to adopting a recognised international licensing standard, with modifications to reflect local conditions. In doing this they also wanted to adopt e-assessment best practices. The awarding body outsourced critical functions of the e-assessment process in order to: Quickly adopt an internationally recognised licensing standard Improve the quality and reliability of their candidate assessment in order to improve aviation safety Implement a best practice end-to-end e-assessment process without capital outlay. Outline of what was kept in-house and what was outsourced: COMPONENT IN-HOUSE OUTSOURCE STATUTORY STANDARD MANAGEMENT National Awarding body Standard > > Agreed standard with government Skills Job Analysis > > Adopted an internationally recognised licensing standard > > Modified to reflect local conditions > > Agreed the details behind the internationally recognised standard Assessment Goals > > Agreed and published > > Proposed and detailed Qualification Registry > > Maintained own registry TEST CREATION & DELIVERY Test Preparation > > Reviewed and approved items > > Leased items on a per-use basis > > Licensed own item bank for use by NAB > > Wrote local items to augment own question bank Test Generation > > Configured and managed process Test Delivery > > Used own test centres and invigilators > > Managed registration and candidate ID process > > Operational support to manage high availability of assessments Candidate Portal > > Provided call centre help desk service > > Provided a self-service portal skinned to NAB s branding FEEDBACK > > Provided operational support to ensure high availability Test Analysis > > Provided statistical analysis Industry Training Organisations > > Worked with industry training organisations > > Proposed changes to standard > > Maintained item bank

11 WHITE PAPER OUTSOURCING E-ASSESSMENT KEY CONSIDERATIONS WHEN OUTSOURCING HIGH STAKES TESTING ASPEQ has been partnering with National Awarding Authorities for over 20 years, across multiple sectors and in different countries, to deliver a wide range of e-assessment solutions. ASPEQ is an assessment specialist dedicated to certifying candidates for professional practice licences or qualifications. Below are some of the key considerations to take into account when a national awarding body is designing their e-assessment implementation road map. Organisations should consider the following things when deciding what elements of the e-assessment process to outsource: 1. Need for specialist process, people and technology Running high-stakes testing in the real world is a specialised operational process. It requires a distinctive blending of technology, process and people skills. Organisations rarely start from scratch with their e-assessments, they often have some basic processes that they are looking to build on or have a capable internal IT function that is willing to build new systems. Inevitably organisations massively underestimate the complexity of what is required to make e-assessment work properly in practice. As a consequence assessment quality is often compromised and costs escalate. 2. Assess the true costs from the outset Organisations should consider the true costs of any assessment system from the outset. This requires disciplined analysis and honest reflection. It is often easier to focus only on parts of the process and workload, assuming the rest will be done by existing resources. The longer term costs of ongoing monitoring and maintenance are also often forgotten when quantifying overall costs. One of the principle benefits of e-assessment to the national awarding body is improved assessment of candidates, both in terms of validity and reliability. This benefit should be incorporated into the cost analysis and given a monetary value. 3. IP Ownership, especially of questions Ownership discussions can span quite a few areas of systems and data. However, the most important area to consider is ownership of the questions. There are many different models of ownership ranging from outright leasing on a per-use basis to full ownership. Each model will suit a particular set of circumstances; carefully consider: Who pays for authoring the questions? Who manages the authoring process? Who is responsible for the quality of the questions? Who is responsible for maintaining the questions on an ongoing basis?

12 WHITE PAPER OUTSOURCING E-ASSESSMENT Understand what you want to outsource or maintain in-house Vendors that offer effective assessment systems will provide the total solution. However, depending on a national awarding body s circumstances and objectives, they may want choose to which parts of the solution they want to maintain control of by keeping them in-house. Organisations should also consider mechanisms to maintain control while gaining the benefits of outsourcing. For example: specific service level agreements, contract monitoring and review processes. 5. Future investment High stakes assessment is evolving and new approaches and advancements are emerging regularly to give better ways to understand candidate competence. Most of these are advancements are enabled by technology. A national awarding body should consider how they can keep abreast with advancements in order to best meet their objectives and ask themselves how the implementation of these advancements could be funded and implemented. When designing the e-assessment process, consider: 1. Assessment as an independent process High quality assessment of candidates can be achieved through an independent process that removes training bias and sets defined standards. Independence provides national awarding bodies with confidence that the assessments have been developed to a consistent standard in order to test candidates knowledge. 2. Assessment as part of the learning process In high stakes assessment, feedback to candidates and training organisations is an integral part of process. Candidates need to know where they went wrong so that they can better prepare next time, and training organisations need to understand what parts of the standards they need to train differently or better. 3. Incorporating assessment into the standard at an early design stage Often assessment is only considered after the standard has been written and agreed by all parties. Structuring and creating tests is complex and involves detailed design around the standard and specifics of its content. Bringing experience and understanding of e-assessment to the standard development process saves a lot of time, potential re-work and ultimately cost further down the line.

13 WHITE PAPER OUTSOURCING E-ASSESSMENT 013 When implementing e-assessment, remember that: 1. The quality of the assessment and questions is paramount The ability to create quality assessments and question banks that can accurately determine a candidate s knowledge or competence is often underestimated. Developing questions from technical subject knowledge is a difficult task without proven practices and principles to construct reliable and valid assessments. As many of these practices and principles are specific to assessment type rather than industry or sectors, it makes sense to leverage the large body of knowledge and learning used by a specialised assessment providers rather than reinventing the wheel. 2. Service level is the key to candidate experience High stakes assessment is stressful for candidates. Technology therefore has to be reliable and intuitive. The technology must work every time, as it is simply unacceptable to have technology problems that affect the candidate s test or performance. The candidate interfaces must be intuitive and easy to understand. Organisations must consider how candidates will interface with all stages of the assessment process, from booking through to getting results. The prime objective is to assess a candidate s level of knowledge on a subject, not whether they can work the technology! 3. Data security underpins reliable assessments it must be built into every stage of the process The integrity of the testing relies on using technology to generate reliable assessments while minimising any opportunity for cheating. This is achieved by: a. Using advanced algorithms and statistical techniques to ensure that each candidate can get a different test while maintaining a standard level of difficulty and assessment. This is effectively achieved through question and option randomisation from a large question bank b. Using technical solutions to secure the items, for example: encryption, locking down browsers, advanced password control. Security has to be built into every stage of the process, from question authoring to candidate identification and supervising exams. It also must be carefully monitored through statistical analysis of candidate and item performance.

14 WHITE PAPER OUTSOURCING E-ASSESSMENT SUCCESSFUL OUTSOURCING PARTNERSHIP Research and real-world experience has shown that there is no one-size-fits-all approach to selecting and working with a partner 7. Organisations stand a better chance of success when they craft a relationship that specifically meets their needs. FIGURE 3: WHAT BUYERS OF OUTSOURCING SERVICES ARE LOOKING FOR IN A PARTNER 000s of Respondents Proven track record Guaranteed service levels Process specialisation Transition strategy Corporate culture Ongoing training Strong governance Centers of excellence Industry specialisation No conflict of interest Source: The Outsourcing Institute 6 The Outsourcing Institute has identified seven factors that contribute to a successful outsourcing relationship and partnership: 1. Understand strategic and tactical objectives and set reasonable and attainable expectations 2. Calculate the true costs of the functions and processes to be outsourced 3. Carefully qualify and evaluate partners 4. Communicate often with internal staff regarding objectives and intentions 5. Develop measurable and achievable service levels and reporting as well as processes for adjusting these levels over time 6. Craft a contract flexible enough to reflect and incorporate changes in requirements, services and pricing, yet explicit enough to lay out the framework, terms and conditions to manage the relationship. 7. Establish a formal and empowered contract management function and team, and recognise that a long term relationship requires constant communication and management. 6 Outsourcing Survey report ; RBC Dexia and Accenture 7 The Evolution of Outsourcing: Moving Beyond Costs to Competitive Advantage James G Schmiedeskamp

15 WHITE PAPER SERIES OUTSOURCING E-ASSESSMENT CONCLUSIONS There are significant benefits to implementing e-assessment to assess and certify candidates for high stakes professional licences and practicing certification. Most importantly, it offers the national awarding body the opportunity to improve the reliability and quality of candidate assessment, enabling it to better meet its statutory obligations and implement its strategic goals. It also offers the whole sector a more flexible and scalable assessment process, and implemented properly it can reduce the total cost of the endto-end process. The e-assessment end-to-end process is made up of three components that require distinctive and very different skills, knowledge, processes and technology to manage effectively: Statutory Standard Management: deep knowledge of sector and the standard Test Creation: knowledge of assessment best practice, technology and process to deliver high quality assessments Feedback and Analysis: understanding the sector, standard, trainining and assessment principles National awarding bodies strategic goals are often broad and aligned to a change agenda within their industry sector, rather than becoming expert in implementing e-assessment processes. For example, a civil aviation national awarding body s goals concern improving aviation safety within a country, whilst a financial services national awarding body s goals are to improve professional standards within the sector. It is unlikely that all the skills, knowledge and technology required to implement an end-to-end e-assessment process can be costeffectively brought together in one organisation. Outsourcing parts of the end-toend process is a realistic method to enable the national awarding body to implement best practice e-assessment. It has some clear advantages over building in-house capability: 1. Access best practice process knowledge and technology 2. Minimise capital expenditure and leverage economies of scale 3. Build a scalable and flexible solution that easily adopts future advances 4. National awarding body can focus on what is their core strengths and achieving their strategic goals Outsourcing is well established in both government and commercial organisations. Many of the processes and practices are mature, benefiting from years of learning and understanding to ensure effective service-delivery by a third party while the organisation remains in control of the overall process. In surveys, overwhelmingly, the top features that organisations look for in outsourcing partners are: proven track record, guaranteed service levels and process specialisation. The right outsourcing partner can bring a wealth of knowledge and practical experience to bear, significantly lowering the risks associated with implementation (eg quality of delivery, cost blowouts) and ongoing service delivery. It should also shorten the time taken to implement the e-assessment process, delivering the benefits to the national awarding body more quickly. There is no one-size-fits-all solution. Organisations should consider carefully their objectives and strategic goals. Vendors can offer the total solution, but depending on the national awarding body s circumstances and objectives, there is a much better chance of success if they craft an outsourcing relationship and solution that specifically meets their needs. Clear advantages to outsourcing over investing to building in-house capability

16 About ASPEQ ASPEQ is a New Zealand based Company that provides global assessment services to national awarding bodies. Our specialist sector area is aviation, but we provide services to other sectors such as finance, maritime, land transport and building and construction. If you have queries concerning this White Paper or would like further information please contact: Mike Lynskey Director Business Development Phone: mike.lynskey@aspeq.com Level 3, 21 Andrews Avenue, Hutt Central 5010 PO Box 30056, Lower Hutt 5040, Wellington, New Zealand