May 2018 Funding & Performance Management Rules V6 Rebecca Rhodes, Senior Associate, UVAC

Transcription

1 DEEP DIVE : MONITORING & GOVERNANCE MANAGING KPIs, RISK & PERFORMANCE IN APPRENTICESHIPS May 2018 Funding & Performance Management Rules V6 Rebecca Rhodes, Senior Associate, UVAC r.rhodes@bolton.ac.uk

2 Documentation (with hyperlinks)

3 Agenda Policy and Rules External Monitoring and KPIs Key Performance Indicators and Setting Standards Reporting and Analysing Data Governance

4 EXTERNAL MONITORING

5 QA-09 QA-10 QA-11 RoATP Narrative Questions HOW do you.. Describe how: Leaders and managers in your organisation communicate their plans and expectations in order to attain high standards of training You will ensure you have appropriate staff in place including trainers with expertise relevant to the apprenticeship training that you will offer The apprenticeship training delivered by staff will be up-todate By considering an apprentices training from start to end, describe how you ensure that your apprenticeship delivery: meets the needs of both apprentices and employers is relevant to the apprenticeship frameworks and/or standards that you offer engages, challenges and inspires apprentices Please note that you must address all the points above including detail on training from start to end Describe: How you assure the quality of your apprenticeship training What staff performance mechanisms you have/will have in place, and how these are managed How you measure/will measure the success of apprenticeship delivery across your organisation and how you will use this to improve your apprenticeship training Please note that you must address all the points above A quality risk identified from any source is escalated to the Pro-Vice Chancellor Education with monitoring via the Academic Standards & Quality Enhancement Committee. We work with appropriate industry fora to ensure a full annual review of the programme annually Feedback from employers is part of the quarterly client/apprentice review meetings, annual reviews and the course review process. The Associate Dean for Education, who sits on Education Executive Committee, chairs the monthly Apprenticeship Steering Group monthly. Data on; retention, progression and success; student and employer satisfaction; and employment outcomes, are all benchmarked annually by our Monitoring Group

6 Managing KPIs & Risk Its Someone s Job - Monitoring & Review people measure, discuss and take action on risk, exceptions and progress Governance the system regularly asks questions about risk, exceptions and progress based on data Accountability named people are accountable at every level for action on risk and progress and exceptions Processes thresholds are set that trigger risk discussions everyone questions about risk, exceptions and progress Data and Evidence is an input to every discussion, and targets are set where issues arise)

7 In Brief Our Survey* Said *not very scientific survey Main KPI in most HEIs was employer pipeline &/or Apprenticeship starts Most governed by the current or ex Project Group Only some regularly produced data and tracking reports to assess risk Very varied and polarised response to tracking attendance 50% had one or more KPIs, though often without a target Most felt academic team monitoring was adhoc or not currently sufficient Some still concerned about tracking the 20 % off the job About 50 % had an Executive Board sponsor as Chair/ present A few had financial / earnings targets for the programme Most had some form of satisfaction survey though no target All conducted module reviews with apprentices but some not mandatory No-one had a compliance improvement KPI!!

8 NATIONAL APPRENTICESHIP MONITORING

9 Funding Rules & Compliance Eligibility ILR matching Subcontracting Documents Evidence Pack

10 Learner & Employer Satisfaction Survey Purpose = informed consumers and purchasers Annual with a minimum sample size Providers manage this survey, distributing links to a standard questionnaire IPSOS MORI follow up if the minimum sample size is not reached Provider s own results are available National Tables provide further data Anonymous l_report_ pdf

11 Qualification Achievement Rates Purpose = informed consumers and purchasers by showing the relative quality of provision Ofsted use them as part of their inspection process QARs are used in the National Achievement Rate Tables NARTS Level 4+ grouped together No HEI or HEP provider category Calculated for individual qualifications or programmes for apprenticeships. No. starts that achieved Typically calculated at a qualification or programme level and aggregated across different types of course, or for particular colleges or providers. ta/file/677223/apprenticeships_qar_dataset_production_spec_ _v3.pdf Specification ta/file/712365/apprenticeships_qar_dataset_production_spec_ _v1.pdf Exceptions (Break in Learning, Transfers)

12 Achievement Rates

13 Achievement Rates by SSA

14 Funding Monitoring Structured analysis of all apprenticeship ILR data to look for issues and concerns Specification published each year In addition the general monitoring applied to all provision, there are some points that are for apprenticeships only

15 MONITORING & KPI FRAMEWORK

16 Monitoring & KPI Framework Compliance Quality Success Audit & Evidence Review RoATP Compliance Funding Monitoring Starts & Achievements Quality Achievement Rates Learner & Employer Satisfaction

17 Can you see anything yet.? VISIBILITY Triage Manage Monitor Remove

18 KPI* & Governance Framework Organisational Success Remuneration, reputation, impact.. Programme Success: 1. Financial (forecast earnings, timely claims and payments ) 2. Compliance (evidence pack and ILR error rate) 3. Quality (achievement, retention / success & satisfaction) 4. Growth (business & starts volumes, new delivery areas, destinations ) Individual Success Progression, completion, satisfaction, destination University Faculty Subcontractor(s) School Course Individual R I S K R E G I S T E R Sponsor Dean / PVC Course Leaders / Head of School Course Leader Academic Mentor *Set KPIs where you have a risk or an opportunity and add trigger points to drive monitoring and review

19 Subcontract Monitoring observation of initial guidance, assessment and delivery of training and/or on-programme assessment include whether the apprentices exist and are eligible i.e. are meeting Funding Rules You must manage and monitor to ensure high quality delivery that meets the ESFA funding rules You must carryout a regular and substantial programme of QA checks visits at short notice You must report any discrepancies You cannot transfer the employer relationship lead You are responsible for the quality of delivery by your subcontractors

20 Monitoring Individual Success/Risks Risk Assess at Tri-partite review :

21 Individual KPI Monitoring Progress Assessment Assignment Grades Contribution Attendance Trajectory..on schedule? * Timeliness of Submissions Action Plans Development Stretch Referrals Complaints Compliance 20 % off the job * Employer Environs e.g. Working Week or support The Commitment Statement is their KPI Monthly Assessment and review Issues Feed Monthly / Bimonthly Course Review Course Risk Register Overall RAG

22 Cohort & Course Review 1. Financial (forecast earnings, timely claims and payments ) 2. Compliance (evidence pack and ILR error rate) 3. Quality (achievement, retention / success & satisfaction) 4. Growth (business & starts volumes, new delivery areas, destinations ) Monthly Review Data drives the triage process Risk Review based on trends Same list as individual, but aggregated plus action planning trends Termly Review Formal data review including module and satisfaction reviews

23 Cohort & Course Review University 1. Financial (forecast earnings, timely claims and payments) 2. Compliance (evidence pack and ILR error rate) 3. Quality (achievement, retention / success & satisfaction) 4. Growth (business & starts volumes, new delivery areas, destinations ) Monthly Review Apprenticeship Hub Data drives the triage process Risk Review based on trends Use cross cutting data analysis Termly Review Steering Group Formal data review including module and satisfaction reviews and compliance audits Reports from wider teams e.g. finance, Records/Registry, business development

24 Satisfaction Monitoring End of autumn term After first Tri-partite review After first module Sufficient work place and off the job With ESFA Annual survey? At end of Summer Term ESFA is anonymous and my not add to the course review

25 Annual KPI Monitoring 1. Financial (forecast earnings, timely claims and payments ) 2. Compliance (evidence pack and ILR error rate) 3. Quality (achievement, retention / success & satisfaction) 4. Growth (business & starts volumes, new delivery areas, destinations ) Time series trends Sector, demographic, size cross cutting trends University, Faculty, School and Course - same questions, different scope

26 Starts and Completions Data Analysis Sector/Subject /School FORECAST TRENDS OVER TIME TRENDS Employer Size Apprenticeship Level Demographic Grouping

27 GOVERNANCE

28 Common Governance Gaps Insufficient scope/remit in the group to drive quality and changes Split between Teaching and Learning or quality or other programme governance structures and Performance Structures No cross organisational remit on Risk Not decision making or taking made recommendations Did not meet frequently enough to add value Focus on Managing the project not improving the product Remit Not on addressing Risk Isolated not Integrated with wider university governance No statistical input to inform performance or risk assessment Risk registers focussed on implementation Meetings don t connect as a circle.one does not feed the other No-one collating a single picture of the overall programme Not triaging sufficiently so very time and resource hungry

29 Timetable of Monitoring Monthly Termly Annually At Course Completion Quality (achievement, retention / success & satisfaction) X X X X Growth (business & starts volumes, new delivery areas, destinations ) X X X Compliance (evidence pack and ILR error rate) X ILR X EP X EP X EP Financial (forecast earnings, timely claims and payments X X

30 KPI, Risk & Governance Matrix Individual Apprentice KPI Meeting Frequency Monthly Remit Data Input Remit Progress Triage Action Plan Trends Take action needed to ensure apprentice is on planned trajectory Cohort / Course Monthly Take action needed to ensure course meets KPIs: School Faculty University Steering Group University Corporate Monthly Termly Termly Annually School / Faculty Annually

31 KPI Management Good Practice Dust off your RoATP and map what you said you would do against a timeline/ schedule Performance Monitoring delivers Performance Management ( Unless you know; you don t know ) - create a data suite and reports Monitor and model based on actual data and forecast frequently -, volumes, achievement/completion forecasts Set trigger points to automatically drive an action to improve e.g. ( outstanding, error rate apprentice progression) and ACT QUICKLY Ensue HESA/ ILR submissions are timely and supported by the paperwork and check they match on key dates. Use the action plan / triage process to inform file selection of file compliance checks Rigorously enforce governance arrangements (Internal and with subcontractors ) everyones priority

32 Questions Please use the questions facility to add your feedback and comments about the webinar Any queries or anomalies - please me r.rhodes@bolton.ac.uk Webinars To Book : Audit and the Evidence Pack Webinar Webinars to Review : Preparing for the RoATP Compliance Visit Subcontracting