Office of Learning Technologies (OLT) Program

Transcription

1 Internal Audit Services October 2006 Audit of the Office of Learning Technologies (OLT) Program SP E

2

3 Audit of the Office of Learning Technologies (OLT) Program Project No: 6526/05 Project team Director General: Senior Director: Audit Director: Team Leader: Auditors: Elizabeth Murphy-Walsh Barb J. McNab Steve Kirk Gloria Muylders Dany Fortin Julie Lévesque APPROVED: SENIOR DIRECTOR: B. J. McNab February 28, 2007 Name Date DIRECTOR GENERAL: E. Murphy-Walsh February 28, 2007 Name Date October 2006 SP E (également disponible en français)

4 Paper ISBN : Cat. No.: HS28-67/2006E PDF ISBN : Cat. No.: HS28-67/2006E-PDF

5 TABLE OF CONTENTS EXECUTIVE SUMMARY... i 1. INTRODUCTION Background AUDIT FINDINGS Criterion Criterion Criterion Criterion CONCLUSION APPENDIX A APPENDIX B... 23

6

7 EXECUTIVE SUMMARY The Office of Learning Technologies (OLT) promotes innovative lifelong learning opportunities for Canadians. The OLT is a contribution program that engages sponsors and their partners using technology to enhance learning and skills development, allowing Canadians to participate fully in the workplace and their community. OLT achieves this through contribution funding, awareness activities and research initiatives. Between 2000 and 2002, funded projects under the OLT included activities under four initiatives: the New Practices in Learning Technologies Initiative (NPLT); the Learning Technologies for the Workplace Program (LTW); the Research in e-learning (ReL); and the Community Learning Networks (CLN) Initiative. However, following the 2002 Summative Evaluation, OLT s focus was redirected to the CLNs umbrella initiative that focuses on communitybased e-learning approaches. In response to the renewal of the OLT Terms and Conditions in November 2002, Human Resources and Skills Development Canada (HRSDC) developed a Risk Based Audit Framework, which provides for the completion of an audit. This audit was undertaken to comply with this requirement. The objective of the OLT audit is to provide assurance to senior management on the adequacy of: controls and practices with respect to sound financial management; stewardship of program funds; and program information for management decision-making. The field work was conducted from May 2005 to October 2005 and included onsite sponsor (organization) visits (Alberta, Ontario, Québec, and New-Brunswick), a sponsor teleconference (Nova Scotia), interviews with program management and staff, and a file review of 17 project files. The selection was based on a judgmental approach from a file population of 237 projects funded between 2001 to 2005 under both EI Part II funds and Consolidated Revenue Funds (CRF). Care must be taken when drawing conclusions from the file review to the whole population, however the file review combined, with other audit work conducted and evidence examined in support to the audit criteria, allows us to provide assurance on the above audit objective. Main findings Adequacy of controls and practices with respect to sound financial management The program s policies, procedures and operational guidelines have been clearly defined, are appropriate, and available with the program administrative processes being consistently applied. Improvements are needed to further streamline the processing Internal Audit, HRSDC i

8 time period between the application and the final approval phase within the context of departmental initiatives. 1 Agreements clearly identify eligible and measurable activities as well as the sponsor s responsibilities. Overall there is full compliance with the Cash Management Policy. Adequacy of stewardship of program funds Program Management has identified several risks related to the program and an action plan was prepared and implemented to mitigate them. However, there are two significant risks that were not identified by Program Management and therefore were not part of specific initiatives: the non sustainability of project accomplishments; and a limited promotion and dissemination of accomplishments. By placing responsibility on sponsors to increase use, promote and disseminate accomplishments to individuals and communities, there is a risk that the program is not achieving the full potential from resources invested and in meeting program objectives. OLT should develop and implement mitigation strategies in relations to these two additional risks. Appropriate monitoring is conducted with the exception of controls for in-kind contributions and the sponsor tendering process. The OLT Terms and Conditions place a limit on the total government contribution given to a project. This should be verified by using a cost effective process developed to clearly identify what supporting evidence will be considered acceptable. As well, the tendering process used by the sponsor, especially for contracts over $25,000 should be verified. There were some inconsistencies in the monitoring approach used by Program Officers and indications that the monitoring form is not optimal for the OLT program. Thus, there is a need to ensure that some key elements unique to their projects are monitored adequately. The project performance is evaluated and documented as required, with the exception of a follow-up exercise by OLT to determine if the implementation of the sustainability plan of accomplishments has taken place. Adequacy of program information for management decision-making There are some indicators and data for determining how the program s expected shortterm outcomes are achieved. However, there are no indicators for determining how the program s expected medium and long term outcomes set out in the Terms and Conditions are achieved. As well, for outcome achievement diagnosis, OLT should use benchmarks to allow making comparisons. 1 Human Resources, Skills Development, Social Development and the Status of Persons with Disabilities Standing Committee ii Internal Audit, HRSDC

9 Audit Conclusion Based on the overall results of the audit and the conclusions reached for each of the audit criteria, we concluded that overall, the program is adequately managed. Based on the evidence examined, OLT has adequate controls and practices in place to ensure sound financial management. However, improvements are needed in some specific areas. OLT should develop and implement mitigation strategies for the two additional risks identified through this audit. Developing adequate controls in mentioned areas will further strengthen the stewardship of program funds. Furthermore, developing mechanisms, establishing benchmarks and indicators to measure medium and long term outcomes will provide program management with appropriate information for decision making. In order to enhance the overall management of the OLT, we have included in this report a series of recommendations and a management Action Plan to address the audit s findings contained in Appendix A. It should be noted that the audit findings have been shared with OLT Program Management throughout the audit process. OVERALL CLIENT RESPONSE: The audit of the Office of Learning Technologies (OLT) was undertaken to provide assurances that the activities were appropriately managed. We are pleased with the overall findings that the OLT is adequately managed and that appropriate controls and practices are in place to ensure sound financial management. We agree with the Audit recommendations and have instituted and will continue to work on mechanisms to strengthen the stewardship of program funds and to establish benchmarks and indicators in support of management decision-making. We would like to note that the OLT was recently integrated into the new Adult Learning, Literacy and Essential Skills Program (ALLESP). The actions identified in the Management Action Plan will be integrated in the ALLESP to ensure effectiveness of financial management controls and adequacy of program information for the purpose of decision-making. Internal Audit, HRSDC iii

10 iv Internal Audit, HRSDC

11 1. INTRODUCTION 1.1 Background In response to the renewal of the Office of Learning Technologies (OLT) Terms and Conditions in November 2002, HRSDC 2 developed a Risk Based Audit Framework, which provides for the completion of an audit with a view to providing assurance that risks are being identified and managed. This audit was undertaken to comply with this requirement. The OLT promotes innovative lifelong learning opportunities for Canadians. The OLT is a contribution program that engages sponsors and their partners using technology to enhance learning and skills development, allowing Canadians to participate fully in the workplace and their community. OLT achieves this through contribution funding, awareness activities and research initiatives. OLT falls under two separate funding authorities; the Consolidated Revenue Funds (CRF) and the EI Part II Funds. The CRF is regulated by the OLT Terms & Conditions for Contributions. The objectives of the OLT Program are: to enable Canadians to develop new knowledge and improve skills through the use of e-learning; and to reduce barriers in the use of learning technologies, thus contributing to reducing the social and economic divide between people who have computer skills and access to the internet and those who do not, referred to as the Digital Divide. In 1997, a submission was made to the Employment Insurance Commission requesting the approval to use Research and Innovation Support Measure (EI Part II Funds) under the Employment Benefits and Support Measures (EBSM) as an additional source of funding. The objective of the OLT Program under the EBSM Research and Innovation Supports Measures is: to identify better ways of helping learners prepare for or keep employment and be productive in the labour force. Until 2003, OLT funded projects under four different initiatives: the New Practices in Learning Technologies Initiative (NPLT); the Learning Technologies for the Workplace Program (LTW); the Research in e-learning (ReL); and the Community Learning Networks (CLN) Initiative. 2 Previously HRDC Internal Audit, HRSDC 1

12 To implement the recommendations of the 2002 program evaluation 3, OLT s focus was redirected in to the Community Learning Networks (CLN) umbrella initiative that focuses on community-based e-learning approaches. This approach explores new ways of helping individuals increase learning opportunities for skill improvement, thereby enhancing employability, and enabling Canadians to create and use information and services online, prepare for or keep employment, and be productive participants in the labour force. The number of funded agreements included in the audit file population 4 is: Type of agreement Population of Agreements Approved by OLT Funding Amount EI Part II Research and Innovation M$ CRF M$ A file sample of 17 files was selected based on a judgmental approach from a file population of 237 projects between 2001 to 2005 under both EI Part II and CRF funds, excluding projects with funding less than $40,000. In the selection, auditors ensured to have adequate representation of the program during the audit period and include sufficient files where projects had come to an end. Therefore, 3 files were selected for , 6 for , 5 for , and 3 for The selection also included files from each initiative (2 ReL, 2 LTW, 3 NPLT) but with special attention to the CLN (10) to place more emphasis on the new program direction. Type of agreement Sample of Agreements Reviewed Funding Amount EI Part II Research and Innovation M$ CRF 6 1.5M$ The maximum level of total government assistance (federal, provincial and municipal assistance for the same eligible expenditures) for this program is not to exceed 75% of eligible expenditures for projects funded under the CRF. Projects funded under EI Part II may allow 100% government assistance (federal, provincial or municipal) to cover eligible costs. The OLT program has set a maximum level of total government assistance at 75% for all projects. 3 4 R.A. Malatest & Associates. Summative Evaluation of the Office of Learning Technologies. Submitted to HRDC, November The file population for the audit include files selected from 2001 to Internal Audit, HRSDC

13 The OLT annual program contribution budget 5 is: Contribution Funds OLT - Consolidated Revenue Funds $2,800,000 $2,800,000 6 $2,800,000 6 EI Part II - Employment Benefits and Support Measures Funds $14,500,000 $14,300,000 $14,300,000 Total Contribution Funds $17, 300,000 $17, 100,000 $17, 100,000 The OLT operates with a staff of 26, and relies on the expertise of consultants as needed to undertake or assist in activities such as targeted research, statistical analysis, surveys, workshops, and public forums related to learning technologies. The OLT is a nationally delivered program. To deliver its program, OLT works with a range of organizations and individuals including sponsors, internal HRSDC contributors (Human Resources Centre of Canada and Regional Offices), subject matter experts, strategic alliances and other government departments. AUDIT OBJECTIVE AND CRITERIA The objective of the OLT audit is to provide assurance to senior management on the adequacy of: controls and practices with respect to sound financial management; stewardship of program funds; and program information for management decision making. The audit conclusion is based on evidence examined in support of the following audit criteria: 2.1 The program s policies, procedures and operational guidelines are clearly defined, appropriate, available and consistently applied. 2.2 Management has identified the risks facing the program and the action plan developed has been implemented to mitigate the risks. 2.3 Operational performance indicators and established mechanisms (including datacapture infrastructure) are timely, relevant, and used for reporting and decision-making. 2.4 Adequate administrative and financial controls have been designed and implemented with emphasis that: agreements clearly identify eligible activities, the expected results and the agreement holder s responsibilities; 5 6 Please refer to Appendix B for Administrative costs. $500,000 was transferred to Learning Initiative Partnerships (LIP) Internal Audit, HRSDC 3

14 terms of payments are met; appropriate monitoring activities are conducted; and the project performance is evaluated and documented. SCOPE: As the OLT is a nationally delivered program, the examination phase was conducted at the National Headquarters (NHQ) and at selected sponsor organization in the regions. The audit scope was limited to the above four criteria. The scope of the audit being examined was determined using a risk based analysis approach. Internal Audit reviewed a sample of 17 project files covering the period of April 2001 to March Performance Tracking Directorate (PTD) results for the same period has also been considered. METHODOLOGY: The following audit procedures were used: Control Self-Assessment (CSA) survey addressed to program management and staff involved in OLT; interviews with program management and staff; examination of relevant documentation; file reviews and some on site sponsor organization visits; and analysis of program data and information. The audit findings are presented in accordance with the above listed audit criteria. To review the detailed Terms of Reference please refer to Appendix B. 4 Internal Audit, HRSDC

15 2. AUDIT FINDINGS 2.1 Criterion 1 The program s policies, procedures and operational guidelines are clearly defined, appropriate, available and consistently applied. Policies, procedures and guidelines are clearly defined and available As mentioned earlier in this report, Office of Learning Technologies (OLT) falls under two separate funding authorities, the Consolidated Revenue Funds (Office of Learning Technologies) and EI Part II Funds (Employment Support Measures under Research and Innovation). The Terms and Conditions of both funding authorities must be respected. These Terms and Conditions set the parameters for the administration of the program and describe amongst other things, program objectives, eligible activities, duration of projects, performance measurements, funding maximums and the delegation of signing authorities. When conducting a review of OLT documents, we found that the established administrative procedures and operational guidelines were in line with the various departmental policies and Terms and Conditions. This review indicated that the guidelines developed were appropriate and that yearly improvements had been made. The most recent version of the guidelines for sponsors includes several helpful guides and contains all the information needed to complete and submit an application. We found the guides easy to follow, the content appropriate and easily accessible on the OLT Website. Policies, procedures and guidelines are appropriate and consistently applied for the most part Starting in 2003, projects selected are through a Request for Proposal process and the selection is based on predetermined assessment criteria. The selection and assessment of these project proposals can be complex due to the variety of innovation and research projects being examined, and the number of various partners and funders involved. The Review Committee, who screens applicants, is composed of program management, staff and non-governmental experts. Program management and project officers (PO) all agreed that including non-governmental experts (evaluators) as part of the review adds expertise, and offers an open and fair process. Interviews revealed that both program management and POs believe that the review and selection process is useful. To assist staff in the interpretation of policies, procedures and guidelines, OLT management organized peer meetings and have added Team Leaders, content experts, coaches, and mentors to help their staff. Interviews conducted with staff as well as comments expressed in the CSA survey 7 indicated that there were instances where staff had differences in interpretation of the various procedures, guidelines and policies that continue to evolve resulting in a need for ongoing clarification. 7 The CSA survey was sent to all 19 employees, including program management and staff involved in OLT, of which 16 responded. Internal Audit, HRSDC 5

16 Most organizations visited reported that there was a lengthy application and review process. Interviews with OLT staff confirmed that the time span between the submission of an application to a negotiated agreement is lengthy. The file review 8 conducted showed that of the 17 files reviewed 47% of the files took 7 to 12 months to be approved and that 41% took an excess of 13 months. This item will be further developed under criterion #3. The reasons cited by staff for the delays were varied; the many administrative requirements, the need to contact sponsors to acquire missing or more detailed information about activities, costs as well as confirming partnership arrangements. Moreover, the many levels of signings authorities required for this program to obtain final approval of the project all contribute to the time delays. Interviews with program officials all confirm that the process is lengthy. Our findings also concur with those of the 2002 OLT Program evaluation, which also raised this as an issue. Program management are working on streamlining the process by reviewing OLT s processes, however they are limited by departmental requirements and practices. The CSA survey revealed that 75% of OLT staff believe that the procedures and operational guidelines are being consistently applied. The file review has confirmed that for the most part the procedures and operational guidelines are consistently applied except for the monitoring process as presented under Criterion 4. Recommendation No. 1: Program management should continue to work towards streamlining the processing time period between the application and the final approval phase within the context of departmental initiatives. Examples of best practices Many good practices were noted during the onsite visits at the sponsors organization. It was frequently mentioned that the OLT internet site has helpful tools to explain the program, provides detailed information about the application process and makes available helpful tools to ensure that the essential elements of a proposal are addressed such as: The Community Learning Network (CLN) Application Map; Guidelines for Pilot Project Proposals; Tips for Writing a Good Proposal; How to Prepare your Application Package, Point-by-Point; Ten Tips for Getting Funded. 8 A file sample of 17 files was selected based on a judgmental approach from a file population of 237 projects between 2001 to 2005 under both EI Part II and CRF funds, excluding projects with funding less than $40, Internal Audit, HRSDC

17 They also mentioned that these tools help them describe in detail the types of activities undertaken, budgetary plans for the entire project life cycle, expected results and milestones to be achieved, and how the accomplishment of the projects will be measured. This is a useful exercise benefiting the sponsors and allowing them to stay on course once the project is underway as well as assisting the project officer to track progress on a quarterly basis and to provide advice when needed. We conclude that overall this criterion has been met. The program s policies, procedures and operational guidelines are clearly defined, appropriate, available and generally consistently applied. However, improvements are still needed regarding the reduction of timelines between the application and approval process. 2.2 Criterion 2 Management has identified several program risks and developed an action plan to mitigate these risks. Office of Learning Technologies (OLT) specific risks were identified in the 2002 Risk-based Audit Framework (RBAF) in anticipation of renewing the Treasury Board Terms and Conditions. These risks include the uncertainty of receiving operating and contribution funds (projects span several years, whereas funding is annual), high staff turnover rate, increased administrative burden for sponsors, overlap of partners mandates (e.g. Industry Canada, Heritage Canada, Agriculture Canada, etc.), information technology systems and performance measures. Note that the performance measures risk was reiterated in the December 2002 program evaluation report. The program must also consider corporate risks pertaining to grants and contributions, such as public confidence and consistent, efficient and effective services. These risks were also identified in the 2002 RBAF, and corporate initiatives were proposed for mitigating these risks. After a review of several documents, interviews with project officers and management, a review of files and on-site visits with sponsors, we observed that for the several risks that were identified, an action plan was prepared and implemented to mitigate them. In our view, there are two additional risks that were not identified by Program Management and therefore were not part of specific initiatives. 1 Risk that accomplishments resulting from OLT projects are no longer available to learners after the funding period. As mentioned in the OLT guidelines, projects must research, develop and test innovative and sustainable approaches that help adult learners to find and keep employment in the knowledge-based economy. The accomplishments resulting from OLT-funded projects need to be available to learners in order to achieve program objectives. The term accomplishment for our purposes refers to new learning approaches, practices, processes, activities or tools developed through OLT-funded projects. Internal Audit, HRSDC 7

18 The expected short, medium and long-term outcomes stated in the 2003 Terms and Conditions of the program were as follows: Short term (up to one year): Enhanced capacity of sponsors to serve clients using e-learning, enhanced community-based approaches to deliver skills and learning development using technology, increased knowledge of e-learning impacts, trends and effective practices. Medium term (two to three years): Increased access to learning and skill development through the use of e-learning and adoption and expanded use of effective e-learning. Long term (over three years): Use by individuals of e-learning as a tool to develop knowledge and to upgrade skills, a decreased digital divide and enhanced employability of Canadians. OLT s medium and long-term outcomes are not achieved unless the accomplishments resulting from contribution agreements (projects) are available to and used by learners after the funding period. We conducted a follow-up on nine closed files from our sample during the audit (the sample involved 17 files in total). We met and contacted sponsors as well as reviewed final project reports to determine which projects still had accomplishments available and accessible to learners. The data collected is as follows: Accomplishments 9 # of Projects % of Projects Contribution Value Available 4 44% $ 814,499 Partly available % $739,714 Not available 4 44% $1,113,020 Total 9 100% $ 2,667,233 The risk of OLT project accomplishments being unavailable after the funding period is also supported by interviews with program managers and employees and by an external consultation report prepared in The reasons provided were varied, but the most frequently mentioned in the interviews was the fact that several non-profit organizations that develop projects with OLT assistance do not have the resources or infrastructure needed for continuing the activities and maintaining the availability of accomplishments once the funding period ends. Program staff mentioned that there is a need to revisit accomplishments that were successful as well as those that were not. It is our opinion that these lessons learned would be helpful in providing advice to new applicants developing their sustainability plan along with providing information for management decision-making. 9 The term accomplishments refers to new learning approaches, practices, processes, activities or tools developed by OLT projects. 10 One project may involve several accomplishments. The term partly available means that only part of these accomplishments is still available. 11 Toward a Culture of Learning Through Technology, OLT, by Yutta Fricke 2005 (unpublished). 8 Internal Audit, HRSDC

19 Sponsors submitting a proposal for obtaining a contribution agreement with OLT must also include a plan showing how the project s processes, activities and accomplishments will be integrated into the organization s day-to-day operations or how the project will achieve self-sustainability once the funding period ends. The file review indicated that insufficient importance was given to the quality of the sustainability plans to ensure that project accomplishments are available to learners after the funding period. These factors combined represent a significant risk to achieving program objectives. 2 Risk of not achieving the full potential of resources invested due to limited promotion and dissemination of OLT project accomplishments after the funding period OLT supports the annual learning technology forums, and some sponsors present the results of their efforts and research at these meetings. These forums also give OLT the opportunity to promote the program and help sponsors identify the learning technology needs of their respective communities. A report that includes the minutes from these meetings is prepared and published. The OLT Internet site provides a list of funded projects by fiscal year starting with This list includes a summary of activities that sponsors were planning to undertake during the projects as well as their contact information. The same internet site includes annual reports produced by external evaluators to report on some project outcomes. However, the list of funded projects contains summaries of activities to be completed and do not include the accomplishments resulting from these projects and the evaluators reports include some examples of projects that recently closed. A review of the Terms and Conditions as well as the Program Logic Model indicates that there is a requirement for OLT to promote and disseminate information and effective practices in e-learning. We have observed that the program has assigned most of the responsibility for disseminating and promoting project accomplishments to individual sponsors. However, there is currently no mechanism in place for OLT to determine whether adequate promotion and dissemination of activities is carried out by sponsors. Moreover, although some accomplishments are specific to a group or community, we are of the view, following the analysis of files, that many accomplishments are valuable and could potentially be adopted and used by other Canadian communities. Given that sponsors have limited resources or infrastructure, placing responsibility on sponsors to increase use, promote and disseminate accomplishments to individuals and communities there is a risk that the program is not achieving the full potential from resources invested and in meeting program objectives. In our view, this is mostly related to program design and program management has indicated that they have insufficient resources to address this item. Internal Audit, HRSDC 9

20 This criterion is partially met. In order to address the program risks identified above, OLT should implement the following recommendations: Recommendation No. 2: To achieve the full potential of resources invested, Program Management should develop and implement a strategy to: increase sustainability of accomplishments beyond the funding period; and increase promotion and dissemination of OLT project accomplishments after the funding period. 2.3 Criterion 3 Operational performance indicators and mechanisms (including a data capture system) produce timely information, are relevant and are used for producing reports and making decisions. Formal operational performance indicators need to be developed OLT uses the common system (Common System for Grants and Contribution - CSGC) and an in-house Access database as a data capture mechanism in the day-to-day management of operations. The information is produced on an on-going basis and is used in decision-making. Program Management has informed us that currently there are no formal operational performance indicators and service standards. As mentioned in criterion #1, the processing time period between the submissions of a proposal to the time when the agreement is finalized is lengthy. Recommendation #1 proposes that Program Management continue their efforts to simplify the process. In our opinion, tracking the time between the different stages of the process would help provide detailed information to identify areas for improvement and measure quality of service. Recommendation No. 3: Program Management should establish an indicator to set an appropriate service target to assess outcomes and facilitate making comparisons. Limited strategic performance indicators Regarding strategic management of the program, performance indicators were identified as a risk in the 2002 program evaluation report. This report indicated that there was not enough data for the evaluators to determine whether OLT was cost-effective. It also mentioned that there was a need to improve outcome tracking with end users. As well, the program s and business plans indicate performance measurement as a weakness. 10 Internal Audit, HRSDC

21 In 2004, a mechanism for collecting data from sponsors was introduced. This collection mechanism uses sponsors quarterly progress reports (monthly for agreements of $350,000 or more) to collect the data used for the report entitled Results of a Preliminary Review of Outcomes of Stream 2 Pilot Projects presenting results every six months. However, there is no benchmark of these results to compare with what should be reasonably expected. We believe that this mechanism is relevant in that it collects data on the program s expected short-term outcomes and relates to the contribution agreement management. In particular, it gathers data on an ongoing basis during the agreement because each sponsor is required to report regularly on activities and on the completion level of the project s accomplishments. However, this mechanism has its limitations, as it tracks primarily short term activities. For effective strategic planning of activities and investments, OLT needs to have access to sufficient information about past projects accomplishments to determine if investments have achieved expected medium and long term outcomes. Recommendation No. 4: Program Management should develop a mechanism for collecting data and establish an indicator to determine the number of project accomplishments that are available and used by learners after the funding period to achieve medium and long-term outcomes. We conclude that this criterion is partially met. Although there are some indicators and data for determining how the program s expected short-term outcomes are achieved, there are no indicators for determining how OLT program s expected medium and long term outcomes set out in the Terms and Conditions are achieved. 2.4 Criterion 4 Adequate administrative and financial controls have been designed and implemented with emphasis that: agreements clearly identify eligible activities, the expected results and the agreement holder s responsibilities; terms of payments are met; appropriate monitoring activities are conducted; and the project performance is evaluated and documented. Internal Audit, HRSDC 11

22 Agreements clearly identify eligible activities, the expected project results and the agreement holder s responsibilities A review of Office of Learning Technologies (OLT) documents demonstrated that adjustments were made to OLT s written agreement in 2002, 2003 and 2004 to meet the various changes brought to the TBS and Human Resources Development Canada (HRDC) policies and directives. During the file review, it was evident that significant improvements have been made in the administration of project files when we compared files from 2001 to those of Contribution agreements after 2002, between the department and the recipient of funding, included a detailed description of eligible activities, listed eligible costs broken down by cost categories, and detailed the sponsor s responsibilities in meeting objectives and expected results to be achieved. In addition, OLT agreement templates used for CRF and EI Part II funded projects between 2003 to 2005 describe in more detail the obligations of both parties to carry out the OLT projects. It has also contributed to clarifying OLT expectations and has increased consistency of program administration as demonstrated in the file review. Other improvements noted included modifications to the agreement Schedule B and lengthening the preservation period of books and records. Schedule B is used to define what expenses can be incurred and under what conditions. As of 2003, the Schedule B provided more details regarding eligible expenses for reimbursement and clearly identified cash and in-kind contributions made by the sponsors and partners. Also, past agreements required the preservation of books and records for audit and inspection by HRSDC representatives for a period of one year only following the end of the funding period. Effective 2003, this period was extended to 6 years. We conclude that the agreements clearly identify eligible and measurable activities and the sponsor s responsibilities. Terms of payments are met The Cash Management Policy included in the TBS Policy on Transfer Payments states that transfer payments should not be paid to recipients in advance of need; payments should be timed to correspond as closely as practicable to recipients' cash flow requirements. The file review confirmed that there is compliance with the payment terms outlined in the agreement regarding the following: cash flow requirements, eligibility of claimed costs, quarterly advances holdbacks, stacking of assistance, and year end considerations. The HRDC CSGC Operations Manual states that progress payments are to be made under the terms of a contribution agreement and paid after sponsor demonstrates that they have achieved the milestones set for the quarter. Files examined showed that the following documentation is to be submitted by the sponsors prior to processing a payment. Sponsors must submit quarterly reports on achievement of performance objectives along with a rational if objectives were not met; 12 Internal Audit, HRSDC

23 Sponsors must submit expenditure details listed by cost category prior to reimbursement of costs; Sponsors must submit quarterly reports concerning the progress of the project. The file review indicated that based on the average of the quarters for all fiscal years 99% of payment claims that had been submitted were verified by a Project Officer (PO) to ensure that claims reflected the actual expenditures according to the agreement and is reasonable based on the activity report and 97% had submitted the quarterly progress report as required. Also all of the files showed that their amounts were adjusted or eliminated as appropriate when the PO responsible for the administration of the agreement disallowed costs. The on site visits and file review confirmed that overall there is full compliance with the Cash Management Policy. Appropriate monitoring activities are conducted The HRSDC Monitoring Policy stipulates that all contribution agreements are subject to monitoring of expenditures, activities and results. The intensity and frequency of monitoring is determined based on a risk assessment approach. Since 2004, the CSGC helps the project officer determine whether an on-site monitoring or off-site monitoring should be conducted as well as the number required. The Project officer then establishes the monitoring plan and an appropriate monitoring schedule. Of the 16 OLT staff surveyed, 94% believe the projects are effectively monitored. The file review confirms that 16 out of the 17 files have been monitored and indicates that both types of monitoring strategies are used. However, some elements of the monitoring process could be improved as indicated below. 1 In-Kind Contributions The OLT Terms and Conditions place a limit on the total government contribution given to a project. Partners must provide a cash or in-kind contribution 12 to the project s activities and is a condition for approval of the project. We found that sponsors did provide information about partner contributions however; only 2 of the 17 files had evidence that Project Officers had reviewed documentation verifying that in-kind contributions had been made. Interviews with staff, discussions with sponsors and the file review confirmed that this verification was done only in a few cases. Staff and sponsors also reported that they find it difficult to track and confirm the in-kind contributions. Sponsors mentioned that tracking the in-kind can be costly and time consuming %, %. Internal Audit, HRSDC 13

24 2 Contract over $25,000 There is also a requirement that all contracts and sub contracts over $25,000 must go through a competitive process with at least 3 bids. The applicant's process must demonstrate that they are receiving the best value possible. We have noted that the sponsor guide and agreement clearly outline the sponsor obligations pertaining to this. However, the file review indicated that 5 of the 8 files where subcontractors over $25,000 were used did not contain evidence of PO verification to ensure that sponsors had obtained 3 bids as required. The 3 files that contained evidence of PO verification indicated that sponsors obligation to obtain 3 bids had not been met. Interviews with staff did confirm that particular attention had not been placed on this aspect of monitoring. 3 Monitoring Approach The file review indicated that there were some inconsistencies in the monitoring approach. Some Project Officers request many source documents from the sponsor s accounting records for expenditures claimed (e.g. invoices, payroll records, travel expenses), while others accepted the OLT template for reporting expense details. As well, we did note that the departmental monitoring form used for both financial and activity monitoring have a high percentage of questions that are not applicable to the OLT program. The forms are not optimal when verifying key elements related to OLT projects such as professional fees, in-kind contributions, end-users experience in using the e learning approaches along with the promotion and dissemination of effective practices in the use of e-learning. The file review and the onsite sponsor visits demonstrate that appropriate monitoring is conducted, but improvements are required for the three above mentioned items. Recommendation No. 5: Program Management should develop a cost effective process for the verification of in-kind contributions while keeping in mind the sponsor s ability to provide the required supporting evidence. This process should clearly identify what supporting evidence will be acceptable from the sponsor. Recommendation No. 6: Program Management should ensure that a verification of the tendering process used by sponsors to sub-contract for amounts over $25,000 with 3rd party contractors is in compliance with the agreement and is monitored. Recommendation No. 7: Program Management should ensure that the OLT monitoring form used includes elements specific to the OLT program. 14 Internal Audit, HRSDC

25 The project performance is evaluated and documented The final evaluation summary is to assess if the agreement objective was achieved. The summary includes a recommendation on whether the sponsor completed the activities undertaken, determines if future funding should be considered and is used for reference purposes. When closing the file, POs must ensure that the final financial and activity reports have been received. All files reviewed indicated that the PO completed final close out summary report (findings, comments, recommendations) and determined if the project objective was achieved. The CSA indicates that the majority of OLT staff believe that a project performance evaluation is consistently completed upon close-out. The file review also indicated that the 10% of funds held back (holdbacks) were released only once the sponsor submitted the final payment claim and the recipient s compliance with the agreement is confirmed. As most sponsors receive advance payments for their project, the unexpended balances that remain at the end of the agreement must be repaid to OLT. The file review confirmed that, when applicable, sponsors did repay the unexpended balances as required. Overall, the project performance is evaluated and documented as required, other than a follow-up of the sustainability plan to determine if the accomplishments are sustained as indicated in recommendation #3. We conclude that overall this criterion has been met. Internal Audit, HRSDC 15

26 16 Internal Audit, HRSDC

27 3. CONCLUSION In summary, overall the Office of Learning Technologies (OLT) Program is adequately managed. However, improvements are needed in some specific areas: We found that the program s policies, procedures and operational guidelines have been clearly defined, appropriate, available and program administrative processes are consistently applied with the exception of the requirement that all contracts over $25,000 must be awarded through a competitive process with at least 3 bids. Also, improvements are needed regarding the timelines of the application/approval process. Program Management has identified several risks related to the program and has established and implemented an action plan to mitigate those risks. However, there are two significant risks that were not identified by Program Management and therefore were not part of specific initiatives: the non sustainability of project accomplishments; and a limited promotion and dissemination of accomplishments. Given that these risks can significantly impact on expected results, OLT should develop and implement mitigation strategies. There are some indicators and data for determining how the program s expected short-term outcomes are achieved. However, there are no indicators for determining how the program s expected medium and long term outcomes set out in the Terms and Conditions are achieved. As well, for outcome achievement diagnosis, OLT should use benchmarks in making comparisons. Agreements clearly identify eligible and measurable activities and the sponsor s responsibilities, and overall there is compliance with the Cash Management Policy. Appropriate monitoring is conducted, but improvements are required for the confirmation of in-kind contributions and the tendering process. We also found that the monitoring form is not optimal for the OLT program and that management will need to ensure that some key elements unique to their projects are monitored adequately. The project performance is evaluated and documented as required, with the exception of a follow-up by OLT to determine if the accomplishments are sustained. The establishment of a performance indicator as recommended above will allow OLT to track progress. In order to enhance the overall management of the OLT program, we have included in this report a series of recommendations (Appendix A). It should be noted that the audit findings have been shared with OLT Program Management throughout the audit process. Internal Audit, HRSDC 17

28 In our professional judgment, sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the conclusions reached and contained in this report. The conclusions were based on a comparison of the situations, as they existed at the time against the audit criteria. This internal audit was conducted in accordance with the TBS Policy on Internal Audit and the Institute of Internal Auditors Standards for the Professional Practice of Internal Auditing. OVERALL CLIENT RESPONSE: The audit of the Office of Learning Technologies (OLT) was undertaken to provide assurances that the activities were appropriately managed. We are pleased with the overall findings that the OLT is adequately managed and that appropriate controls and practices are in place to ensure sound financial management. We agree with the Audit recommendations and have instituted and will continue to work on mechanisms to strengthen the stewardship of program funds and to establish benchmarks and indicators in support of management decision-making. We would like to note that the OLT was recently integrated into the new Adult Learning, Literacy and Essential Skills Program (ALLESP). The actions identified in the Management Action Plan will be integrated in the ALLESP to ensure effectiveness of financial management controls and adequacy of program information for the purpose of decision-making. 18 Internal Audit, HRSDC

29 APPENDIX A MANAGEMENT ACTION PLAN This appendix provides the details of management s response to the recommendations. Detailed management action plans should use the following table format. Internal Audit Recommendations Management Plan Action(s) to be undertaken Planned Completion Date Responsibility Title and RC Number 1. Program Management should continue to work towards streamlining the processing time period between the application and the final approval phase within the context of departmental initiatives. The program has streamlined activities wherever possible and developed effective tools to assist applicants with the process. This has resulted in better project proposals. The program continues to adjust these tools as required to maintain their relevance and appropriateness. The program is actively involved in two departmental committees: the Directors General Grants & Contributions Management Review Committee and the Grants & Contributions Management Review Working Group which are reviewing the processes and tools used to manage Grants & Contributions programs across the department. These groups are identifying administrative efficiencies which will be considered and implemented as appropriate by the program, as they become available. November 2006 ongoing Liaison with the G&C Centre of Excellence and different committees such as the DG Committee and the Director Grants and Contributions Policy and Program Committee is in place to ensure all new policies, procedures and /or directives are implemented. Director, Program Operations- (9788) and Manager, Quality Assurance Units (QAU) (7594) 2. To achieve the full potential of resources invested, Program Management should develop and implement a strategy to: increase sustainability of accomplishments beyond the funding period Monitoring of number of weeks between project approval and start for 1) grants and 2) contributions to begin with projects from 2007 CFP. At present, funding applications include questions on sustainability. In the future, the program will require satisfactory sustainability plans to be submitted with the applications. In addition, the program will develop sustainability indicators and integrate them in both the contribution agreements and the progress report templates to capture progress on efforts made toward sustainability during and after the funding period. Ongoing from May 2007 December 2006 Ongoing with the integration of OLT into new Adult Learning Literacy and Essential Skills (ALLESP) focus of indicators will be on results for learners while continuing to encourage sustainability of activities. Director, Program Operations- (9788) and Director Program Development (7468) Internal Audit, HRSDC 19

30 Internal Audit Recommendations increase promotion and dissemination of OLT project accomplishments after the funding period 3. Program Management should establish an indicator to set an appropriate service target to assess outcomes and facilitate making comparisons. 4. Program Management should develop a mechanism for collecting data and establish an indicator to determine the number of project accomplishments that are available and used by learners after the funding period to achieve medium and long term outcomes. Management Plan Action(s) to be undertaken The program will also ensure greater dissemination of accomplishments by the capturing of promotion and dissemination activities after the funding period through life cycle project reporting. A listing of project outcomes and best practices will also be posted on the program website, to make accomplishments available nationally after the funding period. The program will review the proposal processing stages and develop a service standard indicator that will allow processing times to be measured. The service standard indicator will help identify areas for improvement. This will be done in collaboration with other program and departmental areas as appropriate. At present, the program collects information on short- and some medium-term project accomplishments, including learners served and training products and tools developed or used during the program funding period. Additional efforts will be made to put in place mechanisms to collect data and establish an indicator on medium and long-term outcomes in collaboration with HRSDC Evaluation. Planned Completion Date November 2006 this is an ongoing activity that will continue throughout the implementation of the Adult Learning, Literacy and Essential Skills Program (ALLESP) under which OLT was integrated. In 2007/08, a new indicator will be introduced to measure time elapsed from project approval to final signature of the agreement. November 2006 ongoing A performance measurement and evaluation strategy is under development for ALLESP with completion expected in April Responsibility Title and RC Number Director, Program Operations- (9788) and Manager, QAU (7594) Director, Program Development (7468) 20 Internal Audit, HRSDC

31 Internal Audit Recommendations 5. Program Management should develop a cost effective process for the verification of in-kind contributions while keeping in mind the sponsor s ability to provide the required supporting evidence. This process should clearly identify what supporting evidence will be acceptable from the sponsor. 6. Program Management should ensure that a verification of the tendering process used by sponsors to sub-contract for amounts over $25,000 with 3rd party s contractors is in compliance with the agreement and is monitored. 7. Program Management should ensure that the OLT monitoring form used includes elements specific to the OLT program. Management Plan Action(s) to be undertaken An in-kind verification process was developed and introduced in 2004 for recipients to report in-kind contributions by category. Documentation is required from the recipient to support any claim. The current claim package includes a separate form for the recipient to report non-hrsdc cash- and in-kind contributions. The process for monitoring in-kind contributions is consistent with the one used to verify regular claims. The current in-kind monitoring process will be examined to determine its costeffectiveness and identify the supporting evidence required from funding recipients. A new approach to verification (monitoring) of the tendering process used by funding recipients subcontracting for amounts over $25,000 was developed and introduced in The approach will be examined to ensure full compliance of recipients with agreement requirements. A monitoring form was developed in The form is based on a departmental monitoring checklist which was adapted to include specific program elements. In addition, a "Financial Consultant Report" and a "Divisional Resolution Report" were developed to address program requirements. The new reports work as a record for all steps in the program s monitoring process. The monitoring form will be reviewed and further developed to ensure all appropriate risk elements are captured for monitoring purposes. Planned Completion Date October 2006 completed Current processes have been reviewed and found to be appropriate as per departmental guidelines. October 2006 Completed Current processes have been reviewed and found to be appropriate as per departmental guidelines. November 2006 The current form was reviewed and adjustments were made. Process will be on-going during the implementation of ALLES programming. Responsibility Title and RC Number Director, Program Operations- (9788) and Manager, QAU (7594) Director, Program Operations- (9788) and Manager, QAU (7594) Director, Program Operations- (9788) and Manager, QAU (7594) Internal Audit, HRSDC 21

32 22 Internal Audit, HRSDC

33 APPENDIX B Human Resources and Skills Development Canada Ressources humaines et développement des compétences Canada Terms of Reference Audit of the Office of Learning Technologies (OLT) Program Project No: 6526/05 Project Team Senior Director: Audit Director: Team Leader: Audit Team: Anthea English Barbara J. McNab Steve Kirk Gloria Muylders Dany Fortin Julie Lévesque APPROVED: SENIOR DIRECTOR: Barbara J. McNab June 3, 2005 Name Date DIRECTOR GENERAL: Anthea English June 7, 2005 Name Date May 2005