This Final Evaluation Report was prepared MHCC by: EJD Consulting & Associates Tel/ Fax:

Transcription

1

2 This initiative was funded by the Department of Education, Employment and Workplace Relations (DEEWR) and NSW State Training Services (STS), and managed by the Mental Health Coordinating Council (MHCC). This Final Evaluation Report was prepared MHCC by: EJD Consulting & Associates Tel/ Fax:

3 CONTENTS EXECUTIVE SUMMARY... VI REPORT INTRODUCTION BACKGROUND Origins of the Program Description of the Program Program Aims Intended Outcomes Roles and Responsibilities Host Organisations Mental Health Coordinating Council About the Program Evaluation Evaluation Framework Evaluation Terms of Reference Methodology About this Report LITERATURE REVIEW NSW Mental Health Workforce Mental Health Status of the Aboriginal Population Aboriginal Participation in the NSW Mental Health Workforce Aboriginal Employment and Educational Attainment Employment Rates of Aboriginal People Importance of Educational Attainment Aboriginal Traineeships Value of Traineeships Cross-Jurisdictional Scan of Traineeships Measuring Outcomes from Aboriginal Traineeships Good Practice in Aboriginal Traineeships ACIMH Program- Final Evaluation Report Page i

4 4 FINDINGS: INPUTS AND OUTPUTS Program Numbers Host Organisations Trainees Overall Results Trainee Profiles Trainee Location Gender and Age Education and Employment Backgrounds Previous Trainee Experience Early Exit Trainees Program Documents FINDINGS: TRAINING About the Training Block release training feedback Perceptions of Overall Change FINDINGS: TRAINEE OUTCOMES & PERCEPTIONS Key Trainee Impacts and Change Support Workplace Issues Perceptions of Host Organisations Perceptions of Supervisors Advice for Future Host Organisations Relationships to Mentors Broader Impacts of Program on Trainees Personal and Professional Development Impacts on Family and Community Future Plans Perceptions of Trainee Changes FINDINGS: HOST ORGANISATION OUTCOMES & PERCEPTIONS Program Implementation Key Host Organisation Impacts Key Benefits Key Challenges Internal Policies and Practices Broader Impacts of Program on Host Organisations ACIMH Program- Final Evaluation Report Page ii

5 8 FINDINGS: CULTURAL APPROPRIATENESS OF THE PROGRAM Overall Assessment Trainee Perceptions Host Organisation Perceptions Future Options Suggested Program Pillars FINDINGS: OPERATIONS AND MANAGEMENT Program Design and Operations Program Governance and Staffing Program Oversight and Reporting Program Staffing Program Finances Income and Expenditure Program Incentives Program Contributors Role of STS Aboriginal Employment Unit Mental Health Coordinating Council Program Documents and Resources FINDINGS: OVERARCHING OUTCOME MEASURES Key Outcome Measures Aboriginal Workforce Development Vocational Education and Training Capacity Building Overall Benefits and Future Options CONCLUSION RECOMMENDATIONS GLOSSARY OF TERMS REFERENCES APPENDICES APPENDIX 1- ACIMH SERVICE AGREEMENT APPENDIX 2- ACIMH FINANCIAL INCENTIVES APPENDIX 3- EXTRACT OF EVALUATION FRAMEWORK APPENDIX 4- ACIMH RESOURCES * * * * ACIMH Program- Final Evaluation Report Page iii

6 LIST OF TABLES Table 1: Evaluation Stages and Products... 8 Table 2: Educational attainment by labour force status, persons aged years Table 3: Total Number of ACIMH Trainees per Host Organisations and Table 4: Final ACIMH Program Trainee Statistics Table 5: ACIMH Consolidated Program Finances (Budget and Actuals) projected to May 2015* LIST OF DIAGRAMS Diagram 1: Traineeship by Employment Type Diagram 2: Trainee Location Diagram 3: Trainee Gender Breakdown Diagram 4: Trainee Age Distribution Diagram 5: Trainee High School Year Completed...32 Diagram 6: Previous Trainee Tertiary Qualifications Diagram 7: Trainee Feedback on Change as a Result of the ACIMH Program 48 Diagram 8: ACIMH Program Strengths for Trainees Diagram 9: Diagram 10: Trainee Feedback on Quality of Support Received by Stakeholder Diagram 11: Recommended Aboriginal Mentor Roles and Qualities 62 Diagram 12: Diagram 13: ACIMH Program Benefits for Host Organisations Diagram 14: Common Challenges Experienced by some Host Organisations Diagram 15: 83 Diagram 16: Diagram 17: Host Organisation Plans to Modify Internal Policies and Practices87 Diagram 18: Cultural Appropriateness of Host Organisations in Terms of Supporting Aboriginal Trainees Diagram 19: stakehol Diagram 20: STS-AEU Program Involvement Diagram 21: MHCC Program Involvement Diagram 22: Value and Usefulness of ACIMH Program Resources ACIMH Program- Final Evaluation Report Page iv

7 CASE STUDIES Trainee Case Study 1 Host Organisation Case Study 2 72 * * * * - Acknowledgements- The Principal and Associates of EJD Consulting wish to acknowledge and thank all those who participated in final project evaluation of the Aboriginal Careers in Mental Health Program. Sincere thanks to all those who took the time to complete the surveys and participate in the various discussions, interviews and focus groups. Special thanks also to the Program staff at the Mental Health Coordinating Council for providing valuable background information and other feedback on the report. Edwina Deakin Principal, EJD Consulting & Associates December 2014 ACIMH Program- Final Evaluation Report Page v

8 EXECUTIVE SUMMARY Introduction What follows is a summary of findings from the final evaluation of the Aboriginal Careers in Mental Health (ACIMH) Program overseen by the Mental Health Coordinating Council (MHCC). The independent evaluation was conducted by EJD Consulting & Associates in August- September 2014 at the conclusion of the formal aspects of the Program. The findings draw on feedback gathered from trainees, mentors, host organisation supervisors and coordinators, as well as MHCC Program staff and trainers and other stakeholders. The methodology included detailed analysis of Program documentation, end-of-block training surveys, a Survey Monkey survey of all stakeholders (n=62), as well as a series of focus groups, interviews and case studies. The final report contains the breakdown of survey statistics, findings and recommendations arising. Key Inputs and Outputs The ACIMH Program commenced in late 2012 with the aim of establishing 50 new traineeships for Aboriginal people based in community managed mental health organisations across NSW. The trainees were employed by the organisation for a minimum of 21 hours per week, and supported to complete a Certificate IV in Mental Health run specifically for the ACIMH trainees by MHCC. By mid-2013, 46 Aboriginal trainees had been successfully recruited to participate in the Program. These were hosted by 10 different community managed mental health organisations, with trainees based in 30 different service outlets: 50% of these were in Sydney, 41% in non-metropolitan areas or rural towns, and 9% in the greater Newcastle area. There were a slightly higher number of female trainees (54%) than males (46%) with over one half (59%) aged between years at the start of their traineeship. The ages of other trainees were spread across different age brackets, including six aged over 40 years and five aged 20 years or less. ACIMH- Final Program Evaluation Report Page vi

9 As of September 2014, 32 ACIMH trainees had completed the Program and successfully graduated with their Certificate IV. This represents a retention and completion rate of 69.6%, which is very positive and significantly exceeds general completion rates for NSW students enrolled in Certificate IV qualifications and above (at 44.3%) 1 75% of these trainees have ongoing employment with their host organisation (23) with one trainee working for another host organisation (1) 56% are currently enrolled in further study (6) or have plans to enrol in further study in the near future (12). A further two trainees (4%) who did not graduate were still enrolled or employed by their host organisation as at September trainees (26%) exited the Program prior to completion, with two-thirds of these in the first six months of the Program. While these exits were disappointing, most reported leaving to take up other positions or career options. The majority (7) also reported the experience had been very worthwhile. For example in exit interviews, all but one trainee reported the Program had: improved their confidence and capacity to work in mental health assisted them in getting another job in the future increased their chances of undertaking further study in the future. Further, the exited trainees reported they would definitely recommend other Aboriginal people take up this type of traineeship in the future. Based on the above input and output data, the evaluators concluded the ACIMH Program was successful, broadly meeting its Program targets and goals as further expanded upon below. Key Benefits and Outcomes When trainees, host organisation staff, Aboriginal mentors and other stakeholders were asked to rate various aspects of the Program (n= 62) the feedback was very positive. For example: 75% and above of all respondents also rated the following Program elements good or very good: o ees participating 1 Source: National VET Provider Collection (NCVER 2014). ACIMH- Final Program Evaluation Report Page vii

10 o o o o o Quality of Program management Efficiency of Program management Usefulness of the MHCC website for Program information 100% of respondents also stated they would recommend other Aboriginal trainees participate in the Program. When host organisation staff, mentors and other stakeholders (n= 39) were asked were very strong. For example, 100% of respondents strongly agreed or agreed with the following statements: The ACIMH Program is a good way of increasing the number of Aboriginal people working in community managed mental health The Program is a good way of assisting Aboriginal people to develop skills and requirements to work in community managed mental health The Program is a good way of increasing the number of Aboriginal people undertaking further study or vocational education The Program is a good way of building the capacity of community managed mental health organisations to employ and support Aboriginal staff The Program is a good way of building the capacity of community managed mental health organisations to engage with Aboriginal people and communities. Conclusion The above results, combined with other findings contained in the final independent evaluation report, demonstrate the ACIMH Program was successful in meeting its objectives. The report documents specific benefits for each stakeholder group, with 80% of all respondents reporting participation in the Program was personally as well as professionally beneficial. The report provides a sound evidence base for the continuation of the Program. It also includes 33 recommendations for how to strengthen the model in the future. These include: Enhancing the mentoring component and integrating it more fully into MHCC management of the Program Expanding the role of Aboriginal mentors to include supporting dialogue between the trainee and their supervisor, particularly in cases where supplementary communication is needed Increasing supervisor access to training content and assessment tasks to better assist the trainee in the workplace ACIMH- Final Program Evaluation Report Page viii

11 Wherever possible co-locating two trainees to increase peer support and overcome isolation particularly in organisations without other Aboriginal staff. Of those consulted there was unanimous support for the Program to continue in order that other Aboriginal trainees and community managed mental health organisations benefit from the current model. All consulted confirmed a pressing need for more professionally trained Aboriginal workers in the sector. The evaluators concluded that the MHCC Program model offered a practical and supportive mechanism for this need to be met and delivered upon within a short time period. * * * * ACIMH- Final Program Evaluation Report Page ix

12 REPORT 1 INTRODUCTION In March 2012 the Mental Health Coordinating Council (MHCC), in partnership with the Department of Education and Communities (DEC), instigated the Aboriginal Careers in Mental Health Program (hereafter referred to as the ACIMH Program or Program ). The Program was a NSW community mental health workforce development initiative aimed at establishing 50 new traineeships for Aboriginal people across two years. It was funded by the then Commonwealth Department of Education, Employment and Workplace Relations (DEEWR) (now transferred to Prime Minister and Cabinet) and also by DEC. The first of three trainee cohorts commenced in the Program in October At the end of August 2014, the trainees who had successfully completed their studies graduated with their Certificate IV in Mental Health. The traineeships were hosted in community managed mental health organisations across NSW. To participate in the Program, host organisations were required to offer paid employment to Program participants and support them to complete the education and training components of a nationally recognised certificate qualification. The trainees were employed for a minimum of 21 hours per week. At the conclusion of the Program host organisations were expected to offer trainees ongoing employment. Throughout the Program MHCC and AEU provided information and support to individual trainees and their host organisations. They were also responsible for overseeing this independent evaluation of the Program. What follows is the final evaluation of the Program conducted by EJD Consulting & Associates an independent social policy research firm. The evaluation builds on the preliminary evaluation report that was also conducted in late 2013 and focused on the establishment and implementation phases of the Program. * * * * ACIMH- Final Program Evaluation Report Page 1

13 2 BACKGROUND 2.1 Origins of the Program MHCC is the peak body for community managed mental health organisations in NSW. Over recent years, many community managed mental health organisations have indicated to MHCC their interest in a) improving access for Aboriginal service users who have mental health issues and b) employing more Aboriginal staff to work in the sector. A key strategy for attracting Aboriginal people to use mental health services is increasing the number of Aboriginal mental health workers employed. Since 2000 state and federally funded initiatives have seen a significant increase in the demand for appropriately qualified Aboriginal mental health workers. These have included programs such as the NSW Housing and Support Initiative (HASI) and the Commonwealth Personal Helpers and Mentors (PHaMs) Programs. According to the Australian Bureau of Statistics, nearly one-third (31%) of Indigenous adults (aged 15 years and over) have high to very high levels of psychological distress - more than twice the rate of non-indigenous Australians 2. The Bureau also found that psychological distress rates were higher among Aboriginal women (34%) than Aboriginal men (27%). An informal survey conducted by MHCC in 2010 estimated the percentage of Aboriginal service users accessing community managed mental health services was 11%, with the Aboriginal workforce at approximately 4.9%. MHCC research also found that many community managed mental health organisations report difficulties in recruiting suitably skilled and qualified Aboriginal staff. To address this issue, and in response to advice provided by the MHCC Aboriginal Reference Group, MHCC initiated the development of the ACIMH Program in Australian Bureau of Statistics (2008) National Aboriginal and Torres Strait Islander Social Survey. ACIMH- Final Program Evaluation Report Page 2

14 2.2 Description of the Program On 22 March 2012 the NSW Minister for Mental Health, Kevin Humphries MP, launched the ACIMH Program at the National Centre for Indigenous Excellence. The Program focused on supporting community managed mental health organisations across NSW to host a total of 50 new traineeships for Aboriginal people. The organisations were assisted to recruit a suitable applicant and then work in partnership with MHCC to provide the applicant with the required skills and training to commence a career in the mental health sector. The Program was coordinated by MHCC and funded by the NSW Department of Education and Communities (DEC) and the Commonwealth Department of Prime Minister and Cabinet (PM&C) 3. MHCC was responsible for the overall design, development, implementation and evaluation. As a registered training organisation (RTO), MHCC was also responsible for delivering the Certificate IV in Mental Health training. This occurred through coordinating and running block training sessions, combined with overseeing inter-session assignments and work-based activities. Following the Program launch, MHCC invited interested community organisations to complete an Expression of Interest (EOI) form. The Program was specifically directed towards organisations keen to develop their capacity to recruit, support and retain Aboriginal staff. In all instances these organisations also shared the broader aim of improving engagement with Aboriginal service users and strengthening their links to Aboriginal communities more generally. Applicants were asked for information on: the number of trainees the organisation thought they were able to support additional information to demonstrate how the organisation met a range of selection criteria (see Appendix 1). Program applications closed on 27 April Following an independent review process, ten organisations were selected to participate in the Program. In this report these organisations are referred to as 3 on pages lists the names of the successful host organisations and the number of trainees they hosted. 3 In the Commonwealth funding agency was the then Department of Education, Employment and Workplace Relations. ACIMH- Final Program Evaluation Report Page 3

15 In addition to offering the host organisations assistance with training costs and some salary subsidies (see Appendix 2 for details), the Program offered the trainees and their supervisors access to support and resources. This assistance included: access to an Aboriginal mentor (funded and delivered by NSW Aboriginal Employment Unit (AEU) within DEC) open telephone and contact with MHCC program staff to provide advice and assistance on request help with developing an Aboriginal Employment Strategy and Policy. MHCC also assisted with identifying potential trainees, although most organisations initiated their own local recruitment process. By mid-2013, 46 Aboriginal trainees had been successfully recruited to Program. The trainees were divided into three groups Groups A, B and C. Each group attended different block training sessions. There were four week-long block training sessions held for each group. All training took place in Sydney, with trainees travelling to attend and being accommodated overnight where necessary. The first Group A training began on 19 November The last Group C training concluded 16 May A graduation ceremony for all trainees took place in Sydney on 29 August Section 4.2 provides details of the number and profiles of those who participated in the Program. Further information on other aspects of the Program follows below. 4 5 Note: The last block was brought forward in response to trainees requests to shorten the time frame between blocks, and also to graduate earlier. The original program had the graduation ceremony planned for 10 October see note above. ACIMH- Final Program Evaluation Report Page 4

16 2.3 Program Aims The Program was established to provide community managed mental health organisations with an opportunity to build their Aboriginal workforce, improve engagement with Aboriginal communities and clients, and build their cultural capacity. As a workforce development initiative the Program had five defined objectives: 1) Increase Aboriginal employment in the community managed mental health sector 2) Build the capacity of community managed mental health organisations to employ and support Aboriginal workers 3) Build the capacity of community managed mental health organisations to engage with and better support Aboriginal communities and people 4) Assist Aboriginal people to develop the skills and requirements needed to work in the community managed mental health sector 5) Increase Aboriginal engagement with vocational training. 2.4 Intended Outcomes In establishing the Program, MHCC oversaw the development of a comprehensive evaluation framework 6. This included a detailed set of intended outcomes for each stakeholder group as follows: Host Organisations: improved internal policies and practices in terms of Aboriginal employment improved internal policies and practices in terms of Aboriginal client engagement and retention improved capacity to support Aboriginal communities and people greater staff awareness of Aboriginal issues and improved cultural competence in general, a more culturally inclusive work place in general, participation in the Program proved beneficial to the organisation and its staff. 6 EJD Consulting & Associates, MHCC Aboriginal Careers in Mental Health, Evaluation Framework, ACIMH- Final Program Evaluation Report Page 5

17 Aboriginal Trainees: improved education, training and qualifications improved employment opportunities improved confidence and capacity to work in mental health in general, participation in the Program proved beneficial on a personal and professional basis. Aboriginal Mentors: supported trainees to complete their training and increase their qualifications supported trainees with employment opportunities built trainee confidence and capacity to work in mental health in general, supported trainee participation in the Program resulting in improved personal and professional outcomes. The Program evaluation also identified a number of flow-on outcome measures related to external interests and stakeholders. They included: increased Aboriginal employment in the community managed mental health sector in NSW built the capacity of community managed health organisations to: employ and support Aboriginal workers, and engage with and better support Aboriginal communities and people assisted Aboriginal people to develop the skills and requirements necessary to work in the community managed mental health sector increased Aboriginal engagement with vocational education in other ways, fulfilled each of the Program aims and objectives in general, was a positive initiative in terms of: Aboriginal workforce development, and was well run and managed. Each of the above measures underpin the findings and analysis contained in Section 4 and following. ACIMH- Final Program Evaluation Report Page 6

18 2.5 Roles and Responsibilities The Program establishment documentation clearly defined the roles and responsibilities of each stakeholder Host Organisations Details on host organisation obligations, including in relation to various stakeholder groups and program participants, can be found in the ACIMH Service Agreement at Appendix 1. Topics covered include: Administration and support Training and workshops Fees and funding Reporting framework and evaluations Mental Health Coordinating Council Similarly, details on MHCC obligations, including in relation to various stakeholder groups and program participants, can also be found in the Service Agreement at Appendix 1. Topics include: Support and training services Resources and funding Cultural awareness/ competency Overall project management and Program coordination. An assessment of each of above roles is included in the findings sections that follow. 2.6 About the Program Evaluation Evaluation Framework As the ACIMH Program was the first comprehensive Aboriginal workforce development initiative that had been undertaken by the NSW community managed mental health sector, MHCC and AEU were keen for a thorough review of Program inputs, outputs and outcomes to be conducted using a rigorous and independent evaluation process. EJD Consulting & Associates a specialist social policy and human services research firm was selected in 2012 to oversee the evaluation. This included ACIMH- Final Program Evaluation Report Page 7

19 developing a comprehensive evaluation framework and preparing various reports as part of an action research approach. The key aim of the evaluation was to assess the achievements of the Program against its stated objectives and intended outcomes (see Sections 2.3 and 2.4) and to identify options for how Aboriginal workforce development initiatives, particularly within the community managed health sector, could be improved upon in the future. The EJD Consulting Evaluation Framework was prepared and approved for implementation in April It outlined a number of stages and products as listed in Table 1 below. This is available from MHCC on request. Table 1: Evaluation Stages and Products Product Stage 1: Literature Review plus Baseline Sector Report Details A scan of reports, evaluations and other literature related to Aboriginal or first people workforce development and training programs in NSW, other parts of Australia, New Zealand and select states and provinces in North America. Report on the findings of a sector-wide survey on opportunities and attitudes to Aboriginal employment and training opportunities in community managed mental health services in NSW. Completion Date June 2012 July 2012 Stage 2: Mid Project Review Report Document and assess implementation process to date; identify ways to strengthen the Program as it progresses. EJD Consulting Report titled Evaluation December 2013 Stage 3: Final Evaluation Report (including follow-up sector report) Major product of the evaluation.* November 2014 (This report) Source: EJD Consulting & Associates, * In 2012 it was intended to include a follow-up sector survey as part of the final report. In mid MHCC resolved to delay this activity until a later date. ACIMH- Final Program Evaluation Report Page 8

20 2.6.2 Evaluation Terms of Reference The agreed terms of reference for the Program evaluation were to: 1. Review the inputs, outputs and outcomes of the ACIMH Program, specifically in terms of the aims and objectives 2. Review both the effectiveness of the Program model and the direction and support available to Program participants, including the host organisations, trainees, and Aboriginal Mentors. 3. arrangements, including its resourcing, training, communication and reporting processes. 4. Identify options and make recommendations to strengthen or build on the Program in the future Methodology The Stage 3 final evaluation involved a mix of data gathering techniques. These included: Updated targeted literature review (see Section 3) Review of all program documentation, including online material and the Yarn Up newsletters Survey Monkey surveys independently administered to all Program stakeholders approximately 120 individuals. A total of 62 responses were received representing a 52% response rate. This comprised: Trainees (23) Host organisation supervisors (26) Mentors (4) Block release trainers (3) MHCC Program staff and other key stakeholders (6). Full data sets are available on request from MHCC. Focus groups held with: Trainees In total 8 trainees participated across three focus groups held via telelink-up. Host organisation staff, including trainee supervisors and service Coordinators ACIMH- Final Program Evaluation Report Page 9

21 In total 7 host organisation staff participated across two focus groups held via telelink-up. Program staff Four MHCC staff and consultants participated in this face-to-face focus group held in August Block Training Survey data This final report contains the results of 25 trainee surveys gathered at the conclusion of the final Block 4 training. The results are compared to earlier training data gathered during Block 1 and Block 2 (See Section 5). Exit interviews conducted Seven survey-based exit interviews were conducted by MHCC Program staff with trainees who left the Program early and prior to graduation. Additional written feedback was provided by another exited trainee. Together this represents 67% of all exited trainees. Interviews and Case Studies EJD Consulting also conducted a total of 8 interviews with trainees, supervisors, mentors and other external stakeholders. Three of these interviews resulted in trainee case studies included in this report, in highlight boxes Three resulted in host organisation case studies also included in the report. Included at Appendix 2 are further details about the Evaluation methodology, including the key measures About this Report This final evaluation report is divided into the following sections: Section 3 contains a summary of the results of the independent literature review conducted into Aboriginal trainee projects around Australia in mid and updated for this report Section 4 contains the key program inputs and outputs Section 5 contains findings related to the Program training Section 6 contains findings on trainee outcomes and perceptions of the Program Section 7 contains findings on host organisation outcomes and perceptions of the Program Section 8 contains findings related to the cultural appropriateness of the Program ACIMH- Final Program Evaluation Report Page 10

22 Section 9 contains findings related to the Program administration and management Section 10 contains the conclusion and the results on the key outcome measures. In this report: the community managed mental health organisations who employ the trainees are referred to as host organisations program participants refers to both host organisation staff and trainees all individuals who provided feedback to this evaluation are referred to as stakeholders individuals who completed the Survey Monkey survey are referred to as respondents. Respondent groups refers to the four groups of stakeholders who responded: o trainees (T) o host organisation staff (H) o mentors (M) o all other stakeholders, including Program managers (S). direct quotes from individual stakeholders are included throughout the report indicated by italics. As all feedback was provided on the basis of anonymity, no individuals or organisations are referenced by name, including in the case studies. * * * * ACIMH- Final Program Evaluation Report Page 11

23 3 LITERATURE REVIEW This section summarises the results of an updated online scan of literature focused on the following topics: prevalence of mental health disorders within the Aboriginal and Torres Strait Islander population and their corresponding need for mental health services level of participation in the NSW mental health workforce by Aboriginal people barriers to Aboriginal participation in employment generally the value of traineeships and a jurisdictional scan of traineeships across Australia and in some overseas countries some common elements and good practice guidance found in existing traineeship programs. Other reports relating to these topics have also been prepared for MHCC by EJD Consulting & Associates 7. They are available on request. 3.1 NSW Mental Health Workforce A key goal of the Aboriginal Careers in Mental Health Program is to increase the participation of Aboriginal people in the NSW mental health workforce. This section explores why such a program is needed Mental Health Status of the Aboriginal Population A major collaborative publication by the ABS and the Australian Institute of Health and Welfare, The Health and Welfare of Australia's Aboriginal and Torres Strait Islander Peoples 2011, provides the following information about the social and emotional wellbeing 8 of Indigenous Australians 9 (AIHW 2011): 7 8 A thorough scan of reports and literature related to the ACIMH Program was undertaken by EJD Consulting & Associates in June This culminated in a report, in two parts, being submitted to MHCC containing summaries of the literature together with annotated references by jurisdiction. This scan was the basis of the literature review contained in the Preliminary ACIMH Program Evaluation Report submitted in late What follows in this section is an updated review of literature conducted in September An updated, annotated scan of literature has also been prepared and submitted to MHCC. It is available on request. All sources referenced in this section are listed at the end of this report. The concept of social and emotional wellbeing is based on the Aboriginal and Torres Strait Islander holistic view of health that includes mental health, but also considers the impact of other factors on emotional wellbeing, such as life stressors, removal from family, discrimination and cultural identification (AIHW 2009). ACIMH- Final Program Evaluation Report Page 12

24 The majority of Aboriginal and Torres Strait Islander adults reported feelings of positive wellbeing, particularly in remote areas (2008 National Aboriginal and Torres Strait Islander Social Survey). However, nearly one-third (31%) of Indigenous adults (aged 15 years and over) had high/very high levels of psychological distress more than twice the rate for non-indigenous Australians. These rates were higher among women (34%) than men (27%) (2008 National Aboriginal and Torres Strait Islander Social Survey). Indigenous Australians were hospitalised for care involving mental and behavioural disorders at three times the rate for other Australians (National Hospital Morbidity Database ). Indigenous young people were hospitalised for mental and behavioural disorders at almost twice the rate for other young Australians. The leading causes were schizophrenia, alcohol misuse and reactions to severe stress (National Hospital Morbidity Database ). Mental disorders were one of the two leading causes of the disease burden in the Indigenous population 10 (2003 data sourced from Vos et al. 2007). The rate of service contacts with community mental health services by Indigenous Australians was almost three times the rate of non-indigenous Australians (731 per 1000 and 254 per 1000 respectively) (National Community Mental Health Care Database ). More recent data shows that in , the age-standardised community mental health service contact rate for Indigenous Australians (1008 per 1000 population) was 3.7 times the rate for non-indigenous Australians (273 per 1000 population) (National Community Mental Health Care Database ). Based on the above findings it is likely that the need for mental health services is disproportionately high among the Aboriginal population. Added to this, there is some indication that Aboriginal people prefer to use Aboriginal rather than mainstream mental health services. A University of Sydney study of Aboriginal women with co-morbid drug and alcohol and mental health problems found that was not universal however, as some of the women raised confidentiality issues with seeking treatment from members of their own community (University of Sydney 2010) Due to problems of under-identification of Indigenous clients, hospital and community mental health service figures are likely to be under-estimates (ABS & AIHW 2008). The disease and injury burden for Indigenous Australians was assessed using Disability Adjusted Life Years (DALYS): the sum of years of life lost due to premature death and years lived with disability. ACIMH- Final Program Evaluation Report Page 13

25 3.1.2 Aboriginal Participation in the NSW Mental Health Workforce A review of higher education access and outcomes for Aboriginal and Torres can both respond to the high priority needs of their own communities and make a deeper contribution to the wellbeing and prosperity of the nation through their diverse v participation in the Aboriginal Careers in Mental Health program results in a Certificate IV qualification, rather than a degree, increasing the number of Aboriginal people with this level of qualification is likely to have similar benefits. Detailed data about the number of Aboriginal people employed in the mental health workforce is limited; however, some useful information was found in a 2013 review of Australian Government health workforce programs. The review compared occupational data from the 2011 Census to that from the previous Census in 2006 and found a significant (47%) increase in the total number of Aboriginal and Torres Strait Islander people in the Australian health workforce (Mason 2013). Despite this increase, at 1.8% Aboriginal and Torres Strait Islander people remain under-represented in the health workforce when compared with their 3% representation in the Australian population as a whole. Exclusion of the majority Aboriginal workforce of 91.5%) leads to an even lower overall figure of 1.65% representation. Among the health related occupations canvassed in the Mason review, he fourth highest percentage increase (98%) between 2006 and 2010 in Aboriginal and Torres Strait Islander people employed (Mason 2013). The analysis provides further information about Aboriginal and Torres Strait Islander people working in each health profession compared with non-aboriginal and Torres Strait Islander people. Two mental health professions are mentioned specifically, with eight psychiatrists identifying as Aboriginal or Torres Strait Islander in the 2011 Census, representing 0.31% of all psychiatrists; and 85 psychologists, representing 0.46% of all psychologists (Mason 2013). These figures indicate an even lower level of Aboriginal representation in these mental health professions than in the health workforce as a whole. ACIMH- Final Program Evaluation Report Page 14

26 The under-representation of Aboriginal and Torres Strait Islander people in the health workforce is cited as contributing to potentially lower rates of Aboriginal and Torres Strait Islander people accessing health services (AHMAC 2012). Increasing the rates of participation and completion of training by Aboriginal and Torres Strait Islander people in the Australian health workforce is therefore fundamental to achieving better health outcomes. While specific data about participation rates of Aboriginal people in the community managed mental health sector was not found, the Aboriginal Careers in Mental Health Program plays a part in addressing the general need for greater participation by Aboriginal people in this sector. The Program seeks to build the capacity of community managed mental health organisations to employ and support Aboriginal workers, enabling them to engage with and better support Aboriginal communities and people. While fulfilling these aims, the Program is also helping Aboriginal people develop the skills needed to work in the community managed mental health sector and increasing their engagement with vocational training (MHCC 2012). 3.2 Aboriginal Employment and Educational Attainment This section will discuss the current situation with regard to employment rates of Aboriginal people in NSW, and the importance of educational attainment for positive employment outcomes Employment Rates of Aboriginal People The 2011 Census revealed that the unemployment rate 11 among the Indigenous 12 population of NSW was 16.9%, compared with 5.7% among those identified as non- Indigenous (ABS 2012b). This means that the unemployment rate among the -Aboriginal people. Labour force participation 13 was correspondingly lower for Indigenous individuals (51.2%) than for the non-indigenous (62.6%) (ABS 2012b). Among young Aboriginal people (15 24 years) living in NSW in 2008, 53.7% were either employed full-time, studying full-time, or both employed and studying parttime (ABS 2010). This compares with the 83.2% of non-indigenous young people who were fully engaged in work and/or study Number of unemployed persons expressed as a percentage of the total labour force. Terminology as per ABS data - see above. Number of persons in the labour force expressed as a percentage of persons aged 15+ years. ACIMH- Final Program Evaluation Report Page 15

27 3.2.2 Importance of Educational Attainment Data from the 2011 Census reveals a significant difference in levels of educational attainment for Indigenous individuals versus non-indigenous individuals in NSW. Only 24% of Indigenous people surveyed had completed Year 12, compared with 52% of non-indigenous respondents (ABS 2012a). Likewise, only 26% of Indigenous individuals indicated they had a post-school qualification, compared to 46% of non-indigenous respondents (ABS 2012b). Table 2: Educational attainment by labour force status, persons aged years (2008 data) Educational attainment Employment rate Unemployment rate ATSI Non-Indig. ATSI Non-Indig. Highest year of school Year 9 or below 79.2% 93.0% 20.8% 7.0% Year 10/ % 95.8% 11.6% 4.2% Year % 96.8% 6.7% 3.2% Non-school qualifications Certificate III or above 93.1% 97.1% 6.9% 2.9% Without non-school qualification 83.8% 94.8% 16.2% 5.2% All persons aged % 96.2% 12.6% 3.8% Source: 2010). Data from the Islander Peoples study (ABS 2010) shows a marked correlation between educational attainment and labour force status for Aboriginal and Torres Strait Islander people (ATSI) 14. As documented in Table 2, the unemployment rate for indigenous individuals who had completed Year 12 was 6.7%, while that for individuals who had attained only Year 9 or below was 20.8%. Individuals who attained Years 10 or 11 had an unemployment rate of 11.6%. 14 Note: In this data series ABS uses the term and acronym ATSI to refer to indigenous people. ACIMH- Final Program Evaluation Report Page 16

28 Corresponding data for non-indigenous individuals reveals that educational attainment is not the only factor influencing employment outcomes for ATSI individuals however, as unemployment rates for non-indigenous individuals, while also showing a correlation with educational attainment, were much lower and the correlation was much less marked. Of particular relevance to the ACIMH Program is the data in Table 2 which further shows that the attainment of non-school qualifications such as TAFE certificates at Level III and above significantly improves the employment prospects for Indigenous people; those without non-school qualifications had an unemployment rate of 16.2%, while the unemployment rate for those with TAFE qualifications and above, fell to 6.9%. The benefits of being employed also flow on to improve the educational attainment of the next generation. Research shows that children who have a parent with a job are more likely to attend school and stay on past the compulsory school age. They are also more likely to enter higher education courses (Productivity Commission 2011). In this way, educational attainment and subsequent improved employment prospects create a cycle which can increase the participation of Aboriginal people in the economy and hence their opportunities for self-development, increased income and improved standard of living. 3.3 Aboriginal Traineeships This section examines the value of traineeships for both the trainee and the participating employer. Internet research was also conducted to identify other Aboriginal traineeship and cadetship programs similar to ACIMH across all jurisdictions of Australia and in some overseas countries Value of Traineeships A traineeship allows the participant to combine a job with a structured training program that leads to a nationally recognised qualification. Participants who complete the ACIMH Program obtain a Certificate IV in Mental Health. Because traineeships offer real workplace experience and practice, participants can be sure they are developing the skills they need to be effective in their new career. Traineeships also offer a supportive environment, assisting trainees to gain their formal qualifications; for example, by allowing flexibility so trainees can attend training classes. ACIMH- Final Program Evaluation Report Page 17

29 Organisations employing trainees often receive government assistance in the form of trainee salary subsidies or help with training costs. This can make a traineeship a very cost effective way for an employer to recruit and train a suitable applicant. This is particularly useful in areas of employment where suitably skilled and qualified Aboriginal applicants may be difficult to obtain. The ACIMH Program also offers host organisation employers additional support and resources such as help in developing an Aboriginal Employment Strategy and Policy, and assistance in identifying potential trainees (MHCC 2012). In this way the program builds the capacity of employers to recruit and retain Aboriginal workers Cross-Jurisdictional Scan of Traineeships A large number of traineeship programs for Aboriginal people were discovered across all jurisdictions within Australia, plus a number of programs for similar population groups in overseas countries. A list of programs found is provided in a separate Attachment Report submitted as a companion document to this report. Traineeship programs can be sorted into two major groups: public sector funded and driven programs, and private or community sector programs. The latter group often also receive government funding assistance. By far the highest proportion of all Aboriginal traineeship programs were public sector funded and driven. Many had the aim of increasing Aboriginal participation in the public sector itself, including local government. Examples of public sector Traineeship Program, the Aboriginal Trainee Environmental Health Program (ATEHO), and the Aboriginal Communities Water and Sewerage initiative. The ACIMH Program also falls into this category. Large Commonwealth-funded programs include the Indigenous Australian Government Development Program (IAGDP) and the Indigenous Youth Career Pathways (IYCP) Program. Aboriginal traineeship programs arising from the private or community sectors were less numerous but nonetheless make a significant contribution. Examples from NSW include the Aboriginal traineeships in community housing provided by the NSW Federation of Housing Associations, and the Regional Aboriginal Project for Traineeships (RAPT) run by the Ballina District Community Services Association. Several significant Australia-wide traineeship programs are offered by private companies such as major banks and mining corporations. Overseas traineeship programs are detailed in (see companion EJD Attachment Report, 2014) and an attempt has been made to match them with similar NSW or ACIMH- Final Program Evaluation Report Page 18

30 Australia-wide program(s). These overseas programs were found to have a lot in common with Australian programs in their broad goals and the strategies used Measuring Outcomes from Aboriginal Traineeships It would appear that very few of the traineeship programs found through the cross-jurisdictional scan of traineeships (listed in the separate Attachment Report) have been the subject of a formal evaluation. Key evaluations which were found and considered for the purposes of this literature review included: Review of Australian Government Health Workforce Programs (Mason 2013) Review of higher education access and outcomes for Aboriginal and Torres Strait Islander People (DIISRTE 2012) NSW Government Employment and Economic Development Programs for Aboriginal People: Review of programs and broader considerations (Allen Consulting Group 2011) NSW Aboriginal Mental Health Worker Training Program: Implementation Review (Watson & Harrison 2009). 3.4 Good Practice in Aboriginal Traineeships The National Centre for Vocational Education Research (NCVER) commissioned a systematic review of research on good practice in culturally appropriate VET and adult and community education (ACE) for Aboriginal Australians (Miller 2005). While the review did not specifically deal with traineeships, its conclusions are still relevant in this context. The Miller review concluded that there are seven key factors which lead to positive and improved outcomes for Aboriginal people as a result of vocational education and training (Miller 2005): community ownership and involvement the more authority communities are able to exercise in all aspects of training development and delivery, the more successful the training will be for Aboriginal students incorporation of Aboriginal identities, cultures, knowledge and values recognition provides a space in which Aboriginal people feel comfortable with learning (Marika et al. 2004) strong multilateral partnerships, with communities at the centre, provide a means through which human and financial resources are shared, to reduce duplication of effort flexibility in course design, content and delivery allows for the easy adaptation of training programs to meet local needs and contexts quality staff and committed advocacy students reported that the most important aspect of effective training for them was their relationship with their teachers (Kemmis et al. 2006) ACIMH- Final Program Evaluation Report Page 19

31 extensive student support services a range of support services are required across financial, cultural and social needs, to help students overcome barriers to participation in higher-level VET courses and to retain them in the system appropriate funding that allows for sustainability initiatives that are shown to be effective must be backed up with long-term funding, to avoid perpetuating sporadic and ad hoc practices. Drawing on research of this kind, MHCC provided the following advice to employers participating in the ACIMH Program, to ensure they provided a safe and supportive environment for trainees (MHCC 2012): Provide appropriate supervision at all times. For example, trainees should not be expected to conduct solo home visits when they are starting out. Provide opportunities which expose trainees to a range of mental health work, making sure duties are meaningful and relevant. Encourage and support trainees with constructive feedback provided in a timely and culturally appropriate way. Work collabora receiving appropriate cultural support. Make sure some work time is set aside for the trainee to meet with their mentor. Allow the trainee 2 3 hours of work time a week to devote to their studies and spend time with a supervisor who can answer any questions they have. Support the trainee to attend relevant training opportunities, including all training towards their achievement of the Certificate IV in Mental Health. Ensure supervisors allocated to support the trainee are suited to the role and have sufficient time to provide the support the trainee needs throughout the traineeship. Organisations participating in the ACIMH Program were encouraged to develop an Aboriginal Employment Strategy, and MHCC and AEU provided support and advice about this. MHCC also convened employer network meetings to encourage networking, support, cross-collaboration and general communication about the program and any issues arising. Participating employers were strongly encouraged to take part in these meetings (MHCC 2012). With regard to successful recruitment of trainees, MHCC provided a number of ideas for potential employers seeking suitable candidates (MHCC 2012): Provide a pre-employment course of 3-4 weeks duration for a group of potential candidates so both sides can see if they are suited to the area of work those remaining at the end can undergo a further selection process. Provide a 2-3 day taster course to introduce potential candidates to the mental health sector, including some workplace visits. ACIMH- Final Program Evaluation Report Page 20

32 Make use of a range of Aboriginal media to advertise traineeships. Liaise with a New Careers for Aboriginal People (NCAP) officer to identify potential trainees. Liaise with the Indigenous specialist at your local Job Services Australia office or other local employment service providers to identify potential trainees. * * * * ACIMH- Final Program Evaluation Report Page 21

33 Experience Profile Trainee Case Study 1 Megan* have a direct path or a career plan before and Megan* is 32 years of age whose family are Walgadu people from the Tumut area. She spent most of her life in Redfern. She moved to the South Coast six years ago. Megan left school in Year 10 and worked for many years in various retail positions. She eventually trained as a childcare worker. Before commencing with the ACIMH Program Megan was working as an Indigenous mentor in the local high school. She read about the option of becoming a trainee in the local paper. Her mother works as an Indigenous Welfare Officer with a NSW Government agency. She has had a huge and positive influence Megan was successful in being recruited as an Aboriginal trainee working in a PHaMS program with a large mental health NGO. There were 15 staff in her regional office; one part-time Aboriginal colleague joined the team towards the end of her traineeship. Megan was the only trainee in the team. Megan said the workplace was very welcoming and supportive throughout the Program. They frequently sought her opinion on Aboriginal client issues and ask if she thought they should be trying alternative engagement options. She really appreciated Her first challenge in taking the position was getting used to working in an office environment. I never had had to deal with s and reports and office stuff before. It took a while to get my head around it and what was A second challenge was being on her own as an Aboriginal worker and also trying to promote a new service in the area. s Megan said having another Aboriginal worker, especially one with good community links, would have made the service promotion and outreach activities easier. Continued ACIMH- Final Program Evaluation Report Page 22

34 Future Plans Experience Trainee Case Study 1 (continued) Megan reported travelling for block training was hard on the family. However she saw personal benefits from being away and was able to receive support from her peers not only in the course breaks, but also after hours. support and to be with people who understand the juggling we all were doing with work and study and family and At the outset of the Program there were approximately 100 clients, of which only 3 or 4 were Aboriginal. By the end there were 12 or 15 Aboriginal clients, which she is proud to have had a role in. ed the streets and She indicated it was very satisfying to see the outcomes and the fact that she has had an impact on her colleagues and the service in terms of it being accepted by the community. Megan commented on how much she had grown from being part of the ACIMH Program. She specifically commented on how the experience had changed her understanding of and compassion for people with mental illness and other complex needs. ended up in those places because of their own decisions and support from people who Megan has signed a 12 month contract to continue working with the same mental health NGO. She will be employed as a general PHaMS worker, assisting both Aboriginal and non-aboriginal clients. She feels as though the local community s and would refer clients. Megan has also enrolled with Griffith Open University to commence her undergraduate degree in Psychology, starting from September This is the first step in an 8-year plan that forensic psychologist working in the prison system. I have made loads of * Not her real name * * * * ACIMH- Final Program Evaluation Report Page 23

35 4 FINDINGS: INPUTS AND OUTPUTS This section summarises the key ACIMH Program inputs and outputs as at September 2014 the end of the formal phase of the ACIMH Program. Unless otherwise listed, MHCC consolidated program data is the source of all figures used. 4.1 Program Numbers Host Organisations In 2012 ten community managed mental health organisations across NSW agreed to participate in the Program and host the Aboriginal trainees. While the Program plan specified up to 15 participating organisations, the overall participation rates were considered very positive. Each organisation nominated to host a specific number of trainees, ranging from one through to fifteen trainees, based at specified service locations. At the end of the recruitment phase, trainees were based in 31 different service locations. Table 3 lists the name of each host organisation, the service location, and the total number of trainees hosted over the course of the Program. Notwithstanding changes in host arrangements over time (as documented below Table 3), at any one point in time: 21 service locations hosted a single trainee o Representing 46% of all trainees (21 individuals) based in locations without other Program trainees present 9 service locations hosted two trainees o Also representing 46% of all trainees (21 individuals) based in locations with one other Program trainee present o 3 of these locations replaced their initial trainee with new trainee recruits service location (Neami in Dubbo) co-hosted four trainees at any one time o Representing 8% of all trainees. Analysis related to co-locating trainees is included at Section These were Benevolent Society, Rosebury; Neami, Ashfield; and Uniting Care Mental Health, Parramatta. Whilst Table 3 lists 3 trainees in each instance, only two were in situ at any one point in time. ACIMH Program- Final Evaluation Report Page 24

36 Table 3: Total Number of ACIMH Trainees per Host Organisations and location Host Organisation Location Number of Trainees Aftercare Kingswood (Sydney) 1 Lilyfield (Sydney) 1 Anglicare Bondi (Sydney) 1 North Ryde (Sydney) 1 Benevolent Society + Hurstville (Sydney) 2 Rosebery (Sydney) + 3 Junaa Buwa (Mission Australia) Coffs Harbour 1 Lyndon House Orange 1 Ashfield (Sydney) # 3 Broken Hill 2 Neami Campbelltown (Sydney) 2 Carrington (Newcastle) 2 Darlinghurst (Sydney) 2 Dubbo 4 Charlestown (Newcastle) 1 Gosford 2 New Horizons Marrickville (Sydney) 1 Miller (Sydney) 1 Port Macquarie 1 Tweed Heads 1 ACIMH Program- Final Evaluation Report Page 25

37 Host Organisation Location Number of Trainees Bourke* 1 Dubbo* 1 Richmond PRA* Homebush (Sydney) 1 Newcastle 1 Nowra 1 Tamworth* 1 Dubbo 1 Schizophrenia Fellowship of NSW Moruya 1 Nowra 1 Uniting Care Mental Health^ Parramatta A (Sydney)^ 3 Parramatta B (Sydney)^ 1 Total: Source: MHCC Aboriginal Workforce Development Strategy, September Note: The figures indicate where trainees started their traineeships- see footnotes below. + An initial trainee was replaced with a new trainee at the Rosebury service. As such only two trainees were in situ at any one time. # An initial trainee was terminated and replaced with a new trainee at the Ashfield service. As such only two trainees were in situ at the same time. * During the Program, Richmond PRA lost some core Government funding. As a result the Bourke, Dubbo and Tamworth traineeship positions were unable to be sustained in those services: ^ o The Bourke trainee resigned; the Dubbo trainee was transferred to a NEAMI service in the same location; and the Tamworth trainee was transferred to a New Horizons service in the same location. An initial trainee with Uniting Care Mental Health was replaced with a new trainee. As such only two trainees were in situ at the same time in one of the Parramatta services. One of the Uniting Care Mental Health trainees while also located in Parramatta, was based in a separate Headspace office and therefore was not strictly co-located. ACIMH Program- Final Evaluation Report Page 26

38 4.1.2 Trainees A total of 46 trainees were recruited into the ACIMH Program. To effectively manage the training and support demands, Program trainees were divided into three groups A, B, and C: 16 trainees commenced in Group A 15 trainees commenced in both Group B and C. The vast majority of trainees remained in the same group and participated in the same designated block training throughout the Program. A small minority of individuals however did change groups and/or attend other block training sessions Overall Results Between the commencement of the block training in late 2012 and the graduation date in late August 2014, all participating host organisations continued their involvement in the Program. Table 4 summarises key Program inputs and outputs as at September The results show: Of the 46 trainees recruited into the Program, 32 completed their qualification. This represents a completion rate of 69.6%, which is particularly positive when compared to general completion rates for NSW students enrolled in Certificate IV and above qualifications (at 44.3%) % of trainees had an early exit from the Program. Of these: o half (6) of the trainees successfully completed four or more units of the 15 units in the Certificate IV prior to exit. One trainee completed 11 units. o 3 trainees left to take up specific positions or alternative study. 17 Other reasons for exiting are analysed at Section 4.3 below. 52% of trainees were offered positions with their host organisations or another organisation after graduation Source: National VET Provider Collection (NCVER 2014). In one instance a Graduate Diploma of Indigenous Health Promotion. ACIMH Program- Final Evaluation Report Page 27

39 Table 4: Final ACIMH Program Trainee Statistics Measure Number of Trainees Percentage of Trainees Commenced in Program % Completed Program and graduated in August % Exited Program (prior to graduation): In 2013: In 2014: % Currently enrolled or employed with possible later graduation date# 2 4% As at September 2014, offered ongoing employment: with same host organisation with other organisation* As at September 2014: enrolled in further study^ have plans to enrol in further study in the near future^^ % 39% Source: Consolidated MHCC data, September 2014 (n=46); analysis EJD Consulting. # As at end September 2014, one trainee had left employment though is expected to complete their Cert IV; another was still employed though not likely to finish Cert IV in near future. * The other employer was Life Without Barriers. ^ ^^ Current study courses include: Diploma of Counselling (2); Diploma in Youth Work; Bachelor of Sports Management; Bachelor of Arts (Forensics and Criminology); and Diploma of Community Services. Planned courses include: Diploma of Community Services; Certificate IV in Training and Assessment; Diploma in Mental Health; Diploma in Youth Work; Bachelor of Nursing/Midwifery; Diploma of Counselling; Bachelor of Psychology or Social Work. ACIMH Program- Final Evaluation Report Page 28

40 o At least three additional trainees would have been offered positions however elected to take up alternative positions or left their host organisations for family reasons (for example, one trainee was pregnant) 39% of trainees were enrolled or intending to enrol in further study. These combined results indicate that the Program fully met key Program aims and expectations, in the process providing an additional 32 qualified Aboriginal mental health workers available to work in the mental health sector in NSW. 4.2 Trainee Profiles What follows is a profile of trainees who participated in the ACIMH Program drawn from MHCC trainee data files. All 46 trainees identified as being Aboriginal One trainee also identified as being a Torres Strait Islander. As Diagram1 illustrates: over three-quarters of all trainees participated in the Program on a full-time basis, at 76% the remaining 24% participated on a part-time basis. Diagram 1: Traineeship by Employment Type Employment Type Part-time 24% Full-time 76% Source: Consolidated MHCC data tables (n=46), September ACIMH Program- Final Evaluation Report Page 29

41 4.2.1 Trainee Location As may be expected based on population and service provider numbers, the highest proportion of trainees was based in the Sydney metropolitan area. As Diagram 2 illustrates: 50% of trainees were based in Sydney (23) 41% were based in non-metropolitan or rural towns (19) 9% were based in another metropolitan centre, namely Newcastle and suburbs (4). Diagram 2: Trainee Location Location Rural/ Non Metro Town 41% Sydney 50% Other Metro Centre 9% Source: Consolidated MHCC data tables (n=46), September Gender and Age As illustrated in Diagram 3, slightly more females than males participated in the Program with: 25 females 21 males. ACIMH Program- Final Evaluation Report Page 30

42 Diagram 3: Trainee Gender Breakdown Gender Females 54% Males 46% Source: Consolidated MHCC data tables (n=46), September Diagram 4: Trainee Age Distribution (in years) Source: Consolidated MHCC data tables (n=46), September 2014 ACIMH Program- Final Evaluation Report Page 31

43 Trainee ages spanned over forty years, with the youngest trainee aged 19 years and the oldest trainee aged 61 years. As illustrated in Diagram 4 over half of all trainees were aged between years, at 59%, with 41% aged between years. The average age of trainees was 29 years; the largest group (mode) of trainees were years Education and Employment Backgrounds As illustrated in Diagram 5, slightly less than half of all ACIMH trainees completed high school, at 45%. Of the remaining: 9% completed Year 11 35% completed Year 10, and 13% reported a maximum level of education at Year 9 or less. Diagram 5: Trainee High School Year Completed Source: Consolidated MHCC data tables (n=46), September ACIMH Program- Final Evaluation Report Page 32

44 As illustrated in Diagram 6, more than half of the trainees reported having no tertiary qualifications prior to commencing in the Program, at 56%. Of the 44% who did report completing a tertiary qualification, the highest obtained as follows: 2% (one trainee) had completed a Certificate IV. 22% had completed a Certificate III course 20% had completed a Certificate II course. Three individuals who reported completing a Certificate III course, also reported completing a second Certificate I or II course as well. Diagram 6: Previous Trainee Tertiary Qualifications Tertiary Qualifications Cert III 22% Cert IV 2% Cert II 20% No Prior Tertiary 56% Source: Consolidated MHCC data tables (n=45*), September * they were not included in the data ACIMH Program- Final Evaluation Report Page 33

45 4.2.4 Previous Trainee Experience During Block 1 training all trainees (n= 43) were asked if they had any previous experience in mental health or experience as a trainee. 9% reported previous experience of working in the mental health sector (4) Approximately one-third (35%) reported they had previous experience as a trainee (15). Traineeship industries and employers varied, for example: o community sector (including aged care and disability) (5) o a trade or security position (2) o business administration and finance (4) o ) o other industries included QANTAS (1); McDonalds (1); the AFL (1); and one unknown. While the combination of the above trainee profile information does indicate a broad cross-section of individuals were recruited into the ACIMH Program, it also indicates that the majority (58%) had previously completed Certificate qualifications with over one-third (35%) having past trainee experience. While these two factors alone may have rates, the evaluators note that the Program design with its intensive support options was also able to deliver positive outcomes for individuals without this prior history. There is potential for these individuals to be further targeted in future Program recruitment. R1. Investigate options for expanding the number of highly motivated trainee recruits who meet the selection criteria though may also have: a) no previous tertiary qualification; and/ or b) no previous participation in a traineeship program. 4.3 Early Exit Trainees As documented in Table 2, of the 46 trainees who commenced in the Program, 12 trainees (representing 26%) left the Program prior to graduation. Over two-thirds of these exits (83%) occurred in 2013, most within the first six months of the Program commencing. As part of the EJD Consulting evaluation methodology, all early exit trainees were invited to participate in a structured exit interview with an MHCC Program staff member. The results were recorded in a standardised exit survey. At the time of ACIMH Program- Final Evaluation Report Page 34

46 the final evaluation a total of 7 exit surveys had been successfully completed and returned to the evaluators: 2 from Group A 3 from Group B 2 from Group C. A further trainee provided written feedback via . Trainees reported a range of prime reasons for leaving the Program. These included: Dismissed by host organisation (2) Offered another/better position (2) Dissatisfaction with workplace (2). However, most trainees reported multiple and compounding reasons for exiting the Program. When consolidated these included: had better position or job to go to (4) dissatisfied or disillusioned with current employer or role (4) personal reasons or family commitments (4) interest in further study (2) wanting a change (2). While personal/family reasons was cited as the prime motive to leave by only one trainee, four early exits referenced significant family issues within the context of the interviews. These included: recent suicide of a sibling death of a parent threats of child removal relating to past matters involving Community Services relative being a client of the service and challenges therein. The majority of exiting trainees commented that they would have remained in the Program if the circumstances had been different. Options cited included: going to another organisation (1) going to Aboriginal run organisation (1) greater flexibility to accommodate for personal/family circumstances (3) ACIMH Program- Final Evaluation Report Page 35

47 greater support from host organisation regarding how to meet expectations and improve job performance (3). Overall, exiting trainee comments were positive about the Program, with all but one reporting their involvement in the Program had: improved their confidence and capacity to work in mental health assisted them in getting another mental health position in the future assisted them in getting another job of any kind in the future increased their chances of undertaking further study in the future. Benefits cited from participating in the Program were very similar to those of continuing trainees, as per the block release feedback (see Section 5). Positives cited included: relaxed, youth-specific training environment skills of the block trainers and MHCC staff learning about and working in mental health working with Aboriginal people earning a decent salary. Exiting trainees were also asked to identify Program challenges and areas for improvement. Responses varied considerably, although more than half indicated dissatisfaction with the type and nature of the work they were given. Instances cited included: insufficient work and being bored and under-utilised time constraints and having insufficient time to speak face-to-face with Aboriginal clients not being encouraged to undertake client assessments delays in being approved for outreach work managers being and trainees needing more access to supervisors and structured feedback. Recommended areas for improvement included: better and more thorough work induction program improvements in mentoring, including quality of support and time available providing more opportunities to work with Aboriginal people improvement to role play exercises (in training) ACIMH Program- Final Evaluation Report Page 36

48 expanding time available (in early blocks) to cover all necessary material. Other challenges identified by exiting Program trainees were similar to those identified by other trainees, for example: lack of constructive and practical support from supervisor shortcomings in communication related to trainee performance and organisation expectations lack of cultural awareness among some colleagues. Exiting trainees provided other useful feedback, much of it affirming of the ir appreciation for MHCC Program staff. For example, all trainees (7) commented that they would definitely recommend other Aboriginal people take up a mental health traineeship. Other final comments included: mental health as a whole, more than just the stereotypes, not just what you see. inue to look for work in this area. interested. e traineeship] and how I loved it. I really want to see if I can keep doing my sl [MHCC Program staff member] for all your ongoing support throughout my traineeship time. Without yourself there would be no chance at all for us to complete things so smoothly and correctly. The MHCC should be really proud of this program. As I said it was the best Based on the exit interviews, the evaluators concluded there was no single cause or pattern as to why trainees exited the Program. Further, the evaluators found no commonality in the broadly representative cross-section of: males and females; age ranges (from 26 to 47 years) part-time and full-time positions rural and metropolitan locations prior educational attainment ACIMH Program- Final Evaluation Report Page 37

49 o o from completed Year 12 to Year 9 or less from those with no tertiary qualifications, to one with a Certificate III. Despite these variations the findings strongly indicate that participation in the Program had been positive for those exiting, at least among those who provided feedback. Where relevant, other exiting trainee feedback has been incorporated into the findings that follow (indicated by [E] for exited trainee). 4.4 Program Documents Appendix 4 list resources MHCC developed and distributed to support the smooth implementation of the ACIMH Program. They include numerous guidelines, reports, forms and other documentation. In addition a section of the MHCC website was dedicated to the Program providing open access to many of the resources, as well as the option to download. As listed in Appendix 4, printed resources were generally targeted to different stakeholders, each promoted with a distinctive cover design and branded as part of the Program. The prime Program resource provided to host organisations was the ACIMH This 64-page document contains a range of practical and easy-to-read advice and information to support and appropriately supervise trainees through the Program. It includes useful information on the units of competency and the assessment tasks. It also includes advice and information on creating a culturally safe workplace and other culturally appropriate practices. This resource is an important legacy of the Program. Notwithstanding recommendations for enhancements discussed later in this report, the handbook could greatly assist other employers of Aboriginal staff including in particular those with staff wishing to complete mental health training concurrent with their employment. To complement these formal resources, MHCC also produced and promoted a Program newsletter under the banner of Yarn Up. Between March 2013 and September 2014, nine newsletters were issued and posted on the website, approximately one every two months. Each newsletter was colourful, between two and six pages in length, and contained various photographs designed to appeal to a wide range of stakeholders, including trainees and host organisations. They contained regular features such as: ACIMH Program- Final Evaluation Report Page 38

50 Trainee profile Information about one of the host organisations Transcribed interview with a trainer or mentor The MHCC website also contains various other Program documents, including various program administration tools such as trainee claim forms. Whilst many of the resources were established at the outset, some were developed and distributed in response to issues arising. For example, the Outreach Assessment Kit was developed by MHCC to help organisations and trainees assess their readiness to conduct home visits. After reviewing available documents, the evaluators concluded that the Program was well supported with a wide range of policy and operational documentation. The resources were also found to be well designed and laid out, and appropriate to the target audience. This finding is strongly corroborated in the various stakeholder feedback contained in Section 9.6. * * * * Based on the above input and output data, the evaluators concluded that the Program was very successful in recruiting and retaining trainees consistent with the Program aims and objectives. Involving ten different host organisations ranging in size, number of outlets, program types and also location (metropolitan, regional and rural), the Program succeeded in involving a good cross section of community based mental health services across NSW. While the total number of trainees recruited (46) is slightly short of the Program target of 50, the intake was nonetheless noteworthy, particularly for a new Program that was designed, developed and promoted from scratch in a very short timeframe. The 70% trainee retention and completion rate, complemented by identified benefits and positive pathways for the vast majority of the 12 early exit trainees, provides further evidence of a highly successful, outcome-oriented program, as further expanded upon in the findings that follow. * * * * ACIMH Program- Final Evaluation Report Page 39

51 Operations Establishment Host Organisation: Case Study 1 -skill The host service is a well-established mental health NGO with strong connections across its non- metropolitan region. The service initially thought they were to receive two ACIMH trainees: one for their Aboriginal HASI program and one for their PHaMS program. When only a single trainee was selected a decision was made they she would work across both programs. The service was primarily interested in hosting a trainee because of the importance of growing the Aboriginal workforce and providing individuals with a pathway into professional and rewarding employment. The Aboriginal HASI unit comprised a Case Manager, four staff and the trainee. All staff in the unit were Aboriginal as are the 8 to 15 clients in the HASI program. During some phases of the traineeship the trainee worked full time in either the HASI or PHaMS programs which provided a good range of experience. For the majority of time however her time was split between the two: three days in PHaMS and two days at HASI. The service benefitted from hosting the trainee in a number of ways, including from having an additional female member of the team able to work with female Aboriginal clients. The challenges encountered were not major though and included upskilling the trainee in the responsibilities of working as part of a team. This included reminders and discussions regarding: o Workplace communications for example letting your supervisor know if running late or unwell o o Boundaries in terms of appropriate language and behaviours with clients, for example not using a mobile phone for personal matters Managing potential risks when working with clients. This required reinforcing essential risk management measures such as: Reviewing files and file notes prior to client visits Working closely with colleagues Thinking ahead and being alert during home visits. As the supervisor observed old head on young though with direction, the trainee developed and grew into the position well. ACIMH Program- Final Evaluation Report Page 40

52 Impacts & Advice Host Organisation 1- Case Study (Continues) The service and individual staff benefited from having the trainee including by her providing and from providing staff with opportunities to The traineeship was also identified as a useful mechanism to have extra staff about, as well as a. Lessons learnt from supervising the trainee that could be useful to future host organisations: o o o o o Establish clear boundaries early in the traineeship Ensure the trainee is exposed to good role models and has opportunities to learn directly from experienced staff Treat trainees like you would expect to be treated yourself: such as ca Always insist on professional behaviours and professional communications The service has an ongoing and strong commitment to providing culturally appropriate services to Aboriginal clients and sees the best or only mechanisms for achieving this, through having Aboriginal staff deliver the service. The supervisors indicated they would definitely like to host another Aboriginal trainee given the importance of providing young people with opportunities. * * * * ACIMH Program- Final Evaluation Report Page 41

53 5 FINDINGS: TRAINING 5.1 About the Training The ACIMH Certificate IV training took place over four blocks. Each Group (A, B and C) participated in separate block training with sessions staggered in time. The first Group A block training took place in November The last Group C block training took place in May Each block occurred over a week period (5 days) in Sydney. Trainees who were based outside of the metropolitan area were assisted with travel and accommodation costs. Most of these stayed in the same hotel for the duration of the block. The ACIMH training program used a two-trainer model for the duration of the Program, with each group having an Aboriginal and a non-aboriginal trainer. Bradley Foxlewin trained all three groups. The other trainers who worked in partnership with him were: Justin Noel for Group A Brett Davis for Group B Thomas Franks for Group C 18. In addition, MHCC Program staff and other consultants regularly delivered specific training sessions or ran discussion groups. As part of the action research model, towards the end of each block session 19, trainees were asked to complete an anonymous standardised survey as part of the evaluation process. A report of the results of each training survey was prepared by EJD Consulting and provided to MHCC to inform their future planning. In addition trainees provided feedback on the content of the training that was used directly by the trainers and Program staff. This second source of feedback was not part of this independent evaluation Thomas also trained Group B on two occasions when Brett was unavailable. No surveys were administered at the end of block training 3. ACIMH Program- Final Evaluation Report Page 42

54 5.2 Block release training feedback When stakeholders were asked to rate the block release training at the conclusion of the Program 20 (n= 62) the overall feedback was very positive. For example, 82% of all respondents rated the usefulness of the training as good or very good. Importantly, trainees rated the training as the most useful of all respondent groups, with 100% rating it as useful or very useful. Examples of descriptors used by trainees in respect to the training included:,,,, and When asked what were the best aspects of the training, trainee comments included: not only my life but my Gaining the knowledge and education to fulfil my duties [and do] facilitated in such a way that they were productive and informative new skills, having empathy for people, better When trainees were asked to identify any challenges or concerns encountered in the Program training some trainees left the field blank, while others focused on difficulties encountered in balancing study and family commitments. These comments included:. Other trainee comments on challenges related to training included: 20 Based on final Survey Monkey survey administered in July-August 2014 (n=62). ACIMH Program- Final Evaluation Report Page 43

55 e, When. anything that would improve the block, a common response across the three key respondent groups (namely trainees, host organisations and mentors) was the need for block release training sessions to be held closer to each other. Another common response amongst trainees was the need to provide food or kitchen facilities in the overnight accommodation. Other suggestions for improvement varied, including: - (HO) (HO) Exposing the trainees to more real world experience during training sessions- (HO). (HO). Note: This final comment appears to stem from reports that a small number of trainees drank excessive amounts of alcohol over one block session and as a consequence disrupted other trainees and also the training schedule 21. It is understood that there was no repeats of this conduct following the MHCC intervention. Feedback gathered from interviews and focus groups recommended more communication regarding the course content and the host organisations. This suggestion forms the basis of Recommendation 14 (iii). 21 MHCC responded to this incident by reminding all training participants that they had signed a Code of Conduct at the outset of the Program and as such were expected to abide by the code, including by adopting respectful behaviour at all times. ACIMH Program- Final Evaluation Report Page 44

56 One final comment on the training relates to the noted skills, expertise and commitment of the main trainers with specific commendations to Bradley Foxlewin. Trainees repeatedly commented on their and for his capacity to engage the trainees and provide them with a and learning environment. While many of the strengths of the successful ACIMH Program derives from its sound structure and design (as expanded upon in Section 9), the evaluators wish to acknowledge the dedication and personal contributions of the trainers as individuals. This was evident in the extent they made themselves available to trainees, including through actions such as holding after hours tutoring sessions during some of the final block sessions. As one trainee summed up: rt through the Program by Bradley. He was just 5.3 Perceptions of Overall Change At the first block training sessions (early- mid 2013 depending on the Group), trainees were asked to rate the importance of different aspects of the program. The majority of respondents rated all of the following issues and expectations as Helping you get ongoing paid employment Helping you get ongoing work in the mental health sector Helping you gaining access to more education and training Giving you access to mentors Improving your job opportunities in the future Improving your confidence in working in mental health Improving your skills in working in mental health Your personal development Your professional development Helping you meet other Aboriginal trainees Helping you link with staff who could be future employers. After the final block release training sessions (held between March and August 2014) trainees were again asked to rate how these aspects of the program had changed. ACIMH Program- Final Evaluation Report Page 45

57 In terms of access 100% 22 of trainee respondents (n=25) reported that as a result of the Program and its associated training they had better access to: education and training paid employment professional and career development. When questioned about improvement regarding growth, responses were very positive, with 100% of respondents agreeing or strongly agreeing that the Program meant they had acquired: more skills and capacity to work in mental health more future job opportunities in general (which was rated most positively with 85% of In general, the expectations of positive benefits and impacts that trainees had at the outset of the Program were seen to have been fulfilled by the last training block. These positive results are further expanded upon in the other feedback following. * * * * 22 Based on five point Likert scale ranging from strongly agree to strongly disagree. ACIMH Program- Final Evaluation Report Page 46

58 6 FINDINGS: TRAINEE OUTCOMES & PERCEPTIONS From the outset the independent Program evaluation was designed to gather data on what stakeholders identified as the main strengths and key impacts of the ACIMH Program positive and negative. drawing on the surveys, interviews and focus groups described in Section 2.2. Unless otherwise stated, all statistics are derived from the final trainee survey administered in August Key Trainee Impacts and Change In general, trainee feedback throughout all stages of the Program demonstrated a consistently positive response to the Program. For example, when trainees were asked to rate the overall benefits of the Program all trainee respondents (n= 23) reported: The Program has been beneficial to them professionally The Program has been beneficial to them personally Trainees were also asked to rate the degree to which the Program had changed different aspects of their professional or personal lives. As evident in Diagram 7, survey results showed very positive results on all key change measures. For example, in the final survey 93% of trainees repor their knowledge of mental health their skills in working in mental health sector their confidence and capacity in working with mental health consumers their chances of having a good career. Trainee feedback was also positive on the following measures, with 73% or above their interest in a career in mental health their chances of getting a reasonably paid job ACIMH Program- Final Evaluation Report Page 47

59 their confidence and capacity in learning and studying their confidence in general. The only change measure with slightly less positive results was on their interest in working in the community mental health sector, with all but one respondent indicating a posi Diagram 7: Trainee Feedback on Change as a Result of the ACIMH Program Source: EJD Consulting, Final Trainee Survey (n=23), August 2014 MH stands While these results demonstrate the ACIMH fully met the change measures that were defined for trainees at the outset of the Program, trainees themselves identified numerous other benefits that arose from participating. Diagram 8 below contains a distillation of the most common trainee-identified strengths and benefits drawn from surveys, interviews and focus groups over the duration of the Program. Section 7. ACIMH Program- Final Evaluation Report Page 48

60 Diagram 8: ACIMH Program Strengths for Trainees Source: EJD Consulting, based on consolidated stakeholder feedback. Original diagram November 2013; updated September ACIMH Program- Final Evaluation Report Page 49

61 Related benefits identified by trainees included: access to education and training, particularly within mental health access to career development and improved job opportunities, particularly within mental health opportunity to apply skills and knowledge gained in training at the workplace one trainee referred to these experiences as opportunities to meet, network, learn alongside, and share ideas and experiences with other Aboriginal trainees personal and professional support from various actors within the Program, including MHCC staff, other trainees, work colleagues, supervisors, as well as Aboriginal mentors structured opportunities to acquire expertise and formal qualifications within mental health financial incentives and career related work concurrent with undertaking education and training broader understanding of issues surrounding mental health. Other trainee issues identified at the conclusion of the Block 4 training, and in final focus groups and interviews, included: access to various Program components; support provided by host organisations, supervisors and Aboriginal mentors; aspects of their own growth, the support, and the cultural appropriateness of different aspects of the Program. Each of these is expanded upon in the findings below. 6.2 Support Trainees were asked to comment on and rate the level of support they received from various Program participants, including: their work-based supervisors their host organisation their Aboriginal mentor MHCC block release trainers MHCC program staff State Training Services Aboriginal Employment Unit (STS-AEU) Aboriginal colleagues in the workplace other trainees. The results of each are presented in Diagram 9A and 9B following: ACIMH Program- Final Evaluation Report Page 50

62 Diagram 9A: Trainee Feedback on Quality of Support Received by Stakeholder Group ACIMH Program- Final Evaluation Report Page 51

63 Diagram 9B: Trainee Feedback on Quality of Support Received by Stakeholder Group Source: EJD Consulting, Final Trainee Survey (n=23), August ACIMH Program- Final Evaluation Report Page 52

64 Trainee respondents were asked to rate the quality of support they received on a five-point rating system, ranging from very good (blue) to at all good (yellow), as indicated in Diagram 9 colour keys. Of all participants, trainees reported the block release trainers provided the highest quality of support, with 100% rating it very good (84%) or good (16%). (See Section 5 for further analysis of the training provided). In descending order the next highest support ratings 23 were as follows: MHCC program staff (95% positive) Other trainees (84% positive) STS-AEU and host organisations (each 79% positive) Aboriginal colleagues in the workplace (74% positive). While supervisors and Aboriginal mentors received lower ratings for their level of support (68% and 63% positive respectively), the majority of trainees nevertheless appreciated the support provided, with only two trainees rating the support from their supervisors negatively. Feedback on how support could have been improved upon is expanded below. 6.3 Workplace Issues Perceptions of Host Organisations Trainee feedback on host organisations was generally very positive, particularly in terms of the support they were being given in the workplace. Some examples of final trainee comments included: These types of affirming comments were consistent with remarks provided in earlier stages of the Program, suggesting the positive trainee experiences were sustained throughout. 23 Combined ratings of very good and good. ACIMH Program- Final Evaluation Report Page 53

65 While overall the feedback was very positive, the evaluators did identify individuals that had had difficulties with their supervisors or their host organisations. Challenges identified included: changes in managers and issues regarding new work expectations so- between individual trainees and their supervisors managing competing demands between work and completing study challenges associated with communication and when dealing with work or time management issues (as documented in Case Study 1, for example). While a minority, at least five trainees also com cultural issue and the need for more cultural awareness training for managers and other staff, especially prior to taking on Aboriginal trainees. (This issue is further discussed in Section 8.) When trainees were asked in the focus groups about their attitudes to their host organisations, once again the majority were very positive, although a number did recall earlier as they had at the mid-project review. In most instances these were again reported as having been resolved before the end of the Program, often with the assistance of MHCC Program staff or, less frequently, their Aboriginal mentor. At least three supervisors also referenced groups, recalling the early efforts needed to make the traineeship successful within the organisation. Nonetheless the view of most stakeholders was that early adjustments are to be expected and that in most instances, by all parties trainees, supervisors and managers and other work colleagues ensured they were satisfactorily resolved. As one host organisation respondent summed up: c provide the right environment for staff to grow in, and that staff (HO) ACIMH Program- Final Evaluation Report Page 54

66 6.3.2 Perceptions of Supervisors As illustrated in the data at Diagram 9A, the majority of trainee feedback on their supervisors was positive throughout the Program, particularly in terms of their knowledge and support provided. When asked how helpful their supervisors were 59% reported them as good or very good, with 29% neutral and only 12% suggesting not at all good. These results indicate some variations in trainee experience as borne out in qualitative feedback below. the following comments: E and was always there to help with assessments A couple of trainees commented that they were currently not receiving the level of supervision they needed, usually due to a lack of consistency with supervisors which was described by one disruptive and difficult to deal with. Upon reviewing all feedback on trainee supervision, and analysing all commentary on what defined good supervisor practice within the context of the ACIMH Program, the evaluators have identified a list of common qualities and attributes as defined by trainees. These are listed below and also summarised in Diagram 10. (Note: The descriptors as expressed by trainees are shown in italics). and a preparedness to share ideas and information freely acknowledging learning in the workplace and also personal growth Proactive in communication This relates to not waiting for formal supervision meetings to discuss any work issues or problems. Instead, initiating providing feedback on how the issue is progressing and then ACIMH Program- Final Evaluation Report Page 55

67 particularly with assignments and assessment tasks Inclusive, and as full members of the work team. This includes encouraging trainees to participate in staff meetings and other work-based planning activities. It also includes giving trainees an appropriate share of responsibilities and opportunities to learn new skills such as sharing the role of minute taking, participating in work-based health and safety training, and helping as appropriate with other shared administrative responsibilities and being open to learn from the trainee as one does from any other colleague in the workplace of lived experience and heritage. This also requires avoiding -, or language as well as derogatory tones Acknowledging contributions including in terms of Aboriginal client engagement and also feedback on local Aboriginal community issues. ACIMH Program- Final Evaluation Report Page 56

68 Diagram 10: Source: EJD Consulting, drawn from consolidated trainee feedback. September ACIMH Program- Final Evaluation Report Page 57

69 Many of these qualities are summed up in this single trainee comment offered in response to the question- knowledgeable etc. Always encouraged me, told me if I was doing The same content analysis of trainee responses also led the evaluators to identify supervisor qualities and attributes that were deemed to be the least helpful or detrimental to trainee success. They included: Condescending or use of - Impatience (especially in the early months when trainees were getting used to the work-study experience) (though this was not expanded upon in the context of supervision see discussion below) As both the positive and negative supervisor qualities and attributes were observed to be common to the majority of trainees, the evaluators recommend that they be incorporated into future host organisation induction materials. Further, they recommend that every effort be made to select trainee supervisors who consistently exhibit these qualities in the workplace. R2. Incorporate additional training, practical advice and feedback on recommended supervisor qualities and attributes as part of the Program establishment process, including use of the Diagram 10 summary. ees was the fact that they experienced changes in their supervisors across the course of the Program. For one trainee, this meant three different supervisors in 12 months; for at least four others, two different supervisors. There were also two reported instances where the trainee requested a change in supervisor, or it was done by mutual agreement. While the negotiated changes were viewed positively by the trainees in one instance described as ACIMH Program- Final Evaluation Report Page 58

70 in most instances the change was viewed negatively, with trainee descriptors including and While personnel changes are acknowledged as an inevitable part of modern workplaces, supervisors should be selected, wherever possible, on the expectation that they will remain in the same position for the duration of the traineeship. R3. Maximise options for providing trainees with stable and supportive supervision over the duration of the Program, with any changes in personnel sensitively managed with the trainee Advice for Future Host Organisations Trainees were asked in surveys, interviews and focus groups for advice that might assist future employers of Aboriginal trainees. Apart from reconfirming the above described recommended supervisor attributes, and the provision of high levels of support and encouragement to the trainees throughout, two other clear themes emerged: 1) The importance of host organisations being well prepared for hosting an Aboriginal trainee, including by providing all staff and managers with cultural training This advice is further developed and commented on in Section 8. 2) Wherever possible, host organisations should host more than one trainee per location to provide opportunities for ongoing peer support. Many trainees who were in service outlets with only one trainee reported challenges stemming from perceptions of not being or adequately These types of comments were most evident in trainees hosted in non-aboriginal organisations, although isolation was also reported by at least one trainee in an all- Aboriginal service. It should be noted however that one trainee hosted in an all non-aboriginal service, and with no other trainees present, reported no problems As she observed, all the (non-aboriginal) staff and managers were: d; the While hosting a second and third trainee does not necessarily address the above challenges, overall feedback nonetheless indicates it is a preferred option, with many examples of successful peer support and work-based friendships forming as a result. Further, a number of supervisors responsible for two trainees reported benefits to the organisation and to the work place. While ACIMH Program- Final Evaluation Report Page 59

71 was a noted risk, feedback indicted that having two trainees helped other staff to include them in activities. Other reported benefits included: assisting in rosters and rotations thereby allowing other staff to witness and learn from more than one Aboriginal cultural background or perspective for As a result the evaluators recommend that a minimum of two trainees per service outlet become the norm in future traineeship programs, with options for single placements if supplementary supports can be established from the outset, especially in non-aboriginal organisations. R4. In terms of hosting a trainee: a) Encourage organisations to host a minimum of two trainees per service location to maximise peer support opportunities. b) Where a second trainee per location is not practicable, in consultation with the trainee, investigate providing supplementary support to overcome any perceived or actual isolation, especially if hosted by a non-aboriginal service. 6.4 Relationships to Mentors As indicated in Diagram 9 above, while the majority of trainees rated the contributions of both the STS-AEU and their Aboriginal mentors as good, the overall assessment of support provided through each was less positive when compared to other Program participants. Over the course of the Program the evaluators found different experiences in terms of trainee access to mentors. On the one hand some trainees reported they had good and consistent access to their mentors. Comments regarding this mentee experience included: always able to offer advice regular/consistent contact, arty, I could ACIMH Program- Final Evaluation Report Page 60

72 On the other hand some trainee experiences and access to mentors were far less positive, with a significant proportion describing a lack of consistency with visits and a loss of support. Trainee comments of this mentee experience included: -minute conversations and did not see them a Similarly, feedback from supervisors and host organisation staff also indicated differences in their experience of Aboriginal mentors in the Program. Some supervisors reported a positive experience throughout, but a significant proportion used negative descriptors such as ; ; n ; and When other stakeholders, including Program management staff, were asked for their overall assessment of Aboriginal mentors descriptors included: While all participants consulted agreed access to Aboriginal mentors was an essential component of the required Program support options, many felt that they needed to be combined with the inconsistency in the overall performance of the AEU mentors within the ACIMH Program has led to the evaluators to recommend the following: R5. Ensure Aboriginal mentors are directly linked and accountable to Program managers, even if funded or delivered through other agencies. When respondents were asked for other comments with regard to what was ake the Aboriginal mentor component of the Program effective in the future, various common roles and qualities emerged. These are summarised in Diagram 11 and comprise: ACIMH Program- Final Evaluation Report Page 61

73 Diagram 11: Recommended Aboriginal Mentor Roles and Qualities Source: EJD Consulting, drawn from consolidated respondent feedback. September ACIMH Program- Final Evaluation Report Page 62

74 Being to the trainee as components of the relationship. Providing consistent access throughout the Program. This includes a regular contact schedule, as well as a consistent individual in the role of mentor as much as practicable. Providing advanced notice of host organisation visits, allowing sufficient time to ensure both the trainee and the supervisor are free to meet. Further, in cases where there are issues in the workplace, providing an opportunity for the trainee and/or supervisor to communicate what might be useful topics for joint discussion visit. Being This means able to communicate with the trainee and provide support and advice on any issues they think relevant. In some instances this may be work-related, or related to confidence issues; in other cases it might relate to family and community issues that are impacting on the. As one trainee summed up: Related to the above quality, able to perform the role of -between between the trainee and the supervisor. As two different supervisors summed up: constantly felt very frustrated that the mentor never spoke to me h communicating well on our own. Having the mentor assist in those discussions would have been a god- trainees, I do think they should have a role in helping resolve problems being experienced in the workplace. I for one would have really valued having regular three-way conversations [traineesupervisor-mentor] to help each of us discuss issues and ensure it was all happening in as sa R6. Increase the consistency and quality of Aboriginal mentoring including through promoting the recommended mentor roles and qualities documented in Diagram 11. ACIMH Program- Final Evaluation Report Page 63

75 One final finding relating to Aboriginal mentors within the ACIMH Program was that some trainees reported they did not feel the need. While all trainees consulted reported that they highly valued contact with another Aboriginal person to talk to about the traineeship, particularly one that was not a or, many indicated they to utilise other colleagues for this purpose. This perspective was borne out in the survey results illustrated in Diagram 9. Further, a number of trainees reported that they preferred to use the support and advice of Aboriginal colleagues, their or just contact with other trainees to. This perspective was expressed by a number of trainees based in Aboriginal services, though at least two trainees in non-aboriginal services also provided the same feedback. Overall the findings indicate that while having access to formal external and nonsupervisory Aboriginal mentors was a very important Program, it should be left to the individual trainee and their host organisations to define how to best action this role. R7. Maintain external Aboriginal mentoring as a core component of the Program, however provide enhanced flexibility in how this role is fulfilled including through options such as other Aboriginal staff or local personnel identified by, and jointly agreed to, by individual trainees and their supervisors. 6.5 Broader Impacts of Program on Trainees At the conclusion of the Program trainees were asked to reflect on broader impacts of the Program, including in respect to their family and community, their personal and professional development and their future plans. The combined results of the survey (n=23), focus group and interview feedback are summarised below Personal and Professional Development When trainees were asked what impacts the Program had had on their personal and professional development the responses were extremely positive. For example, 100% of trainee respondents reported the Program had been: - ACIMH Program- Final Evaluation Report Page 64

76 When trainees were asked to describe the most significant impact of the Program professionally or personally, the feedback was varied with many observing growth and development in both domains. Responses in relation to career and knowledge-related issues included ; ; ; and ; ; and nfidence and talking Many trainees also referenced the value of the friendships formed through the Program, and the assurance that came from These benefits were also strongly reinforced in the focus groups, with trainee participants consistently referring to their personal and professional growth, rather than one or the other. The evaluators concluded that the combined strength of these outcomes, albeit somewhat unforseen, further demonstrates the success of the ACIMH the quality of training and the use of groups, which has allowed individual friendships and confidence to grow. This sense of personal and professional growth can be summed up in the following trainee statements: R8. Maintain the small group block training model to foster friendship networks and promote shared personal growth concurrent with professional and career training Impacts on Family and Community When trainees were asked about the impacts the Program had had on their families or communities, the responses varied though for the most part were overwhelmingly positive. While a number of trainees noted some challenges in respect to juggling being away from home block training sessions, the vast majority of other responses were either neutral or positive. For example in focus groups a number of trainees observed that the short absences had made or resp ACIMH Program- Final Evaluation Report Page 65

77 Focus group feedback also indicated that some trainees saw themselves as and that rewards thought it great to see their Mum Many trainees consulted observed that the Program had been highly beneficial to their immediate family by providing a stable and income. Trainee comments included: Much and the knowledge I can support them. It has given me insight into rt me in doing this. Since being employed I have bought a house and shown my children it is In respect to the community, most trainees consulted also noted the impacts had been positive with a number observing the Program improved and Aboriginal community members experiencing difficulties, including those with drug and alcohol as well as mental health issues. Examples of trainee comments related to community impacts include: Aboriginal workers in the sector for the Aboriginal people to reach out to. It has helped my family by giving me the skill to support them As previously noted (see Diagram 8 and related discussion), numerous trainees also saw the major benefit to the community of having more professionally trained Aboriginal mental health workers: Finally, at least four trainees also noted that they were using lessons from their training to engage with family, friends and other members of the community. For example, one trainee reported that their family life had improved because she was better at: anaging and controlling my emotions and reflecting This combined feedback suggests overall impacts on family and community were seen positively by trainees with no significant unforeseen consequences identified. Further, as noted by numerous trainees, the benefits identified are likely to continue to accrue well beyond the duration of the ACIMH Program. ACIMH Program- Final Evaluation Report Page 66

78 6.5.3 Future Plans % of respondents indicated they planned to remain in their current organisation, either: A smaller proportion of respondents reported they were planning to: When trainees were asked about their plans over the following three years, the Adding to earlier feedback, when queried about the types of further study that trainees planned to undertake, responses included: community services and. 24 For this reason, it is not possible to produce single percentage results i.e. percentage exceeds 100%. ACIMH Program- Final Evaluation Report Page 67

79 In terms of work plans in the following three years: 40% of trainees reported they planned to remain with their current organisation, the majority in their current role and a minority in another role in the same organisation 33% reported plans to work in another field. Fi time, what would you like to be 66% reported they still planned to be working in the mental health or health promotions with some responses being very specific in terms of roles, including: o o o Focus group feedback was very similar with the majority of trainees indicating they saw themselves as having an ongoing role in mental health. A significant proportion of respondents also noted they planned to become a positive force in their community, with at least five trainees using the term Once again this feedback indicates the ACIMH Program has not only delivered immediate benefits in terms of professionally trained Aboriginal mental health workers, but has also forged an ongoing majority interest and commitment to working in the sector for a three to five year period. 6.6 Perceptions of Trainee Changes At the outset of the Program a number of key change measures were identified for trainees who participated in the ACIMH Program (see Appendix 2 with evaluation criteria). At the end of the Program all stakeholders were asked for their assessment of the perceptions of the changes. Diagram 12 below contains the feedback for all other respondents (namely host organisation staff, mentors and other stakeholders). ACIMH Program- Final Evaluation Report Page 68

80 Diagram 12: St Source: EJD Consulting, ACIMH stakeholder survey (n=30), August * As with the trainee feedback, assessments of change were very positive overall. For example, 90% or above of all non- Knowledge of mental health (80% a lot changed) Skills in working in the mental health sector (73% a lot) Confidence and capacity in working with mental health consumers (77% a lot) Confidence and capacity in learning and studying (73% a lot) Interest in working in the community mental health sector (73% a lot). on three key development indicators, with over 90% rating the following a lot or a Confidence in general (73% a lot changed) ACIMH Program- Final Evaluation Report Page 69

81 Professional development (73% a lot) Personal development (77% a lot). Finally, trainees and host organisation staff were also asked to rate outcomes on two final change indicators: In terms of chances of having a reasonably well paid job : o o 100% of trainee staff reported a positive change (80% a lot changed) 79% of host organisation staff reported a positive change (63% a lot changed) In terms of chances of having a good career : o o 100% of trainee staff reported a positive change (80% a lot changed) 84% of host organisation staff reported a positive change (63% a lot changed) When asked to provide suggestions that might help the Program better assist trainee development, trainee responses included: ve consistent support available, understand the commitment that One trainee also commented on the need for more information regarding future job opportunities both within and outside of the host organisation, as well as additional explanations of pathways to further study. Other respondent group suggestions related to trainee development included: - employment and it is important for workers to be aware of and truggled with her commitment to work tasks, had to be ACIMH Program- Final Evaluation Report Page 70

82 One host organisation respondent summed up the overall positive assessment of trainee development as follows: strength to strength, their families and work lives have excelled (HO). Notwithstanding some suggestions for improvement identified above and in other parts of this report, the evaluators concluded the Program made a significant impact on trainee development. The vast majority of respondents rated the Program positively in terms of professional and personal development measures, as well as in each of the key skills and understandings measures. These measures, combined with the strong retention data described earlier, evaluation parameters established at the outset. * * * * ACIMH Program- Final Evaluation Report Page 71

83 Operations Establishment Host Organisation: Case Study 2 and we really The host service was a well-established mental health NGO with strong connections to its non-metropolitan coastal region. The service has 15 staff and runs four programs including PHaMS and HASI. The service has had Aboriginal staff in the past, although there were none employed when the trainee was recruited. -Aboriginal. For example approximately 2 of 18 HASI clients and 5 of 60 PHaMS clients identified as Aboriginal. The trainee was recruited by an advertisement placed online in SEEK. An interview was held with two staff members before the successful trainee was selected. The trainee was allocated to work two days on PHaMS and three days on HASI. This was to provide a For the first four months the trainee worked with other staff and was provided with a lot of supervision. The trainee was reported to have greatly benefited from this approach. She and was She also was confident asking questions. As the trainee had worked in customer service prior to the Program, she had good communication skills. She also got on very well with staff. The management of the block training and study release time was reported as being. process by posing work-related skills questions to the trainee. For example: trainee would then research and respond via . The The supervisor reported that this was his way of knowing she was and able to convert her training into practical learning. ot only clever, but also recognised that she was ACIMH Program- Final Evaluation Report Page 72

84 Impacts & Advice and other service staff provided the trainee with lots of support and she was Having an Aboriginal member of the team meant that more talk about culture and about working with [Aboriginal] people and - Through hosting an Aboriginal trainee the supervisor reported that the service became better known in the Aboriginal community and. Further, he reported that rates of Aboriginal client referrals had increased. - In reference to hosting the Aboriginal trainee, it was observed: - Other Program impacts observed included staff asking the trainee for advice. This was commented on as the start of the Program. It was noted that the trainee was also quick to to her colleagues particularly as her confidence grew. It was noted that staff valued her insights and were very receptive to her on their practice and ways of interacting with clients. Lessons learnt from supervising the trainee that could be useful to future host organisations: o Ensure the supervisor has the offering support and advice and is good at o Allocate the role of supervisor to someone with time - different periods, a lot of time o Recruit trainees that demonstrate capacity to get on well with other team members and a o Be sure to define work expectations such as arriving on time to work o Treat trainees similarly to other staff members so as not to undermine the support provided and potentially seed resentment. capacity to make a positive and ongoing contribution to the workplace. The trainee was offered and accepted a full time HASI position with their host organisation. The trainee will be working with all clients not just Aboriginal clients. This was noted as important The trainee agreed. ACIMH Program- Final Evaluation Report Page 73

85 7 FINDINGS: HOST ORGANISATION OUTCOMES & PERCEPTIONS This described in Section 2.2. Unless otherwise stated all statistics derive from the final survey administered in August 2014 (n=62; Host organisation respondents= 28). 7.1 Program Implementation As discussed in the EJD Consulting Preliminary ACIMH Program Evaluation Report (November 2013) host organisation respondents indicated general satisfaction with the process used to promote the Program to potential host organisations and then select and induct them into the Program proper. When Program participants (other than trainees) were asked how well host ementation, only one host organisation (3% 86% reported 10% response in terms of their preparedness. One supervisor, however, did comment that that they felt they were not provided with sufficient information to implement the Program, particularly in respect to supporting the trainee: expectations. It has been quite difficult for some trainees to adapt; working in mental health can evoke distress and trauma which was When asked if there was anything that would have better prepared host organisations to host the Program and support trainees in the workplace, a number of constructive suggestions were provided. The most common related to receiving additional cultural awareness training earlier in the process. R9. Provide additional cultural awareness training to host organisation staff, as well as tailored advice on how to best support Aboriginal staff in the workplace, prior to any future traineeship program commencing. Also see the discussion in Section 8. ACIMH Program- Final Evaluation Report Page 74

86 Other suggestions for how implementation could have been improved included: and the MHCC. This will provide the organisation and supervisors with - (HO) the assessments and guidelines. Arrangements for reimbursement for costs and overnights if not paid that week it While each of these suggestions are worthy of further consideration and action where possible, the evaluators also observed that many participants were aware that they were (HO) and (HO). Further, a number of host organisation staff noted that many of the issues encountered (both positive and negative) related to how individual trainees and staff interacted; would be have been a hard thing to plan for one respondent commented in respect to host organisations: Notwithstanding options for improvement, 76% of all non-trainee respondents at the conclusion of the Program rated the Program positively when asked for feedback on the level of support provided to host organisations in terms of implementing the Program. R10. Consider offering additional host organisation training and support options, at the outset as well as at intervals during the Program, for those staff wishing supplementary advice and feedback in respect to their trainee supervision roles. ACIMH Program- Final Evaluation Report Page 75

87 7.2 Key Host Organisation Impacts While the ACIMH Program was first and foremost a workforce development initiative, from the outset it was also designed as a Program to improve the inclusive practices and operations of participating community mental health providers in respect to Aboriginal staff, Aboriginal clients and also Aboriginal communities. While Section 8 contains some key cultural impact results, many of the identified benefits and challenges discussed below also link to these outcomes Key Benefits When host organisations were asked to identify what benefits they had identified from being involved in the Program, the feedback was rich and diverse. As summarised in Diagram 13 the most common benefits raised included (in approximate order of perceived importance): Contributing to building a skilled, culturally appropriate workforce suited to working in community mental health Improving organisational capacity in terms of working with Aboriginal clients and outreaching to Aboriginal communities Improving staff awareness of Aboriginal issues and having practical options for how to be inclusive of Aboriginal colleagues and clients Being part of an innovative program and having access to new colleagues that can offer a different perspective as well as solutions to known challenges Having access to new resources and practical advice to sustain Aboriginal staff recruitment and retention outside of the Program Having an opportunity to and reflect on work processes and confirm what is good practice and why Personal rewards from getting to know the trainees and benefiting from their heritage, insights and perspectives. One largely unforeseen and yet pervasive positive outcome of the Program was the number of host organisation staff who personally valued and benefited from contributing to the professional and personal development of the trainee over the course of the Program. For example, when host organisation respondents were at least half commented on this issue above others, with comments such as: ACIMH Program- Final Evaluation Report Page 76

88 r practices and A second and related unforeseen benefit identified was the extent to which many of the trainees impacted on their colleagues and work practices, albeit over a relatively short period of time. In interviews and focus groups, as well as in open-ended survey responses, host organisation staff consistently volunteered how much they the or provided by trainees. As one supervisor commented: staff before but there was something about having a young person [trainee] study and in our service over this time that made us all sit up and say - Many respondents also commented on their observations of how much other staff had also learnt from the experience. As one host organisation coordinator commented: trainees] and giving advice and support, to themselves asking questions and getting advice on this or that client or on some other aspect of our service It may be that we were lucky and ended up with this sort of two- While these types of comments were most evident in non-aboriginal services, some Aboriginal staff also commented on this, noting benefits associated with, and also the new learning and updated perspectives provided through the Certificate training. (Note: This perspective is more fully expressed in the Host Organisation Case Study 1). ACIMH Program- Final Evaluation Report Page 77

89 Diagram 13: ACIMH Program Benefits for Host Organisations Source: EJD Consulting, based on consolidated stakeholder feedback. Original diagram November 2013; updated September 2014 ACIMH Program- Final Evaluation Report Page 78

90 7.2.2 Key Challenges Along with benefits, host organisations were asked to identify any challenges experienced over the course of their involvement in the ACIMH Program. While some respondents were clearly reluctant to Program experience some issues, albeit mostly in the first third of the Program. As summarised in Diagram 14, the most common challenges noted (roughly in order of weighting by respondents) were as follows: As previously discussed in Section 6.4, insufficient access and lack of consistency in Aboriginal mentoring support. As one supervisor summed up: organisation managers having to dea When this challenge was unpacked staff reported that some trainees appeared to be unfamiliar with working in office environments, or working as part of a team. Examples of how this manifested in trainee behaviour in the workplace included: o poor time management and being late for meetings o o o managing personal calls, especially on work mobiles frequent distraction by personal matters in the workplace. and work consequences Staff reported that, particularly in the early parts of the Program, some trainees did not appear to understand it was their responsibility to let their supervisor know they would not be at work. in one case described as in terms of the degree to which the work placement would parallel the training modules. Specific topics where work experience was seen to fall short of expectations for some trainees were opportunities to utilise training in: o o o o suicide prevention outreach drug and alcohol counselling plus case management. ACIMH Program- Final Evaluation Report Page 79

91 As two host organisation respondents commented: trainees though we were being- What? - the placement, leading to a lack of confidence and a reluctance to Keeping trainees on task and motivated to complete the Program. In the focus groups, this and was cited by most, though not all, supervisors as well as trainees. Most commented that they saw this as routine in any new activity undertaken, though for some it required on behalf of supervisors as well as other work colleagues. In other instances, Service Coordinators, ACIMH Program managers, and also block trainers were involved in motivating and encouraging commitment to the Program. As one supervisor summed up: trainee] in the Program and able to complete tasks, even if they were Working around block training, study commitments and. This issue appeared to be more challenging in smaller organisations that had rostering affecting other buddy or supervising staff. In some instances and time periods the study demands t, with an -hands-on- cited in respect to one trainee to ensure In another three organisations scheduling was also cited as a challenge in the establishment months when the trainee needed to be at all times and could not provide any direct client support One final common challenge noted by a number of host organisation staff, as well as trainees, related to relationship breakdowns (on two occasions termed ) between a trainee and another staff member. In at least two instances the challenge was experienced between the trainee and their supervisor. In some instances feedback indicated that the trainees were either reassigned, or requested to be moved to another service. This was generally reported to improve the situation. ACIMH Program- Final Evaluation Report Page 80

92 While these differences appeared reasonably common (reported by parties in at least six different services), the evaluators did not deem it a function of the traineeship program as such and therefore have not included it in the diagram or summary points. Diagram 14: Common Challenges Experienced by some Host Organisations Source: EJD Consulting, based on consolidated host organisation feedback, September While each of the above challenges was cited by numerous respondents, it should be noted that by no means were they experienced in all placements. Further, those respondents who did experience challenges usually only cited one or two issues, not all of those listed. In addition, many host organisation respondents commented that while the challenges were problematic at different points in the Program, in many instances they were time-limited, or were worked through and improved or resolved by the end of the formal stage of the Program. Nonetheless the fact that they were evident in a number of contexts suggests the need for amelioration measures as recommended below. ACIMH Program- Final Evaluation Report Page 81

93 Analysis of respondent feedback across the duration of the Program further led the evaluators to identify some common trainee attitudes or practices that underpin successful traineeships. As summarised in Diagram 15, these related to eight core areas: Being open to learning new things and also to receiving advice. Avoid thinking that advice is criticism or suggests failure to perform. It is other senior staff s role to ensure trainees learn on the job and receive advice and support whenever and wherever appropriate. Understanding what it means to work as part of a team and the impacts set aside time for the trainee, or there is a meeting scheduled,. This attribute also means acknowledging where people have made an effort to assist and showing appreciation if appropriate. Being mindful of even vigilant about occupational risks and safety issues, particularly in dealings with clients. As one supervisor commented: There are All workers, including trainees, have a responsibility to manage safety and risks in work settings and on home visits. Trainees need to take this matter very seriously. Asking questions and seeking advice early, rather than waiting for problems to emerge or issues to escalate. Under normal circumstances the another colleague, an Aboriginal mentor or Program staff can be used. One supervisor offered the following advice: option for all concerned is to raise it and then something can course of action. Early, full, and open [discussion] is what Being professional and also punctual at all times. This includes wearing work-appropriate clothing, and establishing boundaries around conduct and communications suited to the work place. This also means considering when and when not to take or make calls relating to non-urgent personal, family or community matters. Being realistic about workplace opportunities in terms of course content. For example, not all host organisations can provide work experience in the same way. ACIMH Program- Final Evaluation Report Page 82

94 Diagram 15: Recommended Trainee Attitudes and Practices Source: EJD Consulting, drawn from consolidated respondent feedback. September ACIMH Program- Final Evaluation Report Page 83

95 Using appropriate language and communication styles in the workplace and ensuring supervisors are directly notified of any changes related to work responsibilities, study commitments or attendance. Being helpful and respectful in advising others of cultural issues. As much as possible, minimising work impacts arising from family and community commitments. This includes notifying the workplace of unplanned leave, the duration of anticipated absences, and also providing updates as required. R11. Using the core messages in Diagram 15, strengthen induction training for trainees regarding workplace expectations, covering issues such as: a) Potential non-alignment of training to work options b) Punctuality and work place attendance c) Communication protocols and mobile phone usage d) Role of supervisors and expected notification and communication channels related to work e) Dress codes. R12. Provide additional advice to supervisors about the importance of work place inductions for trainees including dialogue on how any work issue or challenge will be addressed. 7.3 Internal Policies and Practices As described in Section 2.4, one of the intended outcomes of the Program was for host organisations to improve their internal policies and practices in terms of a) Aboriginal employment and b) Aboriginal client engagement. While the majority of respondents across all groups reported changes in all measures related to these outcomes (see Diagram 16 below) the extent of perceived change varied. For example, across all survey respondent groups, the most significant changes or improvements were recorded for host organisation Two other measures with high levels of reported change (70% or above), albeit General interest and commitment to Aboriginal employment. ACIMH Program- Final Evaluation Report Page 84

96 While positive, consolidated respondent results were less strong in terms of change for the following three measures (in descending order): Aboriginal client engagement and retention Confidence and capacity to work with Aboriginal people and communities Aboriginal recruitment and employment practices. Diagram 16: Change Source: EJD Consulting, Consolidated ACIMH stakeholder surveys (n=62), August Based on host organisation staff only, reported change was strongest in terms -third number of these commented that this was not relevant as they were already Aboriginal providers. The areas where host organisations rated themselves as seeing less change as a result of the Program (albeit the majority still positive) were: Aboriginal client engagement and retention ACIMH Program- Final Evaluation Report Page 85

97 Aboriginal recruitment and employment practices. internal policies and practices, the overall results were more dispersed and in general less positive than staff assessments. For example, the mea little changed (38%) as a result of the Program. In descending order other reported changes were: cultural competence to work with Aboriginal clients (69% combined change; 25% recording changed General interest and commitment to Aboriginal employment Staff confidence and capacity to work with Aboriginal people and co Aboriginal client engagement and retention (50% recru Mentor feedback on these same policies and practices measures also varied. The highest change assessment was given to staff confidence and capacity to work with Aboriginal people, and staff awareness of Aboriginal issues (with For all other measures mentor assessments were split, with one mentor Host organisation staff and trainees were also asked if their any plans to further modify its internal policies and practices arising from the, 7: 33% responded that they do not have any plans for modification at this stage ACIMH Program- Final Evaluation Report Page 86

98 Diagram 17: Host Organisation Plans to Modify Internal Policies and Practices Source: EJD Consulting, ACIMH host organisation survey (n=28), August When host organisations were asked to specify their plans to modify their internal policies and practices, responses included: Develop and modify current policies, including recruitment and leave provisions Take steps to increase staff cultural awareness Continue to employ Aboriginal trainees. When respondents across all groups were asked what host organisations should change in terms of their internal policies and practices, three recurring themes emerged, namely: including by staff participating in additional cultural awareness training (T) by: o ACIMH Program- Final Evaluation Report Page 87

99 o When host organisations were asked to comment on suggestions for change within host organisation internal policies and practices, responses varied, including: aff and also to be involved in 25. While these combined results indicate the Program did make a measurable encouragingly noting the desire to make their services even more inclusive of, clearly other complementary and longer-running strategies are needed to bring about sustainable change. R13. Implement additional measures and initiatives to support the community managed mental health sector in NSW to become more culturally inclusive and responsive to Aboriginal client and community needs. This might include providing additional advice, resources and training opportunities to interested staff utilising some of the learnings arising from the ACIMH Program. 7.4 Broader Impacts of Program on Host Organisations When host organisation staff were asked if their involvement in the Program had been personally or professionally beneficial, 100% responded positively: When asked if their involvement had been beneficial to their organisation, once again 100% responded positively:. Specific positive impacts commented on included: 25 There were no suggestions from stakeholder respondents. ACIMH Program- Final Evaluation Report Page 88

100 consumers, community and partn environment for trainees to learn and have qualifications in mental balancing the work and When asked if there were any negative or unforeseen impacts, host staff as we had two inexperienced staff [trainees] days. Did create a negative impact from what had been a positive traineeship experience with expectation of further employment Finally, when host organisations were asked for any other suggestions as to what would improve the Program, or what other assistance they could be provided with to optimally host Aboriginal trainees in the future, many reiterated topics already discussed elsewhere in the report. For example: Consolidating accountabilities through a single Project management structure to overcome the in roles and responsibilities between Aboriginal mentors- STS staff and MHCC (see Recommendation 5) Improved consistency and access to Aboriginal mentors (see Recommendation 6) Enhanced access to course content and assessment dates in advance of trainee blocks to help inform future work roles and also when and what additional staff support could be sourced (see Recommendation 14) Improved information and advice at the outset on workplace expectations and also trainee expectations (see Recommendation 11). In addition to these topics, a number of host organisation respondents recommended additional activities be undertaken. These included: More face-to-face meetings between MHCC and host organisation staff Improved or more specifically information of reimbursements requirements and how ACIMH Program- Final Evaluation Report Page 89

101 to provide proof of continuation of traineeship attendance and MHCC to, or directly make available cultural appropriateness training for staff that is On reviewing the consolidated feedback related to host organisations the following recommendations arose: R14. Prior to trainee recruitment, enhance Program briefing sessions and training offered to host organisation staff, covering topics such as: a) Culturally appropriate workplaces and practical advice on approaches and practices to encourage and also avoid (see Section 8) b) Good practice in Aboriginal staff recruitment, induction and retention c) Overview of the training program, including individual block content and schedules, plus key assessment dates and other milestones d) Working with Aboriginal mentors and how to ensure these and other Aboriginal supports are optimally supporting the trainee and their supervisors e) As documented in Diagram 14, preparing for potential challenges with practical advice and options for how to manage them f) Administrative and accounting responsibilities under the Program. R15. Ensure trainee supervisors, in addition to host organisation service coordinators and senior managers, have access to a tailored version of the above, with specific emphasis on effective communications and managing potential challenges. While the above recommendation is likely to enhance future traineeship programs, overall the findings related to host organisation impacts and outcomes were very positive and indicate the Program was generally well run and administered as further documented in Section 9. * * * * ACIMH Program- Final Evaluation Report Page 90

102 8 FINDINGS: CULTURAL APPROPRIATENESS OF THE PROGRAM As documented in the literature review in Section 3, a key and often stated success factor in programs targeted to or inclusive of Aboriginal participants is its cultural appropriateness. This section contains an analysis of the cultural appropriateness of the ACIMH Program from the perspective of various Program participants. 8.1 Overall Assessment All stakeholders were asked to rate the cultural appropriateness of host organisations in terms of supporting Aboriginal trainees. As shown Diagram 18, the results show an overall positive assessment with 68% of all respondents rating negative assessment was provided by trainees, with three rating their Diagram 18: Cultural Appropriateness of Host Organisations in Terms of Supporting Aboriginal Trainees Source: EJD Consulting, ACIMH stakeholder survey (n=62), August 2014 * Includes mentor responses as well. ACIMH Program- Final Evaluation Report Page 91

103 Host organisation respondents had the highest proportion of positive responses overall (88% ranking as or very good ), though other stakeholders including Aboriginal mentors also rated at 82%. While only 59% of trainees rated their host organisations positively on this measure, nearly one-quarter (23%) provided a, implying the organisation was neither good nor poor. 8.2 Trainee Perceptions When trainees were further asked to rate how culturally appropriate other aspects of the Program were, all but two trainees rated the training and the staff at MHCC as the most culturally appropriate 26. Aboriginal Mentors also rated well, with all but three trainees rating them as the least culturally appropriate aspect of the Program, albeit the majority still responding positively as documented above. a number of trainees recommended cultural awareness training for all host organisation staff (see Recommendation 17). Other feedback varied and included: Various trainees suggested having more Aboriginal staff within the organisation. For example: Having - I try my best to engage with both men and women but culturally speaking, I will never be able to engage Having more culturally appropriate spaces and indicators in the workplace: and -friendly Encouraging staff to be more involved in the community for example: One trainee also suggested the need for their host organisation to adopt more flexible approaches to financial support: more understanding about money and giving it to supposed. 26 Rating that they are either very appropriate or appropriate. ACIMH Program- Final Evaluation Report Page 92

104 A final trainee recommendation was for a more for example: attitude to leave, always attending funerals. Having compassionate leave, no matter how close or distant the relative, The evaluators envisage that these issues and perspective could be best addressed in the training and staff and trainee inductions already discussed at Recommendation 14 and 11 respectively. 8.3 Host Organisation Perceptions When host organisations were asked what helped them improve their cultural appropriateness, many examples were given. For Aboriginal-run services, most referred to ensuring the trainee felt fully part of the team and facilitating - with other Aboriginal colleagues. For non-aboriginal services, the most helpful actions cited included: Providing in-house cultural awareness training to staff Ensuring trainees had ongoing access to other Aboriginal staff and/or other trainees Utilising or introducing a Reconciliation Action Plan (RAP) committee or establishing other processes such as a ways of promoting better cultural practic Supporting trainees to attend community events Encouraging other staff to participate in local Aboriginal events such as NAIDOC week. One respondent described this positive experience as follows: tural day with members and food tasting enjoyed by all. An art work (HO) Other options and strategies cited included: Establishing or uti Aboriginal staff case this included employing storytelling and dance as a program [for clients] (HO). ACIMH Program- Final Evaluation Report Page 93

105 Another commonly reported strategy that many host organisations acknowledged to be helpful and was also highly valued by other stakeholders including trainees (as documented in Diagram 10 regarding recommended supervisor roles). These types of options and suggestions highlight the need for there to be additional opportunities for community mental health organisations to share their successful experiences and work in terms of community engagement. R16. As part of broader sector capacity building efforts, provide additional opportunities for community managed mental health organisations to exchange experiences and good practice in respect to inclusive practice and effective Aboriginal community engagement activities and strategies. ons 8.4 Future Options As noted above, while host organisations rated themselves more highly than other stakeholders in terms of their cultural appropriateness, many supervisors also offered suggestions for how they thought their organisations could improve in the future. Many of these suggestions were echoed in feedback received from all other respondents including trainees. The topic that received the most comments related to improving the quality and frequency of communications between the trainee and their supervisor. There was also an acknowledgement that more would greatly assist. For example: (HO) -to-knowwere often too work focused I think to make [the trainee] feel fully at - - Other suggestions included more formal mechanisms such as: Recruiting more Aboriginal staff including into management positions Developing a reconciliation statement Improving community outreach and engagement with local Aboriginal services and groups Establishing new service resources and promotional materials specifically targeted to Aboriginal users As noted by trainees, erecting more flags and other Aboriginal-specific signage. ACIMH Program- Final Evaluation Report Page 94

106 Analysis of all feedback in respect to cultural appropriateness led the evaluators to conclude that while no single or set of strategies and actions will necessarily deliver a culturally appropriate organisation, there are nonetheless a number of measures that could be undertaken to improve the foundations for hosting future Aboriginal traineeships. They include: R17. Encourage all host organisations to undertake quality cultural awareness training for all staff and management prior to hosting a trainee. R18. Upgrade training and advice provided to supervisors at the outset, including presentations by Aboriginal staff or past trainees about what is and is not helpful in the workplace. R19. Provide opportunities for trainees to discuss workplace issues and challenges regarding culture as part of block training and distribute the host organisation coordinators and supervisors. Each of these concrete suggestions however, needs to be integrated with a commitment to continuous improvement in respect to cultural inclusion. As confirmed in the literature (see Section 3) ongoing communication, and reflection on internal cultural and work practices, along with building strong relationships with Aboriginal staff and local Aboriginal community members and organisations is critical to successful and inclusive programs and services. As one Aboriginal respondent summed up: would assist an organisation [in terms of cultural appropriateness]. If anything, [improved cultural appropriateness is based on] good This perspective is expanded upon in the suggested pillars discussion below. 8.5 Suggested Program Pillars ural appropriateness was found to be positive overall, reflecting well on the Program management as well as the majority of host organisation staff and other Program participants. While no one respondent, or group of respondents, specifically defined what good practice in cultural appropriateness entails in a program such as ACIMH, the respondents over the course of the Program. These same themes, often using different terminology, are repeatedly referenced in various Program source materials. They are also referenced in a range of resources and advice on good ACIMH Program- Final Evaluation Report Page 95

107 practice in working with Aboriginal people and communities, such as in a recent NSW Family and Community Services resource 27. The pillar themes, summarised in Diagram 19 below, reflect language used by ACIMH Program participants. They comprise: Respect for Aboriginal people and for Aboriginal staff and trainees. This includes having culturally inclusive workplaces, work practices, signage and resources. Recognition of Aboriginal culture, including through the appointment of Aboriginal staff to engage with both male and female Aboriginal clients. It also includes staff being trained in cultural inclusiveness prior to trainees being recruited. Flexibility and acknowledgement of cultural differences including by accommodating trainees who have sensitive family and community commitments. Mutual Openness whereby trainees, their supervisors and other staff commit to and consistently practice open communication. They also respectfully discuss issues, exchange ideas and share perspectives within a supportive and encouraging team environment. Learning Environment where the heritage and lived experience of trainees and other Aboriginal staff are valued and utilised. This is essential not only to improve engagement with individual Aboriginal clients and communities, but also because of the learning opportunities it provides for the service in terms of inclusiveness, and for staff in terms of their own practice. Connectedness, recognising that individual trainees need connections to community and also regular contact with other Aboriginal people, whether other staff, mentors or other trainees. 27 NSW Department of Community Services Working with Aboriginal People and Communities: A Practical Resource, ACIMH Program- Final Evaluation Report Page 96

108 Diagram 19: Recommended ACIMH Cultural Appropr based on stakeholder feedback Source: EJD Consulting, drawn from consolidated respondent feedback. September It is recommended that these pillars be integrated, communicated and regularly reviewed by all parties involved in any future Aboriginal traineeship programs. R20. Integrate the cultural appropriateness pillars into future program resources and staff training and ensure they are regularly reviewed by all Program participants, including host organisation coordinators, supervisors and staff as part of a continuous improvement process. * * * * ACIMH Program- Final Evaluation Report Page 97

109 9 FINDINGS: OPERATIONS AND MANAGEMENT This section reviews the efficiency and effectiveness of the ACIMH Program from an operations and management perspective. It includes an assessment of the relative quality, timeliness, cultural appropriateness as well as value of various Program components. 9.1 Program Design and Operations The evaluators closely examined the Program establishment and implementation documentation and found it was generally well designed, clearly laid-out and articulated key components of the Program operations. When all respondent groups were asked how well run the Program was overall, respondents. Further: 96% rated the quality and efficiency When respondents were asked to rate specific components of the Program operations, the feedback was equally positive and broadly consistent across all four respondent groups. For example, over 75% of all respondents rated each of Overall quality of Program management Overall efficiency of Program management. Nil respondents provided a negative assessment on the above 28. Further, respondent groups were also very positive about the Program components. For example: capacity to adapt to trainee issues 28 That is, all of the 25% remaining assessments were rated as neutral rather than negative on the five point scale. ACIMH Program- Final Evaluation Report Page 98

110 terms of its capacity to adapt to host organisation issues. Across the four respondent groups, stakeholders rated the Program highest overall in terms of general Program design and management. Overall the ratings at the conclusion of the Program were higher when compared to feedback at the mid-project review (i.e. late 2013). When asked, if there was anything that could be done to improve the Program and its operations, responses varied, with some common feedback (across all respondent groups) including: Other comments included: Similar comments were provided in stakeholder focus groups and interviews, leading the evaluators to conclude that the Program overall had been well designed from the outset and also operated effectively. Be that as it may, some adjustments to the Program timeframe was widely endorsed. R21. Remodel the Program timeframes to: a) Extend the establishment and pre-recruitment phase thereby allowing: i) a longer period for conducting and then consolidating cultural training of host organisation staff ii) additional time to implement good practice recruitment practices (see Recommendation 21) and opportunities to identify and then recruit a broad pool of potential trainees (see Recommendation 1) b) Shorten the timeframes between the start and end of block training (for example to 12 months) thereby allowing trainees to graduate sooner. ACIMH Program- Final Evaluation Report Page 99

111 9.2 Program Governance and Staffing Program Oversight and Reporting As described in Section 2, the ACIMH Program involved core funding was provided by the Department of Prime Minister and Cabinet (PM&C) at the Commonwealth level (previously by DEEWR) and by the Department of Education and Communities (DEC) at the state level (see budget contributions in Table 5 below). To help finalise the overall Program design DEC established a Program Steering Committee, comprising senior representation from DEC, MHCC, and PM&C/ DEEWR. The Committee met regularly in late 2012 and early 2013 though thereafter meetings were significantly less frequent. This was in part due to a major restructure to AEU within DEC and a change of key personnel. As the time of preparing this report the most recent meeting of the Steering Committee was held in May 2014 to discuss the end of the formal phase of the Program and the graduation process. For the majority of the Program the meetings were reported to be imarily and served to ensure core MHCC accountabilities were on track. Prior to each Steering Committee meeting MHCC prepared a Program status report. In general the steering committee meetings were not used to discuss operational problems or other issues arising. Further, the committee was reported as to raise or resolve Program challenges, including for example inconsistency in the delivery of Aboriginal mentoring (see Section 6.4). Given the absences of alternative mechanisms for resolving issues between Program partners, this is deemed a weakness in the model. It should be rectified in any future governance arrangements, particularly where there are multiple program partners. R22. Ensure the Program Steering Committee meets regularly and adopts proactive terms of reference, which includes responsibility for discussing and resolving any high level operational challenges or joint management issues. ACIMH Program- Final Evaluation Report Page 100

112 To complement the DEC Program Steering Committee role, MHCC utilised its Aboriginal Reference Group as part of the governance arrangements. The Group was convened approximately three times per year and was used to provide cultural advice throughout the Program, including in respect to Aboriginal staff recruitment. For each meeting Group members received an ACIMH progress report or update. Reference Group members also made themselves available for ad hoc s and phone calls by MHCC staff. This was reported to be cultural i Given the challenges experienced in recruiting and maintaining Aboriginal personnel as part of the MHCC Program team (see Section below), their input was described as This was despite the fact that a number of Group members changed over time. A Program progress report was routinely provided to the MHCC board and also to the ACIMH Employer Network members on a semi regular basis. While each of these stakeholder reports provide some insight on the progress and status of the Program, MHCC did not prepare or disseminate any other formal or regular Program reports, nor any consolidated annual report. While the externally focused Yarn Up newsletters (see Section 9.5) as well as the preliminary independent evaluation report, prepared by EJD Consulting & Associates in December 2013, did document some key process and outcome issues, no other consolidated Program reports, including consolidated Program finance reports, were available for review. Whilst no major management or accountability issue has been revealed as a consequence of not having formal reports, the evaluators believe regular and structured reporting is a foundation of good governance and accountability and should be integrated into future management systems. Further, the evaluators consider the absence of regular reporting poses a structural risk had the Program experienced problems such as financial overruns or misappropriation. This potential risk to governance and Program oversight needs to be rectified in the future. R23. Future Programs incorporate quarterly Program update reports plus Annual Reports. These should be short and concise in format and be submitted to all external funders, as well as to the MHCC Board, for review and endorsement. Report sections might include: a) Program Status: covering the numbers and status of trainees and host organisations; training completed; updates on resources; communiques and other Program activities ACIMH Program- Final Evaluation Report Page 101

113 b) Feedback, Issues and Challenges: covering developments of recent months and how matters were addressed and resolved or where further action is required c) Financial Report: status of current income and expenditure, as well as projections d) Next Steps: highlighting major activities and events in future quarter and where additional input, advice or support is required Program Staffing As documented in the Program budget at Table 5 below, initial staffing allocated to the Program comprised a single Aboriginal Project Officer (costed at $149,000 over two years). While the MHCC did successfully recruit an Aboriginal person into this position on two different occasions, neither individual stayed in the position for as long as was hoped 29. As a result for the majority of time the Program was run by a MHCC team without any Aboriginal staff members. Nonetheless MHCC ensured that every block training included an Aboriginal trainer and that all trainees had access to an Aboriginal mentor (as discussed in Section 6.4). Had these two other avenues of support not been available to the trainees, the absence of an Aboriginal Project Officer may have been more significant. While clearly this situation was not ideal, stakeholder feedback documented in Section 8 indicates that the staffing situation did not appear to pose any significant challenges in terms of the P responsiveness overall. Nonetheless any future program needs to address this staffing situation if at all possible. R24. Investigate further options for recruiting and retaining Aboriginal Program staff, including: a) secondments from other organisations b) sponsorship of an Aboriginal trainee under the same or alternative traineeship program c) use of job-share or other flexible employment options. 29 The first Aboriginal Project Officer worked as a contractor for six months and an employee for three weeks before leaving; the second officer was an employee for just over three months before leaving. ACIMH Program- Final Evaluation Report Page 102

114 Supplementary to the Project Officer position, MHCC established an ACIMH Program Team within the organisation. This included: A Program Manager (Simone Montgomery). Simone also took responsibly for higher level liaison and reporting to funding bodies. Team leader (Chris Keyes). Chris, like Simone, had other ongoing MHCC responsibilities though managed all day-to-day operations and finances of the Program. She also convened the Trainee and Employer Network meetings. Project Support Officer (Liesl Homes). When the Aboriginal Project Officer was not filled, Liesl took responsibility for those duties as well. She was the prime interface with trainees and also was responsible for disseminating resources and s to all stakeholders (See Recommendation 31.) To complement the staff roles and expertise, MHCC also used four regular contract trainers (discussed in Section 5), as well as another contractor- Gillian Bonser. Gillian was contracted to develop and culturally tailor the training and assessment resources, support the trainers, and also provide other forms of cultural support and advice. With the exception of the contractor costs, and the salary and on costs of the Aboriginal Project Officer position, all other MHCC staff resources was provided in kind to the Program. This included significant staff time devoted to project management, project administration, information technology, communications and administration support, as well as the efforts of the MHCC Finance, Community Engagement and Promotions Officers. Over two years, these in kind staff costs were calculated at $90, This represents approximately 8% of actual expenditure. Note: As documented in Table 5, this figure is in addition to the Project Administration fee costed of $85,227 which covered general running costs, such as utilities, communications and postage, insurance and audit fees. In terms of in-kind support it should also be noted that DEC provided considerable un-costed support to the Program, particularly in the early stages through the AEU. For example, the AEU coordinated the recruitment phase of ACIMH, providing the majority of capacity visits, as well as support with culturally 30 This figure was calculated by asking each MHCC staff member involved in the project to estimate hours spent on the project multiplied by their pro-rata pay rates. ACIMH Program- Final Evaluation Report Page 103

115 appropriate recruitment strategies. They also delivered a series of workshops for supervisors and cultural awareness training for supervisors and managers. With the exception of the catering for the workshops, none of this AEU staff and management support is reflected in the budget following. R25. Ensure future Program budgets include an increase for project staff costs in recognition of the labour intensive nature of establishing, implementing and providing ongoing project support to trainees and host organisations involved in the Program. Also see Recommendation 26 regarding mentor costing. ACIMH Program- Final Evaluation Report Page 104

116 9.3 Program Finances Income and Expenditure As detailed in Table 5, the income for the ACIMH Program derived from a number of sources with the largest contribution, comprising approximately $725,000 contributed by PM&C in the form of a reimbursement grant to MHCC 31. Over half of this grant was used to provide incentive funding for host organisations to assist with wages and work-readiness training (see figures at Appendix 2). The remainder was used to cover training, staffing, mentoring and project administration expenses. Over the course of the Program, DEC made two funding contributions to MHCC in addition to their fee-for-service mentoring, host organisation staff training, and other employment advice roles. In late 2011 MHCC received an initial contribution of $100,000 to enable it to employ an Aboriginal Project Officer to assist with recruiting host organisations, stakeholder networking and in other ways promote the initiative. This figure was complemented by a MHCC contribution of $12,500 to cover other establishment costs. The second DEC contribution to the Program was approximately $160,000 to MHCC as a registered training organisation, as an incentive for training new entrant trainees. This figure was based on standing traineeship rates and was allocated on a per-trainee basis. MHCC also received a small fee from each host organisation to contribute to administrative support. A final income contribution, of $20,800, was recouped from additional fee-paying students who participated in the block training. Rather than allocate MHCC an annual or bi-annual grant, PM&C required MHCC to progressively invoice the Department based on actual expenses incurred. As documented in Table 5, this accounts for the significant variation between the projected budget and the actual income and expenditures. For example, the initial budget was based on a starting number of 45 trainees. It assumed some trainee attrition for block release income, but no attrition for employer incentive or mentoring claims. 31 When the Program was established this funding was allocated through DEEWR though through a restructure in 2013, administration transferred to PM&C. ACIMH Program- Final Evaluation Report Page 105

117 Table 5: ACIMH Consolidated Program Finances (Budget and Actuals) projected to May 2015* Item Budget Actuals* Income Grant Income - PM&C $991,226 $725,414 Grant Income - DEC $100,000 $100,000 Project Administration Fee- Host Organisations # $89,773 $85,227 Traineeship Costs- State Training Services, DEC $151,700 $160,534 Fee-Paying Students $10,500 $20,800 Total Income $1,343,199 $1,091,975 Expenses Employer Incentives $580,909 $418,187 Training Expenses $323,227 $237,274 Mentoring ^ $196,335 $99,580 Staffing Expenses: Plus in kind contribution from MHCC $149,000 $208,348 $90,050 Project Administration $90,228 $85,337 Employer and Trainee Networks $3,500 $2,852 Total Expenses $1,343,199 $1,141,628 Source: MHCC, November 2014; Updated December, * The table documents projected costs until the end of the financial year: o In the case of staffing costs, these are costed until end of December 2014 o In the case of employment incentives these are costed until the end of May 2015, at which time the MHCC management of the Program will conclude. + See Appendix 2 for details of the reimbursements provided. # Each host organisation contributed $ per annum per trainee to the Program. In the initial year the contribution was $40,909 in the subsequent year this dropped to $31,818. MHCC contributed the remainder in kind. ^ Mentoring costs were based on a rate of $50 per hour of contact time with MHCC invoiced by AEU as needed. No costs were budgeted or invoiced for travel or administration costs related to mentoring. ACIMH Program- Final Evaluation Report Page 106

118 According to MHCC, the actual attrition (from 45 to 32 trainee placements) accounted for the majority of difference between budgeted and actual grant income in respect to employer incentives and mentoring expenses. Further, some trainees received significantly lower levels of mentoring support than initially budgeted, thereby accounting for the difference in funding drawn down to cover this outgoing. The savings in training expenses primarily related to less outgoings for trainee travel and accommodation during block training than budgeted. In addition, some host organisations did not claim all entitled travel reimbursement. Anticipated travel costs were also reduced by booking well in advance. According to MHCC, the additional income from fee-paying students (totalling 6 individuals), plus the DEC contribution for new entrant traineeships, also helped keep the budget balanced. Whilst the above budget is inclusive of all MHCC income and expenditure, it does not include some other DEC contributions. For example, AEU mentoring costs (budgeted at approximately $196,000) was only reimbursed for face-to-face contact time. No funding was given through MHCC administration costs related to mentoring, nor to any support provided over the phone or in s. Were these additional labour costs billed for and reimbursed (as would be reasonable) the actual mentoring expenditure would have been much closer to the budgeted figure. R26. Ensure the full cost of mentoring, including travel and communication costs, is included in future budgets to more accurately reflect all labour costs involved in its delivery. Given that the traineeship program had not previously been managed by the same configuration of program partners in the same sector, the evaluators concluded the overall budgeting process was sound, notwithstanding previous commentary related to MHCC uncosted in-kind contributions (see Recommendation 25). There was however one aspect of the funding process that the evaluators regarded as inefficient and should be streamlined in the future. It relates to the invoice based grant allocation process that required MHCC to issue frequent, albeit irregular, itemised invoices based on actual expenditure. The specified amount would then be reimbursed to MHCC who in turn would then pay invoices as required. (In the case of mentoring costs, employment incentives and also preemployment training incentives, the process involved a fourth step in the payment process, with DEC or host organisations respectively commencing reimbursements by issuing MHCC with an initial invoice). Once again this process ACIMH Program- Final Evaluation Report Page 107

119 was frequent though not based on any conventional reconciliation period, such as end-of-month or quarterly. This highly labour intensive accounting process, which required double, sometimes triple or quadruple, handing of invoicing occupied a substantial amount of the While this system of payment is understood to be standard across other similar PM&C/DEEWR funding programs, the evaluators strongly believe the process needs to be overhauled and streamlined whilst maintaining transparent accountability and reporting requirements. R27. Streamline how future program grant funding is allocated with preference for consolidated block funding (on a quarterly or half yearly basis) based on standard financial reporting statements with commentary to adjust for variations and under or overspends as needed. Based on the available figures, a crude unit cost per trainee can be calculated at approximately $24,800 ($15,700 if excluding the employment incentives) 32. It should be noted however that these figures also include the significant resources and effort that was devoted to cultural change and capacity building in host organisations as discussed throughout this report. Further, the figures include effort and activities devoted to establishing and refining Program systems, resources and documentation including in respect to: training curricula and training materials trainees and host organisation resources general administration systems, forms and communication tools required of all new programs. As such the evaluators estimate that should the Program be rerun the overall operating and unit costs could be reduced albeit requiring comparable staffing levels due to the labour intensive nature of supporting trainees and their host organisations as documented elsewhere in this report. 32 The calculation is based on total Program expenditure (excluding the uncosted and in kind contributions of DEC staff in terms of host organisation training and employment advice) divided by 46. ACIMH Program- Final Evaluation Report Page 108

120 R28. Based on actual expenditure, and factoring in other key findings contained in this report, prepare an updated ACIMH Program budget to inform future Program funding and operations. The budget should be itemised against specific operational costs and incorporate: a) full costs of Aboriginal mentoring support (see Recommendation 26) b) Specific Program management items (including for example items previously supplied in kind) c) Any additional training or resourcing costs raised by other recommendations in this report Program Incentives As described at Appendix 1, through the PM&C/DEEWR grant funding successful host organisations received a Commonwealth Government employment incentive to employ an Aboriginal trainee. When host organisation respondents were asked to rate the importance of this incentive, the feedback was divided. Of those that expressed an opinion: When asked how important a future employment incentive would be to their decision to employ future Aboriginal trainees, the results were more skewed. Of those that expressed an opinion: 33. When host organisation staff were further questioned on incentives and future options in terms of traineeships, two very clear themes emerged: 1) Community mental health providers are strongly committed to employing more Aboriginal trainees and staff 2) Without some additional support to cover salary costs and other outgoings, the capacity to support future trainees would be impossible HO) or (HO). As one supervisor and service coordinator summed up: 33 One respondent provided a neutral response. ACIMH Program- Final Evaluation Report Page 109

121 and the trainee program is a great option for delivering this, but e This common perspective, combined with earlier feedback contained in this report gives rise to the following recommendation: R29. Advocate for and secure future employment incentive funding, such as that previously supplied by PM&C/ DEEWR, as part of any future traineeship program to provide additional incentive and financial capacity to host organisations. 9.4 Program Contributors While Section 6.2 contains trainee feedback on the level and quality of support they received from different categories of staff, what follows is the consolidated feedback provided on key Program contributors Role of STS Aboriginal Employment Unit Respondents were asked to rate the STS-AEU involvement in the Program. Overall, responses were positive, with nearly three-quarters responding positively to: Providing advice at establishment phase including on trainee recruitment (72%) The value of STS-AEU supervisor workshops (66%). A modest majority (52%) also rated the STS-AEU cultural training positively, As documented in Diagram 20, however, perceptions of the STS role in other aspects of the Program were somewhat less positive, with markedly lower responses for the role of Aboriginal mentors (as discussed in Section 6.4). For example, when assessing Aboriginal mentor support for supervisors, just 45% of 52% of all respondents positively rated the selection of Aboriginal mentors and matching of Aboriginal mentors with trainees 38% rated selection and matching as across all STS measures. ACIMH Program- Final Evaluation Report Page 110

122 Diagram 20: STS-AEU Program Involvement Source: EJD Consulting, ACIMH stakeholder survey (n=62), August 2014 Note: Based on a responses as Well Not Well. Across all four respondent groups, mentors themselves rated the Program highest in terms of STS-AEU involvement in the Program. Overall these numbers show a decrease when compared to those of the midprogram review, most notably with regard to the STS-AEU Aboriginal mentors, who showed the most significant decline. The results of the 2014 survey were similar to that of the mid-project survey, with the exception of the question regarding matching of mentors with trainees which was rated significantly less positively than it previously had been. See recommendations in Section Mental Health Coordinating Council the overall responses were very positive. As summarised in Diagram 21, Timely response to queries and issues (86%) Resources and information to support trainees (76%) Information sessions and resources at start of Program (86%) ACIMH Program- Final Evaluation Report Page 111

123 Resources and information to support supervisors (76%) Advice on availability of and access to incentives (83%). Respondents rated the MHCC somewhat less well in terms of timely processing This was nonetheless an improvement from the figures of the mid-project report. R30. Investigate options to simplify and streamline payment and reimbursement systems for host organisations. Diagram 21: MHCC Program Involvement Source: EJD Consulting, ACIMH stakeholder survey (n=62), August 2014 Note: Based on a 5 point Like responses as Well Not Well. Trainee feedback on MHCC staff, gathered through training block surveys and also in the focus groups and interviews, was extremely positive. For example, helpful, and to describe their perceptions of MHCC Program staff. Further, analysis of respondent feedback from across the Program led the ACIMH Program- Final Evaluation Report Page 112

124 success can be directly attributed to the commitment, professionalism and Program. Liesl Homes, Chris Keyes and Simone Montgomery were all repeatedly referenced by name in focus groups and interviews as key strengths of the Program, with the first two reported as being the and particularly for trainees and their supervisors. (Note: Feedback related to the personal contributions of trainers is discussed in Section 5). While this report finds the overall structure and documentation of the Program to be sound and well designed, it also finds that the individual effort, flexibility and dedication of Program staff, working closely with trainees, their supervisors, trainers and other host organisation staff, should not be overlooked as a major contributing factor. Any future agency overseeing an Aboriginal traineeship program should be aware of this and go to all lengths possible to recruit and retain quality staff that demonstrate and communicate similar levels of professionalism, dedication and practicality as the MHCC ACIMH Program team. The following responses sum up this broad acknowledgement of the MHCC coordinator respectively: me back, (T) ogram was about people and everyone working to make it a success. That initially all sprang from the folk at the MHCC and the enthusiastic way they them that often, the fact that they were there and so readywilling-and-able to help, made our jobs so much easier. It really that we could call them up and acknowledge that, as sometimes new things [programs] can read and sound all very good, and then nothing, and no one to back it up. This was the exact opposite of that. Admin aside, info, advice, support was available at the outset and right to the very end. It ACIMH Program- Final Evaluation Report Page 113

125 Given this wide-spread acknowledgement of individual effort, and the fact that personal contact was so highly valued by all participants, the evaluators recommend that all future correspondence include the staff names. R31. Ensure all future Program e-correspondence include the first name of the When respondents were asked if there were any aspects of the role of MHCC that could be improved, a number of suggestions were nonetheless raised: - rum or briefing for frontline managers, not just Head the trainees would benefit from having would have supported. These suggestions, combined with other feedback gathered over the course of the Program and contained in other sections of this report, led to the following recommendations: R32. Prior to the recruitment of any future trainees, MHCC should: a) Complete and distribute all Program Guidelines and key supporting documentation to host organisations, including direct to nominated supervisors b) Provide briefing opportunities and training directly to frontline managers and potential supervisors as well as to Service Coordinators c) In host organisation briefings and documentation, incorporate additional advice and discussion on good practice addressing: i) trainee recruitment and induction ii) Diagram 10) iii) Diagram 18) iv) Diagram 14) effective communication and trainee support strategies (see what culturally appropriate culture and practice means (see managing or ameliorating anticipated challenges (see ACIMH Program- Final Evaluation Report Page 114

126 v) outline of block training module content and assessment schedule and expected roles and responsibilities (see Recommendation 32) vi) options for making optimal use of Aboriginal mentors for both the trainee and the host organisations (see Recommendation 6) 9.5 Program Documents and Resources As summarised at Section 4.4, across the course of the Program the MHCC Program managers generated significant quantities of resources and communication tools. When respondents rated how useful some of the key resources were, the overall response rate was positive, though there were some notable variations across respondent groups and resource categories. Diagram 22: Value and Usefulness of ACIMH Program Resources Source: EJD Consulting, ACIMH stakeholder survey (n=62), August 2014 As summarised in Diagram 22, (noting no negative responses received): The Yarn Up Newsletter rated highest overall, with 85% of all respondents The MHCC website containing ACIMH information rated well at 80% Across respondent groups it was rated most useful for stakeholders and mentors. ACIMH Program- Final Evaluation Report Page 115