EVALUATING A FINANCIAL LITERACY PROGRAM: THE CASE OF THE AUSTRALIAN MONEYMINDED PROGRAM

Transcription

1 EVALUATING A FINANCIAL LITERACY PROGRAM: THE CASE OF THE AUSTRALIAN MONEYMINDED PROGRAM Roslyn Russell *, Robert Brooks ** and Aruna Nair * * Research Development Unit, RMIT Business ** Department of Econometrics and Business Statistics, Monash University ABSTRACT The promotion of higher levels of financial inclusion has become a major issue of policy concern and action across a number of countries, including the US, UK and Australia. Typically programs to improve financial inclusion involve either a matched savings program and/or financial literacy education. In the context of financial literacy education, Fox, Bartholomae and Lee (2005) have recently proposed a five tiered approach to program evaluation. This paper provides an illustration of the application of this approach in the context of a particular Australian financial literacy program, specifically the MoneyMinded program. The results demonstrate the general applicability of the framework. Further illustrations are provided on how statistical techniques might be used, and how longer-term impact might be assessed. Correspondence to: Robert Brooks, Department of Econometrics and Business Statistics, Monash University, PO Box 1071, Narre Warren Victoria 3805, Australia Phone: , Fax: , robert.brooks@buseco.monash.edu.au Acknowledgements: We wish to acknowledge the financial support of the ANZ Banking Corporation in the conduct of this research. We also wish to thank the participants and facilitators in the MoneyMinded program for their willing provision of evaluation data on their experiences. In addition we also wish to thank two anonymous reviewers for their helpful comments on an earlier version of this paper. 1

2 EVALUATING A FINANCIAL LITERACY PROGRAM: THE CASE OF THE AUSTRALIAN MONEYMINDED PROGRAM ABSTRACT The promotion of higher levels of financial inclusion has become a major issue of policy concern and action across a number of countries, including the US, UK and Australia. Typically programs to improve financial inclusion involve either a matched savings program and/or financial literacy education. In the context of financial literacy education, Fox, Bartholomae and Lee (2005) have recently proposed a five tiered approach to program evaluation. This paper provides an illustration of the application of this approach in the context of a particular Australian financial literacy program, specifically the MoneyMinded program. The results demonstrate the general applicability of the framework. Further illustrations are provided on how statistical techniques might be used, and how longer-term impact might be assessed. 2

3 Introduction The promotion of greater levels of financial inclusion is now seen to be a major public policy and corporate initiative across a variety of countries. The core elements of programs to promote greater financial inclusion appear to include matched savings programs and financial literacy education. In the United States such programs include the American Dream Demonstration and its emphasis on the role of Individual Development Accounts in asset accumulation (see Sherraden (1991), (2000); Beverley and Sherraden (2001); Schreiner, Clancy and Sherraden (2002). In the United Kingdom such programs include the Saving Gateway (see Kempson, McKay and Collard (2003), while in Australia such programs include Saver Plus (see Russell and Fredline (2004); Russell and Wakeford (2005), Russell, Brooks, Nair and Fredline (2005, 2006)). With a range of programs on offer across a variety of countries, effective evaluation of their role in improving financial inclusion is essential. In the context of financial literacy education, Fox, Bartholomae and Lee (2005) have recently provided an overview of the wide range of financial literacy education programs available in the United States and discussed their impacts. Fox, Bartholomae and Lee (2005) also discuss the evaluation of such programs, noting that to date no single framework appears to guide the evaluation process. They then extend the framework of Jacobs (1988) to develop a five tiered evaluation framework against which evaluations of financial literacy education programs can be conducted. The framework proposed by Fox, Bartholomae and Lee (2005) is general in its potential application. As such, the purpose of the present paper is to explore this evaluation framework in the context of a financial literacy program designed and developed in a different country, specifically the MoneyMinded financial literacy program that is offered in Australia. This provides an extension to the work of Fox, Bartholomae and Lee (2005) that developed and discussed their framework in the context of United States programs. The consideration of a program from outside the United States thus provides information on the generalisability of the framework. Further, the results are likely to be of interest across a range of countries given the comparable initiatives relating to financial inclusion across a variety of countries. The plan of this paper is as follows. Section two provides an overview of the evaluation framework that is developed and discussed in Fox, Bartholomae and Lee (2005). Section three discusses the evaluation of the MoneyMinded program undertaken by Russell, Brooks and Nair (2005) in the context of that framework. Section four then contains some concluding remarks, which can be drawn from this wider application of the framework. 3

4 The Evaluation Framework A key component in an effective financial education program is an evaluation of the effectiveness of such a program. To aid such a process, Fox, Bartholomae and Lee (2005) propose a five-tiered evaluation framework based on the work of Jacobs (1988). The five tiers in this framework are: (1) pre-implementation, (2) accountability, (3) program clarification, (4) progress towards objectives and (5) program impact. The pre-implementation stage involves a needs assessment on the financial literacy training required by participants, this might be in terms of the needs identified from more general financial literacy testing and/or specific needs assessment around program participants. For a US example targeted at low income groups see the evaluation of the FLLIP program reported in Anderson, Zhan and Scott (2004). The accountability stage involves the collection of data on the education and services provided as part of the program, the costs involved and demographic information on participants. The program clarification stage involves ongoing assessment of the program s strengths and weaknesses, and plans to review the program in the light of such assessments. For US examples of such evaluations see Anderson, Zhan and Scott (2004) and Lyons and Scherpf (2004). The progress towards the objectives stage involves the collection of data on the impacts of the program on participants. For a US example of the need to clearly define program impact goals in the context of the FDIC Money Smart program see Lyons and Scherpf (2004). Finally, the program impact stage builds on the previous stage in terms of long and short-term impacts, typically involving comparison of participants and non-participants. The Fox, Bartholomae and Lee (2005) framework is potentially very general in terms of its ability to evaluate financial literacy programs. In particular, there is nothing about the framework that is particularly specific to the US context in which it was initially proposed. Therefore the use of the framework to discuss the evaluation of an Australian financial literacy program is valuable in informing as to the general applicability of the framework. 4

5 Evaluating the MoneyMinded program Having outlined the five tiered approach to program evaluation advocated in Fox, Bartholomae and Lee (2005), this approach is now applied to considering evaluation of the Australian MoneyMinded program. The MoneyMinded program has a broad objective of assisting people to make informed financial decisions and take control of their finances for their future. The MoneyMinded program consists of 17 workshops which are separated into six key topics areas, specifically, Planning and saving, Easy payments, Understanding paperwork, Living with debt, Everyday banking and financial products and Rights and responsibilities. The program allows individual participants to choose which workshops they wish to complete. The program was developed, in part, as a response to the results from the first-ever major survey of consumer financial literacy in Australia (see ANZ 2003)). The results of this survey identified both those groups within the community that could benefit from improved financial literacy education and also the issues that could be addressed in such an education program. Further details on the program and its objectives are available on the MoneyMinded website ( The MoneyMinded program is currently at the initial roll-out stage involving training being facilitated by the involvement of five major community groups. Previous research has found that delivery of a matched savings program through the involvement of community groups has been successful (for details see Russell and Fredline (2004); Russell and Wakeford (2005); Russell, Brooks, Nair and Fredline (2005, 2006)), and as such that approach of involving community groups in the delivery of the program has also been extended to financial literacy education. The long-term plan of ANZ is to partner with around 100 community organisations during the next five years to deliver the MoneyMinded program to 100,000 people nationally. Further details on the current status of MoneyMinded and the future plans can be found on the ANZ website ( Given these long-term plans for such a comprehensive and extensive delivery of the program, an evaluation of the initial roll-out would seem critical. In fact, such an evaluation is consistent with Fox, Bartholomae and Lee s (2005) noting of the Jacobs (1988) emphasis on the importance of evaluation being collected and analysed in a systematic manner. Having briefly outlined the MoneyMinded program, attention now turns to considering its evaluation in terms of the five tiered approach. The first stage in this approach is the pre-implementation needs assessment. This needs assessment in the first instance did not involve a formal needs analysis at the individual participant level but instead is more generally based on the results of the financial literacy survey reported in ANZ (2003). The ANZ (2003) financial literacy survey involved asking people questions about financial literacy via a telephone survey customised to their particular needs and circumstances, as distinct from asking general questions about financial literacy that may not have been relevant to particular individuals. The ANZ (2003) survey identified that the following groups were strongly over-represented in the lowest financial literacy quintile: lowest education levels, not working or in unskilled work, lower income levels, lower savings levels, single and those at the extreme ends of the age profile. There was also modest overrepresentation of females. A selection of these quintile results are summarised in table 1. These results suggest a targeting of the program at these over-represented groups. In the context of the MoneyMinded program this has been achieved through the involvement of community groups in facilitating delivery to participants. In the context of the matched savings program Saver Plus, this involvement of community groups has also proven to be successful in the recruitment of participants (for details see Russell and Fredline (2004); Russell, Brooks, Nair and Fredline (2005, 2006)). There are other elements of the MoneyMinded program that have also addressed needs assessment at the pre-implementation stage. The program required participants to complete a pre-training questionnaire. While this questionnaire captured a range of demographic information (which is relevant for the accountability stage in the Fox, Bartholomae and Lee (2005) framework), it also captured data on the areas that participants identified as where they most needed financial literacy education in terms of the modules offered as part of the MoneyMinded program. This information is summarised in table 2 and further qualitative information is available in Russell, Brooks and Nair (2005). Together with the fact that the MoneyMinded workshop is composed of individual workshops the participation in which is chosen at the individual level suggests that the design performs well on the criteria relevant to the pre-implementation stage. The accountability stage of the evaluation framework requires that details be collected on a range of matters including the education and services provided and base participant information. The details on the education and services provided as part of the program are available on the MoneyMinded website ( In terms of basic participant information the pretraining questionnaire collected data on demographics (including gender, age, education levels, income, employment), mathematical and computer literacy (the results in ANZ (2003) suggest that these are important factors in financial literacy), internet usage, current savings and spending behaviour, savings 5

6 intentions, and current usage of a range of financial products. Further, the pre-training questionnaire also captured data on the financial literacy of participants by asking questions on their understanding of credit card liability, their understanding of financial products, their understanding and approaches to bill payment, how they would handle unexpected charges by their banks and their attitude to debt. A summary of the socio-demographic data collected is reported in table 3 and more details on this data and a summary of results are available in Russell, Brooks and Nair (2005). The collection of such data provides the potential for a rich analysis to inform the program clarification, progress towards objectives and program outcome stages of any evaluation by allowing an exploration of the influences of these demographic and financial factors. This type of data collection is also broadly consistent with the data collected in the US studies reviewed by Fox, Bartholomae and Lee (2005). The program clarification stage requires an assessment of the program strengths and weaknesses, and the use of such information in any re-assessment of program goals and objectives. This element is also present in the MoneyMinded evaluation. While the long term plan for MoneyMinded is for a large scale delivery, the initial roll out to a small number of community organisations and participants allows for the type of assessment and re-thinking that is at the core of the program clarification stage. In addition to collecting pretraining data from participants, the MoneyMinded evaluation also involved the administration of post-training questionnaires to participants. This questionnaire requires participants to provide an assessment of each workshop completed in terms of how useful they found the workshop to be, and how satisfied they were with the topics and material covered in the workshop. This data took the form of an ordinal rating on a seven point scale. The post-training questionnaire also required an assessment of overall program satisfaction, again in the form of an ordinal rating on a seven point scale. This post-training evaluation data when coupled with the pre-training data provides the scope for a rich analysis at the program clarification stage. The results reported in Russell, Brooks and Nair (2005) show high overall ratings in terms of satisfaction and usefulness. More specifically, in terms of the workshop level data the results showed high usefulness ratings (40.6% of participants found the workshops to be extremely useful, while another 46.7% of participants found the workshops to be useful), and high satisfaction ratings (42.8% of participants were very satisfied with the topics and material covered in the workshop, while another 47.2% of participants were satisfied with the topics and material covered). These high ratings at the workshop level also transferred to high levels of satisfaction with the overall program (45.3% of participants reported being very satisfied with the overall program, while another 38.7% of participants reported being satisfied with the overall program). While these high ratings in terms of usefulness and satisfaction provide some comfort in terms of the quality of the overall program and the constituent workshops, a richer analysis for the program clarification is potentially provided by considering whether there are differences in the satisfaction and usefulness ratings across categories in the demographic and financial data collected in the pre-training questionnaire. In this context, Russell, Brooks and Nair (2005) found statistically significant differences in terms of usefulness and satisfaction ratings across participant categories. More specifically, Russell, Brooks and Nair (2005) found higher ratings were provided by those in categories that captured lower incomes, education levels, employment status, mathematical and computer literacy, savings levels and intentions and base levels of financial literacy. A selection of these results is reported in table 4, with the full set of detailed results available in Russell, Brooks and Nair (2005). Linking this result back to the needs assessment at the preimplementation stage provides richer information on the program, over and above that provided by the satisfaction and usefulness ratings at the aggregate level. The bulk of the data collected in both the pre-training and post-training questionnaires used in the MoneyMinded evaluation conducted by Russell, Brooks and Nair (2005) is categorical in nature. Therefore the statistical testing was conducted using the χ2 testing framework appropriate to such data. An obvious issue in any formal statistical testing is whether the data collected satisfies the assumptions required of the statistical procedure being used. In terms of χ2 testing of categorical data the key assumption is around the expected cell count size, where the conservative rule of thumb is an expected cell count of 5 for the asymptotic χ2 approximation to work. With an evaluation instrument allowing for ratings on a seven point scale, the evaluation being focused on the initial roll-out, and the heavy concentration of ratings at the high end of spectrum this requirement was not satisfied in the initial data. There are two approaches available to those doing an evaluation in this context. The first approach is to aggregate cells (in this case rating categories) together and then conduct the tests. Given the MoneyMinded ratings this involved aggregating together the lower usefulness and satisfaction ratings categories. The second approach is to conduct a simulation analysis of the exact distribution of the test statistic typically involving some Monte Carlo technique, a feature that is now available in many standard software packages, including SPSS. Overall, this suggests that appropriate statistical analysis can be beneficial at the program clarification stage of a structured evaluation process. The post-training questionnaire also provided useful data on some dimensions of program delivery. A website of resources was made available to participants, however the results suggested that participants 6

7 made little use of the website. Russell, Brooks and Nair (2005) found that 92% of participants did not access the website. This is a strong result, as it is also suggests that use of technology in delivery should be assessed, and just not assumed in the design features. The post-training questionnaire also captured data on the number of individual workshops that participants attended. The median number of workshops attended by participants was 2. The majority of participants attended only a single workshop, while a small number of participants attended 12 workshops. Interestingly the results in Russell, Brooks and Nair (2005) show that participants who attended multiple workshops had higher usefulness and satisfaction ratings than those participants who attended a single workshop. Further, these differences in the ratings are statistically significant. This suggests that consideration be given in the program design to encouraging attendance at multiple workshops via greater bundling, while recognising that workshops are already bundled by broad topic area. This is useful and constructive information to obtain at the program clarification stage of an evaluation. There is however an issue in the statistical analysis of the ratings, where individuals provide multiple ratings across a number of workshops, in that such responses are likely to be correlated across the workshops attended by that individual. This issue is explored in the context of a cluster sampling framework by Fry, Mihajilo, Russell and Brooks (2006). The assessment of MoneyMinded also involved the completion of questionnaires by the facilitators who actually delivered the financial literacy training, in terms of their expected usefulness of the workshops prior to delivery, and then their perceptions of participant usefulness post-delivery. In both of these assessments Russell, Brooks and Nair (2005) found that the ratings were high, and were broadly consistent with the ratings provided by participants. This again is useful data at the program clarification stage in informing how the program goes forward. The final data collected of relevance to the program clarification stage was questionnaire data from both participants and facilitators as regards improvements that could be made to the program in terms of expanding the topic coverage. In this regard, a number of responses suggested the need for workshops on investment, superannuation and retirement planning. To some extent this is to be expected in the Australian context where a major emphasis of government policy in recent years has been to move the funding of retirement incomes from the pension system out of government funds to private funding via long-term superannuation investments. For some discussion of this policy focus see McKeown (2001). The fourth stage of the Fox, Bartholomae and Lee (2005) approach to evaluation is in terms of progress towards objectives. Clearly, the objective of a financial literacy education program is in terms of improved financial literacy, and as a consequence better financial decision making by participants. However in the context of an initial evaluation of the roll-out of what is to become a wider program, such a stage is difficult to effectively action, except in terms of interpretation of ratings of satisfaction and usefulness that have been previously utilised at a program evaluation stage. Following on from progress towards objectives is the final stage of program impact, both in a short-term and long-term sense. Clearly, here one is looking for wider demonstrable evidence in terms of improved financial literacy and better financial decision making by those who have participated in financial literacy programs. The recommendation in Fox, Bartholomae and Lee (2005) is that this measurement of impact be done relative to a control group that have not participated in such training. This stage is clearly not relevant in the current assessment of MoneyMinded, given that the program is only at an initial roll-out stage. In fact, Fox, Bartholomae and Lee (2005) note that this is characteristic of most programs even in the US, thus making it premature to make any definitive statements as regards the impacts of financial literacy education. However, in that context, it is worthwhile speculating as to how such an evaluation might be built for use in the future. The needs assessment for MoneyMinded derives in part from the ANZ (2003) survey on adult financial literacy. One mechanism to analyse longer-term program impacts would be to make sure that future broad surveys of financial literacy include questions for the respondents on whether they have previously participated in a financial literacy education program. One could then compare measures of financial literacy across groups that have/have not undertaken financial literacy education, after making appropriate adjustments for other socio-economic and demographic factors that are known to influence financial literacy. This would have the potential to provide for a longer term assessment. A further mechanism to measure program impacts would be in terms of the impacts on savings behaviour of participants who have been through financial literacy education programs. The current Saver Plus program includes some base financial literacy education (for details see Russell and Fredline (2004); Russell, Brooks, Nair and Fredline (2005, 2006)). Thus a longer-term analysis of the savings behaviour of participants in a matched savings program would provide evidence of the joint impacts on financial decision making of participation in both a matched savings program and financial literacy education. 7

8 Conclusion This paper has conducted an exploration of the Fox, Bartholomae and Lee (2005) framework for the evaluation of financial literacy education programs in the context of a specific program, the Australian MoneyMinded program. In general, the framework developed in the context of US programs appears to generalise well to the Australian context, thus providing support to the general applicability of the framework. In applying the framework the paper also illustrates how formal statistical evaluation might be used at certain stages. Further the paper also speculates on how measurements of longer-term impact might be conducted in terms of the program. References Anderson, S., Zhan, M. and Scott, J. (2004), Targeting Financial Management Training at Low Income Audiences, Journal of Consumer Affairs 38, ANZ Banking Group (2003), ANZ Survey of Adult Financial Literacy in Australia, Roy Morgan Research. ANZ Banking Group (2005), ANZ Survey of Adult Financial Literacy in Australia, AC Neilsen. Beverley, S. and Sherraden, M. (2001), How People Save and the Role of IDAs, in R. Boshara (ed.), Building Assets: A Report on the Asset Development and IDA Field, Corporation for Enterprise Development. Fox, J., Bartholomae, S. and Lee, J. (2005), Building the Case for Financial Education, Journal of Consumer Affairs 39, Jacobs, F. (1988), The Five Tiered Approach to Evaluation: Context and Implementation, in H. Weiss and F. Jacobs (eds.), Evaluating Family Programs, New York: Aldine DeGruyter. Kempson, E., McKay, S. and Collard, S. (2003), Evaluation of the CFLI and Saving Gateway Pilot Projects, Interim Report on the Saving Gateway Pilot Project, Personal Finance Research Centre, University of Bristol. Lyons, A. and Schrepf, E. (2004), Moving from Unbanked to Banked: Evidence from the Money Smart Program, Financial Services Review 13, McKeown, W. (2001), Superannuation, Chapter 12 in D.Beal and W.McKeown (eds.), Personal Finance, John Wiley and Sons. Russell, R., Brooks, R. and Nair, A. (2005), Evaluation of MoneyMinded: An Adult Financial Education Program, RMIT University, Russell, R., Brooks, R., Nair, A. and Fredline, L. (2005) Saver Plus Improving Financial Literacy Through Encouraging Savings: Evaluation of the Savers Plus Pilot Phase 1 - Final Report (Melbourne: RMIT University). Russell, R., Brooks, R., Nair, A. and Fredline, L. (2006), The initial impacts of a matched savings program: the Saver Plus program, Economic Papers 25, Russell, R. and Fredline, L. (2004) Saver Plus Progress and Perspectives: Evaluation of the Savers Plus Pilot Project Interim Report (Melbourne: RMIT University). Russell, R. and Wakeford, M. (2005), Saver Plus: Saving for the Future, Proceedings of the Transition and Risk: New Directions for Social Policy Conference, Centre for Public Policy, University of Melbourne. Schreiner, M., Clancy, M. and Sherraden, M. (2002), Saving Performance in the American Dream Demonstration: A National Demonstration of Individual Development Accounts, Final Report, Center for Social Development, Washington University. Sherraden, M. (1991), Assets and the Poor: A New American Welfare Policy, Armonk, NY: M.E.Sharpe Inc. Sherraden, M. (2000), From Research to Policy: Lessons From Individual Development Accounts, Journal of Consumer Affairs 34,

9 Table 1: Representation of certain groups in the lowest financial literacy quintile This table reports details of a selection of the groups that are over represented in the lowest financial literacy quintiles of the ANZ financial literacy surveys in 2003 and The table reports the percentage of respondents in the lowest financial literacy quintile for a series of socio-demographic groups Group 2003 Survey (%) 2005 Survey (%) Females Education Less than Yr Looking for Work Semi-skilled Unskilled Single Living Alone Aged Aged Renting Data Sources: ANZ Surveys of Adult Financial Literacy in Australia 2003 (p.5) & 2005 (p.22). Table 2: Areas of interest for greater financial knowledge amongst participants in terms of MoneyMinded modules This table reports the MoneyMinded modules identified by participants as bein areas where they had an interest in acquiring greater financial knowledge. Module Area Number of Interested % of Total Participants Participants Planning and saving Living with and managing your debt Your consumer rights and responsibilities Understanding financial paperwork Your everyday banking and financial products Your payment options Source: Russell, Brooks and Nair (2005), Table 4 p.12. 9

10 Table 3: Selected characteristics of the MoneyMinded participants This table reports the characteristics of the MoneyMinded participants across a range of socio-demographic characteristics. Number of Participants Gender Male Female Age Education Primary/Some Secondary All Secondary TAFE/Workplace Training University Employment Status Full time Part time Casual Unemployed/Not looking Student Home duties Retired % of Total Source: Russell, Brooks and Nair (2005), Table 1 p.8; Table 2 p

11 Table 4: Selected comparison of satisfaction and usefulness of the overall program and workshops This table presents a selection of the results on whether the ratings of satisfaction and usefulness provided by participants vary across the socio-demographic, financial product usage and financial literacy groupings. Group χ 2 d.f. χ 2 value p-value Program Satisfaction Income Source Employment Status Income Level Savings Intentions Savings Level Spending Behaviour Understanding Bills Credit card usage Workshop usefulness Income Source Employment Status Income Level Savings Intentions Savings Level Spending Behaviour Understanding Bills Credit card usage Workshop Satisfaction Income Source Employment Status Income Level Savings Intentions Savings Level Spending Behaviour Understanding Bills Credit card usage Source: Russell, Brooks and Nair (2005), Table 17 p.26, Table 18 p.28, Table 19 p.29 11