Training Return on Investment (ROI) on the rail and road infrastructure manager Portugal

Transcription

1 Training Return on Investment (ROI) on the rail and road infrastructure manager Portugal Bernardino, G. (Infraestruturas de Portugal), Curado, C. (ISEG, Universidade de Lisboa) Article Information Keywords: Training Training Evaluation Return on Investment (ROI) Alternative causal configurations Corresponding author: Gonçalo Bernardino Tel.: eportugal.pt Abstract The purpose of this paper is to contribute with evidence on training evaluation, estimating the return on investment (ROI) of 327 training programs by using data from the Portuguese railway infrastructure manager. We address return on investment from numerous training programs in different functional areas in search for differences. This study uses company records on training programs; direct and indirect costs, and output indicators on global performance levels. The estimated ROI for each training program considers the associated costs and estimated benefits. Results show that training programs from different functional areas present unlike ROI values. Findings offer alternative configurations of causal conditions associated with the training ROI and its absence.

2 1 Introduction 1.1 Reasons for research The relevance of "skills" as the core aspect of the training activity demands appropriate organizational training strategies. Skills certification pushes traditional training models to consider the use of training valuation models. When considering training evaluation, the most common measurement model was developed by Kirkpatrick (1959), and published in a series of articles at the "US Training and Development Journal". The model gained huge acceptance in the context of vocational training by addressing the particular aspect of training transfer to the workplace [12][1]. Currently the Portuguese railway infrastructure manager applies an evaluation design that only covers the immediate reaction of the trainees to the training experience regarding trainees satisfaction. This study aims to apply the higher level of training evaluation in the Kirkpatrick s model hierarchy: the return on investment (ROI) analysis. The main goals of this paper are: 1) Estimating the ROI of all training programs taking place over a year (using the 2014 data base); 2) Searching for differences among estimated ROI considering the associated training programs; 3) Analyzing the necessary and sufficient conditions of ROI and its absence; 4) Identifying the causal configurations associated to ROI and its absence. 1.2 Return On Investment (ROI) Organizations make large investments in training because training increases organizational knowledge, as a result they expect the outcomes of training to be measurable [19]. Thus, effectiveness of the investment made in training is a concern of organizations [15]. Several models and theories offer contributions, like the general organizational assessment tools, including the Balanced Scorecard, developed by Kaplan and Norton [16]. Regarding training in particular, one of the most popular available tool is the Kirkpatrick s four levels model. This model originally had a great impact on training evaluation regarding financial and planning aspects, preceding the appearance of the "ROI model" of Phillips [21] which maintains a modular structure very similar to that used by Kirkpatrick and adding a fifth level, the "return on investment". Training ROI estimation is based on a formula that determines the organizational return on training investments. The Phillips framework of 5 levels introduced an approach to training focusing on the returns associated to the training investment, expressed as a percentage or ratio [23] [22]. Managers are sensitive to the organizational impacts of training. Moreover, since they make large investments in training they expect the outcomes or benefits related to training to be measurable [19]. They are interested in ensuring the return on such investments and to be able to measure it. There is often some confusion between the return on investment and the relationship between the cost / benefit ratio. The ROI estimation process involves collecting performance data, associating it to the training events, considering the related total costs and determine the associated benefits. ROI valuation should be based on conservative estimates [2]. 1.3 Associated difficulties to the ROI estimation Managers argue that training ROI estimation is difficult to preform, thus it is hard to measure it accurately [2]. When determining the training ROI, a series of inaccuracies can occur leading to ambiguity [21]. Costs can be identified before the training event occurs, but the benefits may accumulate up slowly over time. There are wellknown difficulties involving the ROI estimation; the lack of preparation or professional qualification of human resources; the time consumed; the financial costs associated [5] [4] [17] and the difficulty of isolating the effects that can be assigned to training [29]. For example, the complexity in finding the right measures to quantify the impact of training on the turnover, on customer feedback, on cost reduction or sales increasing [8]. In order to obtain a credible analysis, measures should be taken to isolate the effects of the training activities from other influences [23] [22]. The Kirkpatrick / Phillips model is questioned by Parry [20], which states that training ROI estimation is challenging. The author also recommends that companies should focus on considered critical programs, including monitoring of costs that are recognizable, available and easy to calculate. ROI estimation is not free from errors [5], as a result, some human resources professionals believe that it is a waste of time to estimate the ROI of training programs [12].

3 2 Methodology and data 2.1 ROI estimation For each training event associated benefits and costs were calculated, thus allowing for the ROI estimation. Using the Kirkpatrick method we were able to estimate the returns for unitary investments in training, by comparing the benefits and costs for each training program. The use of these criteria varies depending on the training program objectives. Procedures taken ensured the use of consistent, relevant and predictive criteria. The process accounted for performance differences before and after training, and thus making it possible to extract the desired information [19]. In order to overcome the classic limitation of Kirkpatrick's model regarding isolation of effects, we ve addressed prior results (2013); 2014 results, and expected results for 2014, supposing results should evolved at the same rate (%) of trainsxkms (CK) production. Regarding cost calculation we ve considered direct and indirect costs of training events. 2.2 Qualitative comparative analysis We ve applied fsqca technique to uncover causal configurations leading to ROI using fsqca The QCA is based on the mathematical Boolean logic, in which a given variable can assume only a finite number of values. Through this method, it is possible to make organized casework, based on the logical combination of the established conditions, removing the alternatives for objective analysis of selected cases [13]. As a result of using fsqca we can identify alternative configurations leading to the presence and the absence of the outcome (represented by the use of ~ previous to the outcome). Such possibility is an improvement compared to quantitative traditional statistical methods that only provide a single estimated solution to the dependent variable [28]. On the contrary, fsqca accepts that variables can be causally related in one configuration yet, they can be unrelated or even inversely related in others [18]. fsqca accepts alternative combinations of causal conditions, equifinality and asymmetry [10] which, applied to the present case, allows for more than one combination (or configuration) of causal conditions leading to ROI; alternative causal configurations can lead to ROI, and causal conditions of ROI may differ from causal conditions of its absence. Causal conditions in this study are the variables related to each training event: Training topic (top), Training source (sou), Number of participants (part), Duration (dur) and Total cost (cost). The outcome is the Return on Investment (ROI) for each training event. Since ROI is the result of complex relations [12] [2] the use of fsqca is adequate for it presents the ability to untie complex structures [3]. Each configuration of causal conditions and the associated outcome are designated as a case [9]. 3 Data collection and analysis 3.1 Contextualization of the study This study uses data from the Portuguese railway infrastructure manager (REFER E.P.E.) regarding the year At that time, REFER, E.P.E. had an average workforce of 2513 workers, and investments in training over 340,000 euros a year. The company should respect the legal obligation to carry out an annual average of 35 hours of training per employee. In that sense, there has been a sustained growth of the training effort over the last couple of years (68,409.06h in 2012 to 88,412.5h in 2014). The company created an autonomous department responsible for the training and assigned it an individual budget. The department ensures the necessary training to the workers, in support of the company's activities and the organizational change. The key areas to sustainability have been especially addressed: railway engineering and technology, leadership and management. Nevertheless, in 2014 the average training per employee stood at 28.9 hours. Furthermore, during 2014, REFER, E.P.E. was certified as a training organization, Such certification guarantees the company owns the proper internal structure and organization, the means and resources to the development of training activities and training processes, as well as the evaluation mechanisms to preform activity results analysis. Regarding internal training, the company focuses on cost savings of overall training and providing technical train training. Internal training accounted for 67% of total hours in Regarding internal training, the company focuses on leadership and management training, such as project management, advanced program of management and leadership, quality and personal development. The most substantial costs of internal training regard the availability of trainees and trainers. 3.2 Estimated results In order to estimate ROI, it is necessary to identify the company's results. We ve considered EBITDA (earnings before interest, taxes, depreciations and amortizations) as an adequate proxy. EBIDTA is a financial indicator representing how a company generates resources through its operational activities, excluding taxes and other

4 financial contributions; it is able to measure productivity/business efficiency. We ve considered the EBITDA before the training period (2013) and after the training period (2014) in the study, as well as the expected results for 2014 assuming that the company's results should evolve in the same way as trainsxkms (CK) production. We ve computed: - Real CK in 2013: 35,952M of CKs - Real CK in 2014: 36,923M of CKs - % Growth of production: + 2.7% - Real EBITDA in 2013: -23,714M - Expected EBITDA in 2014: -23,074M (considering a similar evolution of + 2.7%) - Real EBITDA in 2014: 1,440M - Difference between Expected and Real EBIDTA in 2014: 24,514M In 2014, real EBITDA value considers some contributions which are worth highlighting: - Extraordinary income from the acquisition of the entire share capital of Lisbon Intermodal Station (GIL), which included both the acquisition of loans to the remaining former shareholders: 26,929m - Reduction of State Compensation: 6,714M - Workforce reduction: 9,151M - Revenue Anticipation in 2013: 1,872M In order to calculate the effect of training, considering all the factors described above, we propose the following equation, which should be based on conservative estimates [2]: M M + 6,714M + 9,151M - 1,872M = 11,581M The estimated effect of training, keeping all other factors constant, would thus be 11,581M, and the total costs were approximately 1,512M. In training effects took place. 3.3 Costs calculation Total costs associated to training involve both direct and indirect costs. Direct costs are the ones associated with trainees (salaries, travel expenses and accommodation) as well as the ones associated to trainers, teaching materials and other costs such as rooms, consumption, projection equipment, etc. We also include the costs of the actions that were conducted by external entities. In some cases, the value considered was calculated based on the cost per participant, while in others cases, we considered the value of the whole class, regardless of the number of participants, (the form of contracting was not the same for all actions). Indirect costs were calculated considering: office supplies (photocopies, pens, paper, etc.); administrative support and coordination of the training manager. 3.4 ROI estimation In order to estimate ROI, the training course s benefits were distributed among the different training programs, we directly associate the benefits to the training programs in proportion to their training hours. For the ROI estimation we followed several steps: a) we ve identified a measure for results (EBIDTA); b) we ve established the value for the results (real and expected); c) we ve quantified the change in performance after the removal of other influences from results; d) we ve established the benefits of the training programs; we ve determined total costs of training; we ve calculated the difference between benefits and training costs, and finally we ve estimated the ROI for each training program following the formula [8] [19]: ROI (Benefits- Costs) Costs For each monetary unit invested in training the ROI value shows the associated return. 3.5 Qualitative Comparative Analysis Calibration QCA was originally developed for the analysis of configurations of crisp sets i.e. conventional Boolean sets. With crisp sets, each case is assigned one of two possible membership scores: 1 (membership in the set) or 0 (nonmembership in the set). When using crisp sets the researcher has got simple data sets composed of binary variables, coded 1 for present and 0 for absent. Some variables are binary, so the calibration procedure is not necessary and only reflect two possible situations (presence or absence of a certain characteristic) and thus are

5 considered to belong or not the given set. These variables comprise the crisp database (crisp set). The variables that take multiple values (various degrees of a characteristic) had to be calibrated: (Number of participants (part), Duration (dur) and Total costs (cost). Calibration is the process of classifying conditions in each case from full membership (1.00) to full non-membership (0.00). The outcome (ROI) was also calibrated. These variables make up a fuzzy database (fuzzy set). Fuzzy membership scores address the varying degree to which different cases belong to a set. Thus, fuzzy sets assess varying degrees of membership between full inclusion and full exclusion. In this sense, a fuzzy set can be seen as a continuous variable that has been purposefully calibrated to indicate degree of membership in a well defined set. Such calibration is possible only through the use of theoretical and substantive knowledge, which is essential to the specification of the three qualitative breakpoints: full membership, full nonmembership, and the point of maximum ambiguity regarding membership [25]. Following Ragin [27], we ve defined the three different anchors that are necessary to calibrate survey data to fuzzy set values establishing the degree of membership of each score; 0.95 for the threshold for full membership; 0.5 for the crossover point of membership ambiguity, and 0.05 for the threshold for full nonmembership. In this study we calibrated both causal conditions and the outcome based on theoretical assumptions and the interpretation of data, and thus attending to particularities of the variable s descriptive statistics. Table 1 shows the cuts used for calibration for causal conditions; for each condition it presents the descriptive statistics and the scores regarding the three different thresholds for calibration. Conditions Descriptive Statistics Calibration cuts (0.95; 0.50; 0.05) Training topic (top) Binary variable: min = 0; max = Training source (sou) Binary variable: min = 0; max = Number of participants (part) μ = 22.9; σ = 56.4; min = 1; max = 584 (100; 8; 4) Duration (dur) μ = 13.9; σ = 12.3; min = 0.3; max = 80 (35; 16; 3.5) Total costs (cost) μ = 4,624.9; σ = 10,801.1; min = ; max = 143,663.6 (20.000; 2.000; 400) Return on Investment (ROI) μ = 17.8; σ = 12.4; min = -0.8; max = 42.1 (36; 17; 2) μ = average, σ = standard deviation, min = minimum, max = maximum Tab. 1 Descriptive statistics (n=327) and calibrations of causal conditions and outcome Necessity and sufficiency analysis Causal conditions are assessed for necessity and sufficiency. The causal condition s degree of necessity indicates the extent to which it is needed to achieve the outcome. Conversely, the causal condition s degree of sufficiency shows the extent to which it can be related to the explanation of the outcome [11]. The sufficient condition sets are also designated as configurations (of several causal conditions leading to the outcome variable). Necessary conditions should present a consistency score exceeding the threshold of 0.80 [24]. When addressing the necessary conditions, regarding the presence of the outcome (ROI) the necessary conditions are: ~part and ~cost. Regarding the absence of the outcome (~ROI) there are no necessary conditions. Table 2 reports the necessary conditions for both the outcome (ROI) and its absence (~ROI). Results show that by using fsqca the conditions may be needed to achieve the outcome but not its absence. Outcomes Conditions ROI ~ROI Consistency Coverage Consistency Coverage top ~top sou ~sou part ~part dur ~dur cost ~cost top = Training topic; sou = Training source: part = Number of participants; dur = Duration; cost = Total costs; grey text highlight identify values for necessary conditions identification. Tab. 2 Necessary conditions Summary

6 When addressing the sufficient condition sets, and regarding the presence of the outcome, results show three parsimonious causal combination and five intermediate ones (for debate among complex, parsimonious and intermediate solutions see Fiss [10] and Ragin [27]). All the configurations, as well as the parsimonious and the intermediate solutions, regarding the presence of the outcome present consistency levels above 0.84 complying with the threshold of 0.80 suggested by Ragin [27], Crilly [7] or Fiss [10]. Overall solution coverage value is 0.76 which is within the suggested limits of 0.25 to 0.90 [27] [30]. When addressing the sufficient condition sets, and regarding the absence of the outcome, results show two parsimonious causal combinations and three intermediate ones. All the configurations, as well as the parsimonious and the intermediate solutions, regarding the presence of the outcome, present consistency levels above 0.92 complying with the threshold of 0.80 suggested by Ragin [27], Crilly [7] or Fiss [10]. Overall solution coverage value is 0.72 which is within the suggested limits of 0.25 to 0.90 [27] [30]. Consistency reflects the extent to which the cases sharing a given combination of conditions agree in displaying the outcome in question [27]. Coverage reflects how much of the variation in the outcome is accounted for by a causal condition or combination [26] similar to the R2 regarding linear regressions [11]. Specifically, unique coverage shows the relative importance of each particular configuration [10] Causal configurations Causal configurations in Tables 3 and 4 present the core and peripheral conditions for both the outcome variable (ROI) as well as its absence (~ROI), following the best practices. Core conditions are the ones included in both the parsimonious and intermediate solutions, while peripheral conditions are only part of the intermediate solution [11] [24] [27]. The most parsimonious solution contains only core conditions highly linked to the outcome. The intermediate solutions are more conservative assuming the most plausible simplifying assumptions [27]. Since fsqca admits asymmetry, causal conditions for ROI differ from the ones for its absence [10]. Causal conditions Coverage Consistency Configurations top sou part dur cost Raw Unique Overall Solution coverage: 0.76 Overall Solution consistency: 0.87 top = Training topic; sou = Training source: part = Number of participants; dur = Duration; cost = Total costs; full black circles ( ) indicate the presence of a condition, and center white circles ( ) indicate its absence. Large circles indicate core conditions; small ones, peripheral conditions. Blank spaces indicate don t care. Tab. 3 Causal configurations for Return on Investment (ROI) Configurations Causal conditions Coverage Consistency top sou part dur cost Raw Unique Overall Solution coverage: 0.72 Overall Solution consistency: 0.95 top = Training topic; sou = Training source: part = Number of participants; dur = Duration; cost = Total costs; full black circles ( ) indicate the presence of a condition, and center white circles ( ) indicate its absence. Large circles indicate core conditions; small ones, peripheral conditions. Blank spaces indicate don t care. Tab. 4 Causal configurations for the absence of Return on Investment (~ROI) Such results confirm that variables can be causally related in one configuration yet, they can be unrelated or even inversely related in others. The findings reflect the described assumptions of fsqca: a) More than one configuration of causal conditions leading to both the outcome and its absence (alternative combinations of causal conditions), b) Alternative causal configurations can produce the same outcome (equifinality), c) Causal conditions of the outcome may differ from causal conditions of its absence (asymmetry).

7 4 Discussion and analysis of results Among all training events, the highest ROI estimates (values above 30) regard: the topography and welding programs (business skills) and business management programs (support skills to the business). It is worth noticing the existence of 59 training events with only 1 participant. This is mainly due to participation in congresses, seminars or very specific training. Regarding these cases, only 4 present ROI values below average. These training events are among the best performing, perhaps due to the subsequent dissemination of knowledge by teams where these participants are integrated after their return to the organization. 4.1 Analysis of the necessary and core conditions (ROI and ~ROI) There are two necessary conditions leading to ROI (we can interpret as the necessary conditions to reach highest ROI results): ~part (we can interpret as a small number of participants) and ~cost (we can interpret as a low training event cost). Regarding the absence of ROI (we can interpret as the necessary conditions to reach less successful ROI values) there are no necessary conditions. The core conditions are those included in the parsimonious and intermediate solutions and therefore are most important. Peripheral conditions are only included in the intermediate solutions, and therefore are not as important. Regarding the configurations to achieve better ROI, the core conditions are: type and ~type; orig and ~orig; ~part; dur and ~cost; there are no peripheral conditions. Results suggest these conditions are important. Regarding such conditions: ~part is present in all combinations; ~cost is present in 4 of 5 causal configurations and dur in 3 out of the 5 combinations. These results indicate that having a small number of participants per training event is one of the conditions to achieve the best ROI values. Similarly, not having high total costs and having long training events seems to be important in order to reach the best ROI values. Regarding the configurations to achieve less successful ROI values there are some conditions worth paying attention to. There are three core conditions: part, ~dur and cost and a single peripheral condition: ~orig. The core conditions: part, ~dur and cost are present in 2 out of the 3 configurations. These results seem to indicate that having a large number of participants per training event, short lasting training events and high total costs per training event seem to be important conditions preventing to reach the best ROI results. 4.2 Analysis of causal configurations (ROI and ~ROI) The fsqca produces several causal configurations leading to the same outcome and thus it allows identifying different paths towards ROI and ~ROI. The use of fsqca revealed several configurations leading to ROI and ~ROI that could not have been uncovered otherwise. The intermediate solution configurations to achieve better ROI results present 5 configurations with 4 conditions each: - (~top, ~sou, ~part, ~cost) - (top, sou, ~part, ~cost) - (top, sou, ~part,, dur) - (~sou, ~part, dur, ~cost) - (top, ~part, dur, ~cost) The intermediate solution configurations for the absence of better ROI results present 3 settings with 2 or 3 conditions each: - (part, ~dur) - (part, cost) - (~sou, ~dur, cost) There are five paths to success (ROI) and three paths to the non-success (~ROI), which is good news for managers in charge of training. There are more trails leading to success than the ones leading to less successful outcomes! To achieve the best returns we have a variety of paths; if it is not possible to follow a certain path, there are several other alternatives, though more stringent (having more conditions to be met, either by the presence or absence of a condition). To achieve not so successful ROI values there are only three possible trails, however, the non-success pathways are more difficult to avoid, since two of them only involve two conditions to be met. Causal configurations leading to ROI are more numerous and involve more core conditions, indicating greater constraints when comparing to configurations leading to the absence of ROI. Regarding ~ROI, configurations present less core conditions, indicating the existence of fewer constraints and thus reflecting easiness to comply with, which is tricky, one may consider they represent traps. Pathways to ~ROI are fewer but they are less demanding than trails to ROI; such results should alert managers.

8 5 Conclusions It's commonly accepted that training ROI is very difficult to determine accurately. Still, there is a broad consensus on the effectiveness of its estimation and the produced knowledge: this generates debate in the organization and causes better resource allocation. This paper offers a proposal for training ROI estimation at REFER (current IP), using a comprehensive data base from This calculation is an absolute novelty in the company since the returns of training had never been estimated. Results show programs presenting estimated ROI values above company average: business skills, external actions and actions whose total cost is less than the average cost. We ve estimated the ROI of over 300 training events. Nevertheless, the major contribution of this study does not regard the strict estimated values, but the possible relative comparisons between them and the consequent management decisions, in terms of resource allocation. The ROI estimation process is not free from criticisms, the individual ROI values depend on criteria used for benefits valuation. In contrast, the comparisons of several ROI values that were estimated using the same criteria is not questionable. This study provides the ROI values of hundreds of training events allowing for comparative analysis. Using fsqca we identified necessary and sufficient causal conditions to ROI and ~ROI. We also deliver alternative configurations (combinations of causal conditions) leading to ROI and ~ROI. There are more configurations leading to ROI results than the ones leading to ~ROI. There are more ways to achieve better returns on training than to reach its absence. This is a good sign for managers: they have five available pathways to ROI and just three leading to its absence. But these paths are more demanding: they demand to meet four causal conditions. On the contrary, the paths leading to ~ROI appear to be less demanding because they just include two or three causal conditions for this to happen. Such comparison comprises implications for management. Pathways to success seem to be more demanding, while the ways to the absence of such success require the compliance with fewer conditions. This is a warning sign for the organization: it may be more difficult to meet four causal conditions for success than meeting two or three for his absence. Findings reveal that the necessary conditions to achieve better returns (ROI) are the absence of many participants and the absence of very high costs. It thus appears that these two conditions carry some advice to managers. Although there are several sufficient conditions, there is no necessary condition for the absence of ROI. Not existing a necessary condition for less successful results may be tricky, in the sense that there is no particular condition to avoid in special. The road to the fulfilment of the research objectives has not been free from obstacles. Some limitations apply to this study: some training related data is missing, like event s trainer gender; prevailing gender among trainees for each event, trainees evaluation scores or trainer s evaluation scores. Having more complete data could allow for additional analysis and discussion. We believe that addressing physical and digital dimensions of training events separately is a challenge worth pursuing. References [1] Arthur Jr, Bennett Jr, Edens, P. e Bell, S. (2003). Effectiveness of training in organizations: A meta-analysis of design and evaluation features. Journal of Applied Psychology, 88, [2] Bartel, A. P. (2000). Measuring the Employer's Return on Investments in Training: Evidence from the Literature. Industrial Relations 39(3), [3] Basedau, M. e Richter, T. (2014). Why Do Some Oil Exporters Experience Civil War but Others Do not? Investigating the Conditional Effects of Oil. European Political Science Review, 6(4), [4] Brotherton, P. (2011). Organizations lag behind in measuring learning s value. Training & Development, [5] Brown, B. (2001), Return on Investment in Training, Myths and Realities, Vol. 16, available at: (accessed November 3rd 2016). [6] Ceribeli, H. B., Ceribeli, A. C. e Goés, T. A. (2015). Implantando e Avaliando Programas de Treinamento: Um Estudo de Caso. Tekhne e Logos, Botucatu 6(2), [7] Crilly, D. (2011). Predicting stakeholder orientation in the multinational enterprise: A mid-range theory. Journal of International Business Studies, 42, [8] Curado, C. e Teixeira, S. M. (2014). Training evaluation levels and ROI: the case of a small logistics company. European Journal of Training and Development, 38(9), [9] Fiss, P. C. (2007). A set-theoretic approach to organizational configurations. Academy of Management Review, 32(4), [10] Fiss, P. C. (2011). Building better causal theories: A fuzzy set approach to typologies in organization research. Academy of Management Journal, 54, [11] Fiss, P. C., Sharapov, D. e Conqvist, L. (2013). Opposites Attract? Opportunities and Challenges for Integrating Large-N QCA and Econometric Analysis. Political Research Quarterly, 66(1), [12] Flynn (1998), The Nuts & Bolts of Valuing, Training Workforce, 77 (11), [13] Freitas, V., Neto, F. (2014). Qualitative Comparative Analysis (QCA): usos e aplicações do método. Revista Política Hoje, 24,

9 [14] Grossman, R. e Salas, E. (2011). The transfer of training: what really matters. International Journal of Training and Development, 15(2), [15] Instituto para a Qualidade na Formação (2006). Guia para a avaliação da formação, 1 Ed., Instituto para a Qualidade na Formação, I. P. [16] Kaplan, R. S. e Norton, D. P. (1996). Using the Balanced Scorecard as a Strategic Management System, Harvard Business Review, 74(1), [17] Kirkpatrick, D. e Kirkpatrick, J. (2005). Transferring learning to behavior: using the four levels to improve performance. Koehler Publishers. [18] Meyer, A. D., Tsui, A. S. e Hinings, C. R. (1993). Configurational approaches to organizational analysis, Academy of Management Journal, 36(6), [19] Noe, R. A. (2010). Employee Training and Development (5 ed.), NY: McGraw-Hill. [20] Parry, S. B. (1996). Measuring training's ROI. Training & Development, 50(5), [21] Phillips, J. (1997). Handbook of Training Evaluation and Measurement Methods. 3 ed., Gulf Publishing Company. [22] Phillips, J. e Phillips, P. (2013). Measuring ROI in Learning and Development: Case Studies from Global Organizations, ASTD Press. ROI Institute. [23] Phillips, J. e Phillips, P. (2009). Using ROI to demonstrate performance value in the public sector. Performance Improvement, 48(4), [24] Ragin, C. C. (2000). Fuzzy-Set Social Science, Chicago: Chicago University Press. [25] Ragin, C. C. (2005). From Fuzzy Sets to Crisp Truth Tables. Tucson, Arizona: Department of Sociology, University of Arizona. [26] Ragin, C. C. (2006b). User s guide to fuzzy-set/qualitative comparative analysis 2.0. Tucson, Arizona: Department of Sociology, University of Arizona. [27] Ragin, C. C. (2008). Redesigning Social Inquiry: Fuzzy Sets and Beyond, Chicago: University of Chicago Press. [28] Rihoux, B. e Ragin, C. C. (2009). Configurational comparative methods: Qualitative comparative analysis (QCA) and related Research, Los Angeles: Sage. [29] Williams, R. C. e Nafukho, F. M. (2015). Technical training evaluation revisited: an exploratory, mixed-methods study, Performance Improvement Quarterly, 28(1), [30] Woodside, A. and M. Zhang. (2013). Cultural diversity and marketing transactions: Are market integration, large community size, and world religions necessary for fairness in ephemeral exchanges? Psychology & Marketing, 30(3),