ANNEX II: Description and evaluation of results

Transcription

1 Author Population (n, function, sector, country) ANNEX II: escription and evaluation of results esign Intervention Outcomes Measures Findings Limitations Level (Abdullah & Suring, 2010) 120, various, public, Malaysia Survey Transfer factors: Motivation to transfer; Training design; Transfer climate; Transfer of All of the transfer factors have a positive and significant relationship (r ) - ross sectional, self-report, small sample, questionable validity of measures - Relevant information on the variables is lacking. (Amagoh, Literature review effectiveness - should be comprehensive and systematically integrated into the organizational, leadership should not be treated as a one-shot event. - Non-systematic review, expert - No evidence is provided on the effectiveness of the leadership initiatives discussed. X (Avolio et al, 13/28, various, profit/military, various Metaanalysis, including experiments and quasiexperiments Training or intervention; Affective (e.g. follower liking, satisfaction, enjoyment), behavioral (e.g. leader emergence, participation) and cognitive outcomes Training or interventions have a moderate effect on affective (d. 39), behavioral (d.43) and cognitive (d. 62) outcomes. onclusions are not drawn on interventions in particular, but on experimental leadership interventions in general. A (lume et al, 2010) 89 studies, students/manag erial/nonmanagerial/ mainly USA and anada Metaanalysis including field and lab studies Trainee characteristics, environmental factors, pre- or post interventions, learning outcomes, reactions and moderators Transfer of : generalization (over settings, people, situations) and maintenance (persistence over time) Moderate effect sizes for cognitive ability (r.37), conscientiousness (r.28), motivation (r.29), supportive work environment (r.21), transfer climate (r.27), post knowledge (r.24), post self efficacy (r.22), pre optimistic preview (r.20) Moderators: overall more transfer for open (e.g. interpersonal) vs. closed skills oncrete and elaborate information on the variables is lacking. A/ 1

2 (iucur & Pirvut, 2012) 30, managerial, automotive industry, Romania Quasiexperiment Transactional Analysis based Training Program Emotional stability, social boldness & warmth attell s 16PF inventory Significant differences for emotional stability and social boldness - Small sample - Personality is used as an outcome to evaluate the effectiveness of (ummings et al., 2008) Nurses in leadership positions Systematic review (9 studies, designs?) leader participation in educational activities. (mostly 3-4 day programs) Effectiveness of nursing leadership All nine studies found positive results for (mostly) self-rated leadership and observed leadership. - ontrol group? - Randomization? - Self-report - No effect sizes provided (ay, 2000) See abstract - Highly non-systematic review - Expert - Virtually no evidence is provided on the effectiveness of the leadership practices discussed (erue et al, 2012) 173, MA students, various, USA ohort study, 9 months Structured reflection through after-event reviews Leader ehavior escription Qu. (LQ) Small positive effect on leadership -MA students -Same source bias -Low incremental validity (uygulu & Kublay, 2011) 30, nurses, health sector, Turkey efore after study, repeated measures Transformational leadership program practices: model the way, inspire a shared vision, enabling others to act, encourage the hart Practices Inventory- Self and Observer Significant effect on leadership practices - Small sample size - No control group - No randomization - No effect-sizes reported - Rated by self (and observers) (Eden et al., 2000) Meta analysis of 7 RT s Managerial in Pygmalion leadership style Leader and follower perceptions, performance data measures Small overall effect. - 4 of 7 studies survey-data only A (Kelloway & arling, 2010) Literature review, exp interventions Individual well-being and safety in organizations There is an impact of leadership on individual well-being and safety Limited evidence is provided on the effectiveness 2

3 (Krejci & Malin, 1997) 87, nurses, health care, USA eforeafter study competencies (12) competency instrument -Significant difference in the perception of the understanding and the ability of the leadership competencies - Self report - Halo effect - No effect sizes reported (Leslie et al., 2005) 56, pediatricians, health care, USA eforeafter study program Effectiveness of leadership program -etailed needs assessment -post-survey - Participants were confident in many of their leadership qualities but desired increased, particularly in areas of time and priority management and leading from the middle. -Participants positively evaluated the program and improved in selfreported basic competencies; 87% also reported fully or partially achieving a leadership-related goal - Self-report - Small sample (Malling, et al, 56, consultants responsible for postgraduate medical education, health sector ontrolled before- after study course (3 modules in 7 days over 6 months) evelopment of leadership skills Multi source feedback - No differences in multi-source feedback scores at one year follow up compared to baseline measurements, either in the intervention or in the control group (p = 0.149). - course following a MSF procedure compared to MSF alone does not improve leadership skills. - Prone to selection bias - High dropout in control group - Pooling the scores from the respondents into one score - Limited follow up (1 year) (Markuns, et al, 2010) Expert Individual interviews and focus groups with local medical directors A five-step call to action (see abstract) - Expert X 3

4 (Morahan et al., 1998), physician executives, hospitals, USA Multiple case study - Internal leadership institutes have advantages over external programs 1-10 recommendations are given (see abstract) - Expert : limited evidence is provided on the effectiveness (Morrow, Jarrett, & Rupinski, 1997) 9/25, various managerial levels, pharmaceutical sector, USA Meta analysis of 18 controlled and uncontrolled studies Managerial & sales/technical Effectiveness and utility (economic) of programs -Multi-attribute utility analysis model -Multi source feedback evaluation - Great variation of effectiveness of the programs (range of d-.09 to 1.11) - Managerial d.31, utility (ROI) 45%, which less than d.64 and ROI 418% for sales/technical - Use of surveys; rater bias? - Small sample sizes - One organization (Powell & Yalcin, 2010) 85 samples, n = 4.779, entry level and middle management Metaanalysis of various designs Managerial intervention Kirkpatrick levels of transfer: 1. Learning 2. ehavior 3. Results - The overall effect of managerial interventions is small:.25 - Effects are weakest for objective measures and behavioral or result outcomes. They also vary depending on the research design. - The effects did not improve over time (50 years) -Limited information on the measures or the characteristics of the interventions. - onfidence interval includes zero. - No information on whether the participants were randomly assigned in these studies. A/ (Solansky, 2010) 303, administrators & leadership mentors, education, USA Posttest with survey 360-leadership skills assessment and leadership mentoring (% of time spent with the mentee, personal contact mentor mentee) -Self report vs observer report - Information shared by the mentee with the group & mentor - Practices Inventory scale (LPI) - ifferences between self report and observer report - oaching time (β.18) and number of contacts (β.02) are both significantly, positively and moderately related to mentees willingness to share information with mentors and the mentee group regarding their leadership - Observers were chosen by the participants - Relevant? 4

5 (Straus et al, 2013) samples, physicians health sector, anada Systematic review (3 controlled pre/post, 4 pre/post, 3 cross sectional) programs at academic medical centers Physicians knowledge, skills, attitudes and behavioral outcomes - programs have modest effects on outcomes important to University hospitals - programs affected participants advancement in academic rank (48% vs 21%, P =.005) and hospital leadership position (30% vs 9%, P =.008) - Participants were more successful in publishing papers (3.5 per year versus 2.1 per year, P <.001) - ontains weak study designs A/ (Taylor et al, 2005) sectors, employees and countries Meta analysis of 117 studies, randomized and nonrandomized ehavior modeling eclarative knowledge, procedural knowledge skills, -related attitudes, job behavior, workgroup productivity and workgroup climate Effects were largest for learning outcomes, smaller for job behavior and results outcomes: (declarative knowledge δ1.20; procedural knowledge δ1.18; attitudes δ0.33; job behavior δ0.27; workgroup productivity δ0.13; workgroup climate δ0.11) Although effects on declarative knowledge decayed over time, effects on skills and job behavior remained stable or even increased. Skill was greatest when learning points were used and presented as rule codes and when time was longest. Transfer was greatest when - mixed (negative and positive) models were presented - practice included trainee-generated scenarios - trainees were instructed to set goals - when trainees superiors were also trained - when rewards and sanctions were instituted in trainees work environments - ontains 0 in some of the confidence intervals. A 5

6 (Stoller, 2013), managers and physician executives at leveland linic Expert Training programs in health care - areer enhancement and organizational satisfaction ollaboration and synergy Innovation See abstract - Expert X (Storey, 2004) samples, various sectors, SME s, various countries studies without control group or pre test Formal in small firms Effect on small firms performance Hard outcome measures and objective data -Mixed evidence for the effect of formal - Weak study designs (Taylor et al, samples, managers, various countries Meta analysis of 107 studies, randomized and nonrandomized Training programs ifferences in transfer effects (ratings of trainees on the job behavior after ) Measurement s by alternative rating sources (self, peer, subordinate, supervisor) -study & design - Transfer effects based on trainees selfratings (δ0.64) and ratings from superiors (δ0.53) were largest across studies. In contrast, transfer effects based on peer ratings (δ0.26) and particularly subordinate ratings (δ0.13) were substantially smaller and more homogeneous. - Transfer of was greatest when raters were likely to have known whether the manager being rated had attended, when content was derived from an analysis of tasks and skill requirements (instead of theory or literature), and when included opportunities for practice. - Surprisingly, the topic with the greatest transfer perceived by subordinates was not for focused on improving managers interpersonal skills with subordinates, a topic for which subordinates would be in a uniquely strong position to observe - ontains 0 in some of the confidence intervals. - Sometimes conclusions are drawn on a rather small amount of studies (and small total n) 6

7 changed management job behavior, but for general management skills, which typically includes a wide variety of skills, some of which subordinates may not have the opportunity to observe. Subordinaterated transfer was also greater for goal-setting and performance appraisal skills programs than for interpersonal skills. - It appears that all rating sources perceive both short managerial programs to result in quite small transfer effects and long programs to result in modest transfer. (Yeung et al, 2012) 40, life support providers, health care sector, UK Post test: video recordings of a standardized cardiac arrest simulation Team leadership skills Factors affecting team-leadership skills - ehavior escription Questionnaire ardiac Arrest Simulation test score - There is an association between team leadership skills and cardiac arrest simulation test score, pre-shock pause, and hands off ratio. - skills were not significantly associated with more simple technical skills such as chest-compression rate, depth, and ventilation rate. - Prior in team leader skills was independently associated with better leadership behavior - eveloping leadership skills should be considered an integral part of resuscitation. - Small sample - Awareness of videotaped behavior 7