RASCH ANALYSIS: PSYCHOMETRIC PROPERTIES OF MALAYSIAN SECONDARY SCHOOL STUDENTS LEADERSHIP SCALE

Transcription

1 Man In India, 96 (1-2) : 1-8 Serials Publications RASCH ANALYSIS: PSYCHOMETRIC PROPERTIES OF MALAYSIAN SECONDARY SCHOOL STUDENTS LEADERSHIP SCALE Mei-Teng Ling 1* and Vincent Pang 1 The purpose of this study is to produce empirical evidence of psychometric properties of a survey questionnaire, the Malaysian Secondary School Students Leadership Scale (M3SLS) Version 2. The scale was analysed by using Rasch Model. The questionnaire was administered on 2340 students from 26 government schools in four main divisions of Sabah, Malaysia. The results showed that secondary dimension is noticeable and the constructs were suggested to be analysed separately in future. PTMEA Corr are all in positive values. Person statistics (separation>2; reliability >0.8) and item statistics (separation>3; reliability >0.8) were within the acceptable range. All items in the M3SLS were in the range of 0.5 to 1.50, except item V5 (2.01), item V3 (1.64), item V4 (1.63), item V1 (1.63) and item V7 (1.53). All the items were maintained after the test. Keywords: Validity, Reliability, Leadership Competency, Rasch Measurement Introduction Instruments such as tests and questionnaires are usually used to measure students performances and perceptions in schools. There are a number of instruments used to assess leadership competencies. Examples of scales that have been developed are Penn State Leadership Competency Inventory (Yoon et al., 2010), Adaptive Leadership Competency Profile (Sherron, 2000), Prospector Instrument (Spreitzer et al., 1997), A 360-Degree Peer Evaluation of Leadership Competencies Scales (Rogers, 2001), and Multifactor Leadership Questionnaire (MLQ) (Bass and Avolio, 1994). However, most of these scales were developed in the western countries. Leadership qualities required in the Malaysian context may be different. Furthermore, extra efforts are needed to translate an instrument into target language if a researcher uses an instrument which is of different context and the culture. As suggested by Tuleja et al. (2011), a good questionnaire constructed in single culture may not necessarily travel well across cultures. In Malaysia, most of the researchers such as Mohd Yusri Ibrahim and Aziz Amin (2014), Bity Salwana Alias et al. (2010), Yahya Bin Don (2009), Azita (2007) and Fareeza (2010) were more interested in assessing the leadership competency of principals, assistant principals, coaches and teachers rather than students in school. However, secondary school students also possess leadership potential (Fertman and Linden, 1999; Hine, 2011) and should be concerned so that educators can help students in developing 1 Faculty of Psychology and Education, Universiti Malaysia Sabah,88400 Kota Kinabalu, Sabah, Malaysia, * lingmeiteng@gmail.com

2 2 MAN IN INDIA leadership skills and building character at the earlier stage. The instrument used in this study is the second version of questionnaire built to assess the level of leadership competency of secondary school students based on their personality, values and leadership skills. The Rasch Model has been used in a variety of evaluation settings and provides detailed information on several quality aspects of a test such as item fit, unidimensionality, and reliability of the items and persons. Besides, it provides valid measures for tests with missing data (Sköld, Hermansson, Krumlinde-Sundholm, & Eliasson, 2011). The purpose of this study was to produce empirical evidence of psychometric properties of the Malaysian Secondary School Students Leadership Scale (M3SLS) by using Rasch Measurement Model. Instrument and Method M3SLS is in a form of questionnaire which contains 68 items of five-point rating scale that represent three main constructs, namely personality, values and leadership skills (Tubbs and Schulz, 2006). M3SLS was used to gauge the three constructs which comprises of personality- P (15 items), values V (18 items), and Leadership skills S (35 items). Item are quantitatively analysed using WINSTEPS to assess the suitability of items. Samples After the instrument was pilot tested, it was distributed to a larger group of respondents to ensure the stability of the scale. The instrument was administered to 2340 students from 26 government secondary schools in four main divisions of Sabah. 121 questionnaires were dropped due to incomplete and invalid responses, reducing the number of cases to They consisted of 877 (40.2%) male and 1306 (59.8%) females. Form 2 respondents showed the highest proportion (34.3%) while Form 1 students showed the lowest (32.7%). The location of the schools were classified as rural or urban area. The proportion of the respondents from the urban areas is 8% more than that from rural areas. Results and Discussions Unidimensionality In PCAR analysis, the raw variance explained by measures was 22.3% while the unexplained variance in first contrast was 5.4%. The raw variance explained by measures of 22.3% closely matched the expected target of 22.2% (Table 1). Although this meets the minimum requirement of 20% for acceptable calibration by Reckase (1979), the more stringent threshold of 40%, as suggested by Conrad et al., (2011) was not achieved. The reason for not achieving the minimum 40% as recommended by Linacre (2003) is the disruption of items as noise. The noise

3 RASCH ANALYSIS: PSYCHOMETRIC PROPERTIES... 3 level was 6.2%, considered as acceptable because it was far from maximum of 15% as recommended by Linacre (2003) and Fisher (2007). There were five items showing standard residual correlation values of more than 0.50 (Table 2). This indicates that respondents interpreted the items related to the same thing. The items showed infit mean squares of more than 1.5, therefore investigation on the two items was done based on the item fitness. Besides, the test was probably unidimensional when a secondary dimension have the strength of less than 3 items (Linacre, 2003). However, the Eigen value of the unexplained variance in first contrast was 5.4, which means that five items fall in secondary dimension in the instrument. Therefore, each constructs was suggested to analyse separately in the next study. TABLE 1: TABLE OF STANDARDIZE RESIDUAL VARIANCE OF M3SLS V.2 Empirical Modeled Total raw variance in observations = % 100.0% Raw variance explained by measures = % 22.2% Unexplained variance in 1st contrast % 8.0% TABLE 2: NOISE ITEMS IN M3SLS V.2 Contrast Loading Measure Infit MNSQ Outfit MNSQ Item V V V V V7 Item polarity The result shows the values of PTMEA Corr in MSLCS version 2 are positive. The positive values of PTMEA Corr prove that the items are carefully constructed (Bond and Fox, 2001). The maximum PTMEA Corr is 0.47 for item 19, Item 21 and Item 29 in Leadership Skills Construct (Table 3). A high PTMEA Corr indicates that an item is able to distinguish between the ability of respondents (Linacre, 2003). There is no negative PTMEA Corr found in this study, therefore, it can be concluded that the items will contribute to the measurement of leadership competency scale. A negative value or zero indicates that the link for the item response or respondents is in conflict with the variable or constructs (Linacre, 2003). Furthermore, positive value of PTMEA Corr shows that the items are able to discriminate or differentiate the level of leadership competencies held by the respondents.

4 4 MAN IN INDIA TABLE 3: ANALYSIS OF PTMEA CORR. IN M3SLS VERSION 2 Items PTMEA Items PTMEA Items PTMEA Items PTMEA Corr. Corr. Corr. Corr. CP V LS LS CP V LS LS CP V LS LS CP V LS LS CP V LS LS CP V LS LS CP V LS LS CP V LS LS CP V LS LS CP V LS LS CP V LS LS CP V LS LS CP V LS LS CP V LS LS CP V LS LS V V LS LS V LS LS LS Person and Item reliability and separations The person separation index of M3SLS is 2.90 (Table 4). The person separation index indicates that the number of strata which measured the leadership competency of individuals can be divided into three. The person reliability of M3SLS is The person reliability is more than 0.8, which is an acceptable and strong value. The item separation index of M3SLS is (Table 5), which is good and acceptable, based on Linacre (2003) that the separation index of more than 2 is good. High item separation (> 3, item reliability > 0.9) implied that the person sample is enough to confirm the item difficulty hierarchy, which is the construct validity of the instrument (Linacre and Wright, 2012). The higher the number of separations, the more confidence the researcher can place in the replicability of the item placement across other samples (Bond and Fox, 2007). Item reliability is 1.00, which is acceptable and considered strong because they are more than 0.8. The high item reliability might be due to the wide difficulty range of items and a large sample size. When the value is high, it indicates that the sample size is enough for stable comparisons between items (Linacre and Wright, 2012). Table 5 shows the summary of item statistics of LCS produced by Winsteps. The maximum level of item measurement was logit (SE: 0.38) while the maximum measure of person is logit (SE: 0.44). The range of the item scale that can be exploited were only from to There was a small gap showing respondents that were item-free, with a width of 1.46, from to logit. There were items that were person-free, which shows the item measure at logit compared with respondents who were at least competent at

5 RASCH ANALYSIS: PSYCHOMETRIC PROPERTIES... 5 TABLE 4: SUMMARY OF PERSON STATISTICS FOR M3SLS VERSION 2 Raw Count Measure Model Infit Outfit score error MNSQ ZSTD MNSQ ZSTD Mean S.D MAX MIN REAL RMSE.14 ADJ.SD.42 SEPARATION 2.90 PERSON RELIABILITY.89 MODEL RMSE.13 ADJ.SD.42 SEPARATION 3.18 PERSON RELIABILITY.91 S.E. OF PERSON MEAN =.01 PERSON RAW SCORE-TO-MEASURE CORRELATION =.99 CRONBACH ALPHA (KR-20) PERSON RAW SCORE RELIABILITY =.91 TABLE 5: SUMMARY OF ITEM STATISTICS FOR M3SLS VERSION 2 Raw Count Measure Model Infit Outfit score error MNSQ ZSTD MNSQ ZSTD Mean S.D MAX MIN REAL RMSE.02 ADJ.SD.38 SEPARATION PERSON RELIABILITY 1.00 MODEL RMSE.02 ADJ.SD.38 SEPARATION PERSON RELIABILITY 1.00 S.E. OF PERSON MEAN =.05 UMEAN=.000 USCALE=1.000 ITEM RAW SCORE-TO-MEASURE CORRELATION = DATA POINTS. LOG-LIKELIHOOD CHI-SQUARE: with d.f. p=.0000 This situation shows that several items, within 0.29 logit are the standard personality, values and skills that were practiced by the respondents in performing leadership tasks in school. This instrument is found to have low standard error of measurement (SE), ± 0.02 and quite high person Infit MNSQ SD at 0:46 logit. Both the Infit MNSQ and ZSTD values were close to the ideal values of 1 and 0. Fit statistics All items in the M3SLS were in the range , except item V5 (2.01), item V3 (1.64), item V4 (1.63), item V1 (1.63) and item V7 (1.53). Except for item 20 (V5- Friendship), the item fit map generated by Quest showed all items were fit. A cross-plot between person with 63 items and 68 items was carried out to investigate the effects of the deleted items on the result of the study (Linacre, 2010). None of the observations were out of the two curves and most of the observations clustered around the Plot-Empirical line (Figure 1). Less than 5% (5/100 x 2183 = 109) of the observations dropped outside of the straight line (perfect line). This concludes that the person measures obtained with 63 items were substantially similar to those obtained with 68 items. Therefore, the items were not deleted.

6 6 MAN IN INDIA Figure 1: Invariance of Person Measures of 63 and 68 items in M3SLS Conclusion With the examination of the psychometric properties of the M3SLS instrument it was found that the validity and reliability of the instrument are acceptable to measure leadership competency of secondary school students. It has also fulfilled other psychometric properties required of an acceptable instrument. These imply that M3SLS is a truly leadership competency instrument that can be used to measure the level of leadership among secondary school students in Malaysia. The measurement of leadership level will enable educators to plan and execute various leadership programs so that students of various levels of leadership can be developed as effective leaders. References Amrin, H. A. (2007). Penilaian Pelajar Terhadap Kompetensi Pemimpin Program Pendidikan Luar di Institutsi Pendidikan Guru Malaysia (Students evaluation on the instructors competencies in the teaching and learning process of the Outdoor Education Program at the Malaysian Teacher Education Institutes.). (Unpublished Doctoral Dissertation). Universiti Putra Malaysia, Serdang, Malaysia. Bass, B. M., & Avolio, B. J. (1994). Improving Organizational Effectiveness Through Transformational Leadership. Thousand Oaks, CA: Sage Publications Inc. Bity Salwana Alias, Ahmad Basri Md. Yussof, Ramlee Mustapha, & Mohammed Sani Ibrahim. (2010). Analisis Kompetensi Pengetua Berdasarkan Kualiti Peribadi, Pengetahuan, Kemahiran dan Amalan dalam Bidang Pengurusan Sekolah Menengah Malaysia. Jurnal Pendidikan Malaysia, 35(2), Bond, T. G., & Fox, C. M. (2007). Applying the Rasch Model: Fundamental Measurement in the HUman Sciences (2nd ed.). Mahwah, New Jersey: Routledge. Conrad, K. J., Conrad, K. M., Dennis, M. L., Riley, B. B., & Funk, R. (2011). Validation of the Crime and Violence Scale (CVS) to the Rasch Measurement Model GAIN Methods Report 1.2. Evaluation review. Chicago, IL.

7 RASCH ANALYSIS: PSYCHOMETRIC PROPERTIES... 7 Fareeza, A. R. (2010). Assistant Pronsipals Leadership Capacity Building Through The Malaysia Education Quality Standards Instrument. (Unpublished Doctoral Dissertation). Universiti Malaya, Kuala Lumpur, Malaysia. Fertman, C. I., & Linden, J. A. van. (1999). Character education for developing youth leadership. National Association of Secondary School Principals Bulletin, 83(605), Fisher, J. W. P. (2007). Rating Scale Instrument Quality Criteria. Rasch Measurement Transactions, 21, Hine, G. (2011). Exploring the Development of Student Leadership Potential Within a Catholic School: A Qualitative Case Study. (Doctoral Dissertation, University of Notre Dame, Australia). Retrieved from viewcontent.cgi?article=1062&context=theses Jackson, T. R., Draugalis, J. R., Slack, M. K., Zachry, W. M., & Agostino, J. D. (2002). Validation of Authentic Performance Assessment/ : A Process Suited for Rasch Modeling. American Journal of Pharmaceutical Education, 66, Linacre, J. M. (2003). Dimensionality: contrasts & variances. Retrieved January 20, 2015, from Linacre, J. M. (2010). When to stop removing items and persons in Rasch misfit analysis? Rasch Measurement Transactions, 23(4), Linacre, J. M., & Wright, B. D. (2012). A user s guide to WINSTEPS ministeps Rasch model computer programs. Chicago: Mesa Press. Liu, X. F. (2010). Using and Developing Measurement Instruments in Science Education: A Rasch Modeling Approach (HC). Charlotte, North Carolina: Information Age Publishing, Inc. Mohd Yusri Ibrahim, & Aziz Amin. (2014). Model Kepemimpinan Pengajaran Pengetua Dan Kompetensi Pengajaran Guru. Jurnal Kurikulum & Pengajaran Asia Pasifik, 2(1), Nardi, P. M. (2003). Doing Survey Researc/ : A guide on quantitative method. United States of America: Pearson Education, Inc. Reckase, M. D. (1979). Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of Educational and Behavioral Statistics, 4(3), Rogers, A. (2001). A 360-Degree Peer Evaluation of Leadership Competencies. (Doctorate Dissertation). Available from ProQuest Dissertations & Theses Global. (UMI No ). Rust, J., & Golombok, S. (2009). Modern Psychometrics: The Science of Psychological Assessment (3rd ed.). United States of America: Routledge. Sherron, C. T. (2000). Psychometric Development of The Adaptive Leadership Competency Profile. (Doctorate Dissertation). Available from ProQuest Dissertations & Theses Global. (UMI No ). Siti Rahayah Ariffin, Bishanani Omar, Anita Isa, & Sharida, S. (2010). Validity and reliability multiple intelligent item using rasch measurement model. Procedia - Social and Behavioral Sciences, 9, doi: /j.sbspro Sköld, A., Hermansson, L. N., Krumlinde-Sundholm, L., & Eliasson, A.-C. (2011). Development and evidence of validity for the Children s Hand-use Experience Questionnaire (CHEQ). Developmental Medicine and Child Neurology, 53(5), doi: /j x

8 8 MAN IN INDIA Spreitzer, gretchen M., Kizilos, M. A., & Nason, S. W. (1997). A dimensional analysis of the relationship between psychological, empowerment and effectiveness, satisfaction and strain. Journal of Management, 23(5), Tubbs, S., & Schulz, E. (2006). Exploring a taxonomy of global leadership competencies and meta-competencies. Journal of American Academy of Business, 8(2), Tuleja, E. a., Beamer, L., Shum, C., & Chan, E. K. Y. (2011). Designing and Developing Questionnaires for Translation Tutorial. IEEE Transactions on Professional Communication, 54(4), doi: /tpc Yahya Bin Don. (2009). Kolerasi dan Pengaruh Kompetensi Emosi terhadap Kepimpinan Sekolah: Perbandingan antara Sekolah Berkesan dengan Sekolah Kurang Berkesan (Correlation and Effect of Emotional Competence of School Leadership: Comparison between the Effective School and Less Effective School). (Unpublished Doctoral Dissertation). Universiti Malaya, Kuala Lumpur, Malaysia. Yoon, H. J., Song, J. I. H., Donahue, W. E., & Woodley, K. K. (2010). Leadership competency inventory/ : a systematic process of developing and validating a leadership competency scale. Journal of Leadership Studies, 4(3), doi: /jls