Annual Cycle of Effectiveness Assessment Handbook

Transcription

1 Annual Cycle of Effectiveness Assessment Handbook LSC IE (ACE) Handbook

2 Table of Contents Preface..1 Introduction.2 What is IE?...2 A. Definition B. Myths and Facts about Assessment Why do We Assess?.3 A. Benefits of Assessment B. SACS Expectations C. Creating a Culture of Continuous Improvement Annual Cycle of Effectiveness (ACE) Assessment Process at Lone Star College. 4 ACE Assessment Timeline Writing the ACE Assessment Plan...10 Vision.10 Mission..11 Unit Objectives..11 Action Steps..12 Methods of Assessment Achievement Targets...14 Writing the Progress Report...16 Writing the Closing-the-Loop Report...18 Final Results Interpretation of Results..19 Recommendations..20 Continuous Quality Improvement.21 Implementation of the Recommendations..21 LSC IE (ACE) Handbook

3 Assessment Tool: Compliance Assist 22 Web Link to Compliance Assist..22 ACE Form: Guide to Data Entry.22 Compliance Assist Navigation Tool...22 Glossary of Terms...23 Resources...25 Appendices 27 Appendix A: LSC Strategic Plan ( ).27 Appendix B: Worksheet for SWOT Analysis 28 Appendix C: Worksheet for Developing a Vision Statement.29 Appendix D: Worksheet for Developing a Mission Statement 30 Appendix E: Worksheet for Creating an ACE Plan..31 Appendix F: Worksheet for Creating an ACE Progress Report and Closing-the-Loop Report.32 Appendix G: ACE Self-Assessment Rubric. 33 Appendix H: Samples of Exemplary ACE Assessment Plans and Reports LSC IE (ACE) Handbook

4 Preface There is a story set in medieval times that tells of a traveler who comes upon three stonemasons. He asks each in turn: What are you doing? The first answers, without hesitation, I am cutting this stone. The second, who appears to be doing the identical job, makes a gesture with his hand and says, I am completing the wall. The third, who again seems to be doing the same job, slowly raises his eyes to the sky and says, I am building a cathedral. This story highlights the power of vision, of seeing situations as opportunities for greatness. The Annual Cycle of Effectiveness (ACE) process at Lone Star College (LSC) provides the platform for LSC leaders to build the synergy and work in unison to advance our shared vision for students and our community. To support the ACE process, we have weaved resources together in this ACE Handbook. The purpose of the handbook is threefold: First, the handbook provides a common understanding and set of expectations regarding ACE process at Lone Star College. Second, it serves as a resource to aid the campuses in their assessment efforts. Finally, it acts as a communication piece for our internal and external stakeholders, updating them about our assessment process and accomplishments. As such, it is a living document that will be updated on a regular basis. Additionally, the Office of Strategic Planning and Assessment provides online resources and regular professional development workshops. We want to support you in any way we can as we build the cathedral of excellence together. LSC IE (ACE) Handbook 1

5 Introduction What is IE? A. Definition IE, an acronym for Institutional Effectiveness, refers to "the systematic, explicit, and documented process of measuring performance against mission in all aspects of an institution (SACS/COC Resource Manual, 2012, p. 115). It is a continuous process of planning, assessing, reviewing results, and using results for ongoing improvement. At Lone Star College (LSC), the IE assessment consists of the assessment of both operational effectiveness and learning effectiveness. To assess the operational effectiveness, all LSC units are engaged in the Annual Cycle of Effectiveness (ACE), which consists of developing the assessment plans, implementing the plans, and collecting data to gauge the extent to which the expected outcomes are achieved, and then using the assessment results for continuous improvement. To assess learning effectiveness, all Lone Star College s educational programs measure student learning outcomes at both the course level and the program level. A detailed explanation of the learning outcome assessment practice at Lone Star College is included in a separate SLO Handbook and posted on the SLO Website. To further clarify what IE is, let s debunk some myths about assessment. B. Myths and Facts about Assessment Myths Assessment is something required by the external agencies; it is disconnected with what we do daily. Assessment is not worth the time. Facts Assessment is an integral part of what we do daily. We carry out action steps, implement initiatives, provide services, and tackle projects to support our institution s mission. Without assessment, we won t know what we do daily is effective or not. We wouldn t know how we can improve what we do daily. If we want to achieve the highest level of excellence, assessment is the means to that end. Clearly, assessment of a program should not usurp the actual services; however, assessment offers insight into what is working and what is not, and in a time of limited resources, assessment can yield information that helps you save time and resources. LSC IE (ACE) Handbook 2

6 Assessment will go away. Assessment data will be used as evidence against us. SACS/COC mandates the use of assessment data to continuously improve institutional effectiveness. It is part of the institutional practices, so it is not optional and not going away. It is up to us to make assessment meaningful and useful so that we can use assessment data to improve our services to our students and our community. The focus of assessment is on improving our units and our institution so that we can serve our students and our community better. It is not used to negatively impact any individual faculty or staff member s job. Why do We Assess? A. The Benefits of Assessment Simply put, we assess ourselves in an effort to become better. The benefits of assessment are manifold: Assessment helps to improve programs and services through results that indicate areas for change. It informs students, faculty, staff and other stakeholders of the state of programs and services and their impact on students and the community at large. It validates that a program or service is accomplishing what it was planned to accomplish through a demonstration of assessment results. It supports campus decision making processes, strategic planning, program review and additional accountability activities such as SACS reaffirmation and reaccreditation of academic programs by professional accrediting bodies (adapted from Selim et al., 2005). In sum, assessment promotes a better workplace for employees, greater value to our students, and a more productive organization for our community. B. SACS Expectations Not to be forgotten is that assessment and a well-documented IE process is mandated by our accrediting body, the Southern Association of Colleges and Schools (SACS) under core requirement 2.5 and Comprehensive Standard See more detailed description of these presented below: Core Requirement 2.5 The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that (1) incorporate a systematic review of the institutional mission, goals, and outcomes; (2) results in continuing improvement in institutional quality; and (3) demonstrates the institution is effectively accomplishing its mission. (Institutional Effectiveness). (SACS/COC Resource Manual, 2012, p. 16) LSC IE (ACE) Handbook 3

7 Comprehensive Standard The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: (Institutional Effectiveness) educational programs, to include student learning outcomes administrative support services educational support services research within its educational mission, if appropriate community/public service within its educational mission, if appropriate. (SACS/COC Resource Manual, 2012, (p. 48) C. Creating a Culture of Continuous Improvement Obviously, Lone Star College must follow the mandates of its accrediting body; however, SACS is not the sole reason we undertake this endeavor. Rather, Lone Star College pushes forward with its IE process because it can act as a cornerstone to a culture of continuous improvement. In order to create and sustain a culture of continuous improvement, Lone Star College engages in both informal and formal assessment. Informal assessment is constant and ongoing (embedded in everything we do). Formal assessment is a systematic process and occurs on a regularly scheduled basis. LSC has numerous formal assessment processes, for example, the Annual Cycle of Effectiveness (ACE) process, the Program Review Process, and the Student Learning Outcome assessment process. This Handbook will limit its discussion to the Annual Cycle of Effectiveness (ACE). Annual Cycle of Effectiveness (ACE) Process at Lone Star College In his seminal work, Good to Great, Jim Collins describes the process of qualitative change and improvement as time consuming and arduous especially, in the beginning: No matter how dramatic the end result, the good-to-great transformation never happened in one fell swoop. There was no single defining action, no grand program, no one killer innovation, no solitary lucky break, no miracle moment. Rather, the process resembled relentlessly pushing a giant heavy flywheel in one direction, turn upon turn, building momentum until a point of breakthrough and beyond. Collins comments highlight three important observations: Becoming great does not happen overnight. There is no silver bullet. It is an arduous task especially, at first. LSC IE (ACE) Handbook 4

8 The flywheel analogy also parallels the ACE process of improvement. This process has been described in slightly different formats (e.g., see Deming s Plan-Do-Study-Act model of improvement.) However, the Planning, Implementing, and Evaluating model reduces the model down to its most elemental phases, making it as easy as a PIE to remember. Planning Evaluating Implementing LSC IE (ACE) Handbook 5

9 The Planning Phase Each Lone Star College (LSC) unit develops an Annual Cycle of Effectiveness (ACE) assessment plan to support the Lone Star College s Strategic Plan, which is developed every five years. The planning model we use is a two-way flow model It is both bottom-up and topdown. See the pyramid charts below that illustrate this model: ACE LSC Strategic Plan Set Strategic Objectives Set Strategic Goals Update Vision, Mission, and Values External & Internal Environmental Scanning: Research & SWOT Analysis Bottom-Up Model: LSC Strategic Planning Process LSC IE (ACE) Handbook 6

10 Top-Down Model: LSC Annual Cycle of Effectiveness Planning Process LSCS Strategic Plan Chancellor, VCs, Presidents Annual Cycle of Effectiveness Planning Vice Presidents, AVCs ACE Deans, Executive Directors ACE Departments, Programs, Offices ACE The bottom-up and top-down approach used at LSC is an important part of our unit effectiveness planning process. It ensures that every LSC unit keeps in mind the important outcomes for the College and the people it serves, yet allows each unit to employ its own particular professional expertise in planning its work in support of the College as a whole while engaging in continuous quality improvement in its own area. For more information about how to develop an Annual Cycle of Effectiveness assessment plan, please refer to pp of this Handbook. LSC IE (ACE) Handbook 7

11 The Implementation Phase Planning alone will not solve any problem, what is needed is action implementing the plan. The implementation phase involves carrying out the action steps laid out in the ACE assessment plan. It also involves collecting formative assessment data (outputs and feedback) to measure if the action steps are implemented as desired, and if what is implemented helps the unit move toward its expected outcomes. A mid-year Progress Report is required to help the units review their progress, outputs, and feedback so that they can adjust their action steps and it enhances strategies to reach their expected outcomes and fulfil their objectives. For more information about how to produce the Progress Reports, please see pp of this Handbook. The Evaluation Phase The evaluation phase includes collecting summative data, engaging faculty or staff members in data analysis and data interpretation to make sense of the data. Then the unit should make a collective decision on what recommendations to make what further action should be taken to improve the actual outcomes for the coming year. A Closing-the-Loop Report should be produced to capture the actual outcomes, data interpretation, and recommendations. For more information about how to produce the Closing-the-Loop Report, please see pp of this Handbook. Annual Cycle of Effectiveness Assessment Timeline September 1-August 31 Implementing the ACE Assessment Plans Jan. 15-Feb. 15 Reporting Progress June 1-August 31 Closing the Loop for Current Year June 1-August 31 Planning for Next Year LSC IE (ACE) Handbook 8

12 Please note that during the June 1-August 31 timeframe, ACE leaders are engaging in twolayers of assessment activities: closing the loop for the current year and planning for the subsequent assessment cycle. For more detailed information about the ACE assessment calendar, please refer to the following table: Unit Effectiveness Assessment Calendar Timeframe Assessment Activities Description Person Accountable June 1-15 Presidents/VCs June July 1-15 July 15-July 30 ACE: Planning *CA: Strategic Plan alignment, unit objective, action steps, method of assessment, achievement targets and resources needed textboxes *Recommended: Team meetings to review data and identify strengths and weaknesses for each unit VPs/AVCs Deans/Ex. Dir. Department/Program/Office Unit Leaders/Dir. August 1-August 30 ACE Plan reviewed and approved by supervisors ACE Leaders review for content ACE Leaders by hierarchy September 1- Subsequent May 30 ACE: Implementation ACE: Implementation ACE Leaders January 15-February 15 ACE: Mid-Year Progress Report CA: Mid-Year Progress textbox and status dropdown Recommended: Team meetings to discuss status of projects and team needs. ACE Leaders May 15-June 1 June 1- July 30 August 1-August 30 ACE: Evaluation (Data production and publication by AIR) ACE: Evaluation (Closingthe- Loop Reporting) ACE Closing-the-Loop reports reviewed by supervisors AIR analysts to run data analyses specific to ACE assessment *Close the Loop *CA: Textboxes Final Results and Data Interpretation/ Recommendations, Status dropdown - updated, and identify which Objective should be made public *Recommended: Team Retreat to discuss and celebrate successes for the year. UEP Leaders review for content AIR Analysts ACE Leaders ACE Leaders LSC IE (ACE) Handbook 9

13 Writing the ACE Assessment Plan An ACE assessment plan consists of the following components: Vision Mission Unit Objectives Action Steps Methods of Assessment Achievement Targets Resources Needed Assessment Schedule This section explains how to compose the major components. How to Create a Vision Statement Vision statement refers to an organization s description of its desired future state in terms of performance excellence (Miller, 2007, p. 247). A vision statement addresses the following questions: What would you like your unit to look like in the future? What would you like your unit to strive for (what level of excellence)? The frequently used phrases include the best in, a leader in, or..nationally recognized for. The following are examples of vision statements: Example 1: Midlands Technical College will be a leading community college, recognized for excellence in education and student success. The college will be a primary choice of citizens preparing for the job market, pursuing career advancement opportunities, and seeking to transfer to senior colleges and universities. Partnerships with area schools, business groups, and governmental agencies will contribute to an educational system that enhances the economic development and quality of life in the region. Example 2: Tarrant County College will be a nationally recognized community college of choice for academic excellence and lifelong learning. LSC IE (ACE) Handbook 10

14 How to Create a Mission Statement Mission statement refers to an organization s statement of purpose, clarifying whom the organization serves, how it serves them, and what results it seeks as a consequence of that service (Miller, 2007, p. 244). A mission statement addresses the following questions: What are the primary functions and activities that your unit performs? Why do you perform these activities (What's the purpose of your unit)? For whom does the unit conduct the activities? See examples of mission statements below: Example 1: Midlands Technical College is a comprehensive, public, two-year community college serving the Central Midlands of South Carolina. College programs and services provide accessible, affordable, high-quality post-secondary education that prepares students to enter the job market, allows them to transfer to senior colleges and universities, and assist them in achieving their professional and personal goals. Through its programs and services, the college equitably provides higher education opportunities and supports the economic growth of the community. Example 2: Tarrant County College provides affordable and open access to quality teaching and learning. How to Create Unit Objectives An objective is a measurable action to be implemented in a short to medium timeframe, designed to support the accomplishment of your goal. While goals are directional and general, objectives are actionable and specific. In other words, they should be SMART (Specific, Measurable, Attainable, Realistic, and Timely). Examples of unit objectives in support the goals are listed below: Example 1: In support of the goal, Promote student completion and student success, we can have the following unit objectives: Increase the Developmental students course sequence completion rates in Math, Reading, and Writing; Increase Part-time faculty s course completion and success rates; Increase Fall to Fall persistence rate for black male students. LSC IE (ACE) Handbook 11

15 Example 2: In support of the goal, Promote a college-going culture, we can have the following objectives: Increase the FTIC students enrollment Increase the dual-credit enrollment How to Lay out Action Steps Action steps are strategies, tasks, or events you design to accomplish your objectives. They bring the objectives to the operational level. Completion of action steps can produce outputs that can be used to assess if we are moving closer to the completion of objectives. See examples of actions steps below: Example 1: The objective, Implement the Service Learning program, can be broken down into the following action steps: Step 1 Promote the Service Learning Program to different academic departments Step 2 -- Assist faculty to develop service learning curriculum Step 3 Assess student learning outcomes and success outcomes for the Service Learning courses Example 2: The objective, Increase the dual-credit enrollment, can be broken down into the following action steps: Step 1 Collaborate with the high school principals to develop Memo of Understanding Step 2 Provide professional development for the dual-credit teachers once every semester How to Design Methods of Assessment Assessment methods refer to what type of data (measures) to collect, how to collect and analyze them, and who will be responsible for collecting them. Areas of Organizational Performance The Big Picture To assess organizational performance, Barbara A. Miller (2007) identifies seven areas of organizational performance in higher education: 1. Effectiveness: the extent to which a unit achieves its intended outcomes 2. Productivity: ratio of inputs consumed vs. outputs created 3. Quality: of partnership, work process, work life, etc. 4. Efficiency: cost and benefit, resource utilization 5. Customer Satisfaction: level of satisfaction 6. Innovation: creative changes implemented to improve the unit s performance 7. Financial durability: financial health and well-being (pp ) LSC IE (ACE) Handbook 12

16 Measurements: In each area, measures can be identified to gauge the performance. The most commonly used measures are academic direct measures, academic indirect measures, and administrative measures. See explanations and examples for these types of measures illustrated below: Academic Direct Measures: Refer to the type of data that can be used to directly measure students knowledge and skills. Examples: Pre-test, post-test, comprehensive subject matter exam, licensure exam, portfolio, thesis, writing assignment, internship evaluation, capstone assignment, and performance (recital, art exhibit, or science project) Academic Indirect Measures: Refer to the type of data that can be used to infer student achievement. Examples: Surveys, Interviews, Focus Group Studies, Document Analyses, or Audit Measures (course grades, retention/graduation/job placement rates) Administrative Measures: Refer to the type of data that can be used to gauge the effectiveness of an organization or a work unit in terms of the quantity, quality, productivity, efficiency measures. Examples: Activity volume, compliance with government or professional standards, work-place environment, customer satisfaction, etc. Measures can also be categorized into output or outcome measures: Outputs: Deliverables from an organization s key work processes experienced by customers, or products, services, and information designed to meet customers needs Examples: # of participants in the 1 st year student orientation, # of students referred to Early Alert intervention, # of accelerated Dev. Ed. Program designed and implemented Outcomes: End results an organization seeks to achieve as a consequence of customers receiving or experiencing the organization s outputs; or changes in knowledge, skills, or behavior due to services or intervention programs provided Examples: Accelerated Dev. Ed. Programs (output) Less time spent completing the Dev. Ed. Program (outcome) Auto-award of credentials (output) More students who met graduation criteria graduate (Outcome) Academic coaching program (output) -- Participants take initiatives to seek assistance from academic advisors, etc. (outcome) Quantitative Method vs. Qualitative Method There are two major methods of assessment: Quantitative Method and Qualitative Method. See differences between the two methods illustrated with examples below: LSC IE (ACE) Handbook 13

17 Quantitative Method Collect data that are numbers Use data to test hypothesis Help us understand WHAT is the level of success Examples: Test score comparison Analysis of survey with numeric ratings # of participants comparison Qualitative Method Collect data that are words, narratives Use data to explore possibilities, identify problems Help us understand the WHY behind the numbers Examples: Interviews Focus Group Studies Document Analyses A good assessment method design should incorporate both output and outcome measures, utilizing both quantitative and qualitative methods. How to Set Achievement Targets To determine if any actual outcome is considered success, a reference point is needed to provide a benchmark against which the actual outcome is compared. Achievement targets serve as such kind of reference points. Therefore, an achievement target should be set by researching the trend of the data and by looking at the benchmark data. See examples presented below: Example 1: Unit Objective: Implement the Service Learning Program. Achievement Target: The course completion rate of students in Service Learning courses will be 3% higher than that of students in similar courses with the traditional approach for Fall 2013 and Spring Target Benchmark Example 2: Unit Objective: Increase the dual-credit enrollment. Achievement Target: Increase the dual-credit enrollment by 5% for Fall 2013 as compared to Fall Benchmark Target LSC IE (ACE) Handbook 14

18 Also, keep in mind that the assessment measures must match your unit s achievement targets See examples of misaligned and aligned measures presented below: Example 1: Achievement Target: At least two outreach events will be hosted each year for Texas junior and senior high school STEM students. Misaligned Measure: Survey student participants and faculty volunteers from junior and high schools Aligned Measure: Number of outreach events hosted each year Example 2: Achievement Target: Increase in first attempt success on the Accuplacer Assessment Test Misaligned Measure: Collect and establish baseline data to measure effectiveness of process Aligned Measure: 1 st attempt Accuplacer pass rate LSC IE (ACE) Handbook 15

19 Writing the Progress Report During the mid-year, between January 15 and February 15, each unit is required to examine the progress of implementing the action steps, conduct formative assessment to check if interim targets are being met, and then submit the Progress Report. This section shows how to produce the Progress Report. Follow the directions below to produce the Mid-year Progress Report: Review your action steps; Report on which action steps have been implemented, which are in the process of being implemented, and which are delayed or become infeasible to implement; If the action steps are not implemented as planned, report on how you would address the issue and adjust your plan. See examples of Progress Report below: Example 1: Unit Objective: Implement the Service Learning program Action Steps: (1) Promote the Service Learning Program to different academic departments (2) Assist faculty to develop service learning curriculum (3) Assess student learning outcomes and success outcomes for the Service Learning courses. Progress Report: (1) Promoted the Service Learning Program to different academic departments (English Department offered 5 sections of SL course) (2) Assisted 20 faculty to develop service learning curriculum (English and History Department Completed the curriculum design) (3) Developed student learning outcomes and success outcomes for the Service Learning courses. We will assess the SLOs and measure success outcomes for Fall LSC IE (ACE) Handbook 16

20 Example 2: Unit Objective: Increase FTIC students' Fall to Fall persistence rate Action Steps: (1) Work with Student Success Division to offer intrusive advising; (2) Create a student club in each division. Progress Report: (1) One-to-one advising was offered to 65% of the FTIC students (2) 100% of the divisions created a student club, but the participation in the club activities was low (15%). Example 3: We will encourage more students to participate in the club activities. Unit Objective: Increase accessibility of police assistance phones. Action Steps: (1) Request budget to purchase the Police Assistance Phones (2) Purchase and install the phones (3) Promote the awareness of the phones Progress Report: (1) Phone purchase budget was approved (2) 100% of the phones were purchased and installed. (3) Promoted the awareness of the phones via web news (1,200 hits). Example 4: We will increase the number of hits by sending out s to encourage reading of the news on the web. Unit Objective: Help employees become knowledgeable of system resources. Action Steps: (1) Revamp new employee orientation (2) Collect feedback (3) Create a new employee process for adjunct instructors Progress Report: (1) Revamped new employee orientation (2) Collected feedback from faculty and staff (3) Created a new employee process for adjunct instructors and conducted 5 trainings with 100 attendees. We will continue providing more training. LSC IE (ACE) Handbook 17

21 Writing the Closing-the-Loop Report At the end of each assessment cycle, all units need to submit the Closing-the-Loop Reports, which consist of the following components: Final Results Interpretation of Results Recommendations How to Present the Final Results The results are actual outcomes that need to be compared with the Achievement Targets. The results should be data collected based on the specified measures. When reporting on results, present the actual data and draw conclusion on the data objectively. See the following examples: Example 1: Achievement Target: At least 95% of responses to the faculty and staff satisfaction survey on campus environment will indicate satisfaction with the campus cleanliness and functionality. Final Results: Present the actual data: According to the 2012 Satisfaction survey, 91% of faculty and staff members were satisfied with the cleanliness and functionality (58% very satisfied and 33% satisfied). Draw conclusion (Was the target met? What did the data show?): The 91% satisfaction rate was very high; however, since the target was 95%, the actual satisfaction rate was 4% lower than the target. Example 2: Achievement Target: The course pass rate ("C" or above) will increase by 5% for the whole Division of Developmental Studies for Fall 2011 as compared to Fall 2010 and for Spring 2013 as compared to Spring Final Results: Present the actual data: For Developmental Studies Division, the Fall 2011 pass rate was 67%, which was 1% higher than the Fall 2010 pass rate (66%). The Division's Spring 2013 pass rate was 61%, which was the same as Spring 2012 pass rate -- 61% as well. Draw conclusion (Was the target met? What did the data show?): The Fall pass rate comparison showed a small increase from Fall to Fall. However, the Spring pass rate comparison showed no increase from Spring to Spring. The target was not met. LSC IE (ACE) Handbook 18

22 Example 3: Achievement Target: Increase the number of external grant awards by 5% from to Final Results: Present the actual data: In , we got 21 grant awards, and in , we got 24 grant awards. Draw conclusion (Was the target met? What did the data show?): The actual % of increase was 14%, which was 9% higher than the expected outcome of 5%. The target was exceeded. How to Interpret Data Once you have gathered data and compared the actual outcome with the expected outcomes, you will need to engage your team in the discussion of data to make sense of the data so that you can take further action to improve the actual outcome. Making sense of the data interpreting data requires you to address the question, What factors might have contributed to the actual outcome? See examples of data interpretation below: Example 1: Results: According to the 2012 Satisfaction survey, 91% of faculty and staff members were satisfied with the cleanliness and functionality (58% very satisfied and 33% satisfied). The 91% satisfaction rate was very high; however, since the target was 95%, the actual satisfaction rate was 4% lower than the target. Data Interpretation: The high satisfaction rating (91%) is a result of timely attention to client needs classrooms, work-spaces, etc. The goal of 95% was not met possibly due to the learning curve of new staff and the time it takes for all the staff to learn about the standards and procedures of providing quality services. A lack of communication between clients and staff may also be the possible cause for not meeting the targeted satisfaction rating. Example 2: Results: For Developmental Studies Division, the Fall 2011 pass rate was 67%, which was 1% higher than the Fall 2010 pass rate (66%). The Division's Spring 2013 pass rate was 61%, which was the same as Spring 2012 pass rate --61% as well. The Fall pass rate comparison showed a little bit increase from Fall to Fall. However, the Spring pass rate comparison showed no increase from Spring to Spring. The target was not met. LSC IE (ACE) Handbook 19

23 Data Interpretation: The lower pass rate for Spring semesters may be caused by a number of factors, including the possible interruption of the Spring Break and the Semester Break, less dual enrollment sections in the Spring, and longer span of the semester. Example 3: Results: In , we got 21 grant awards, and in , we got 24 grant awards. The actual % of increase was 14%, which was 9% higher than the expected outcome of 5%. The target was exceeded. Data Interpretation: We were able to exceed our target due to a number of factors: (1) We were proactive in establishing relationship with the funders; (2) we planned the timeline well, and (3) we had competent and dedicated grant writing staff. How to Make Recommendations Once we identified issues or factors that might have contributed to the actual outcomes, we are in a position to make recommendations for further action. Please note that even if we have met or exceeded the anticipated targets, we still need to make recommendations for continuous improvement. See the following examples of recommendations made based on the interpretation of data: Data Interpretation Recommendation The high satisfaction rating (91%) is a result of timely attention to client needs classrooms, work-spaces, etc. The goal of 95% was not met possibly due to the learning curve of new staff and the time it takes for all the staff to learn about the standards and procedures of providing quality services. A lack of communication between clients and staff may also be the possible cause for not meeting the targeted satisfaction rating. The lower pass rate for Spring semesters may be caused by a number of factors, including the possible interruption of the Spring Break and the Semester Break, less dual enrollment sections in the Spring, and longer span of the semester. We were able to exceed our target due to a number of factors: (1) We were proactive in establishing relationship with the funders; (2) we planned the timeline well, and (3) we had competent and dedicated grant writing staff. Facilities Operations & Maintenance Department will provide more frequent training for the staff. Clients will be encouraged to communicate their expectations and desires for classroom settings. Future building projects will anticipate the need for classrooms that can be re-arranged quickly and easily. The Developmental departments will 1) provide review lessons right after the breaks to improve course pass rate and 2) strengthen retention efforts by working with the Student Success Specialists to promote course persistence rate. 3) Research also needs to be conducted to find out what causes the lower pass rates in Spring semesters. We will connect with more funders and also strengthen retention efforts to retain our staff members. LSC IE (ACE) Handbook 20

24 Continuous Quality Improvement After we have made recommendations, we need to implement these recommendations. Based on the implementation results, we can make further recommendations. Thus, it is a continuous cycle of using data for continuous quality improvement. One important way to implement recommendations is to carry the recommendations to the subsequent assessment cycle by turning the recommendations into the unit objectives. See examples below on how to turn recommendations into the subsequent cycle s unit objectives: Units Community Leadership Institute Advising & Outreach College Relations Recommendations from the Cycle We were able to exceed the target of getting 50 applicants for the Leadership Institute because of the implementation of new marketing initiatives, including the face-toface meetings, increased efforts of community partners, and printed marketing materials. We will continue face-to-face meetings with the board and develop new ways to reach a broader audience, such as targeting specific industries and organizations through target marketing. We have exceeded the target of providing workshops to students about utilizing online services; however, the workshops need to be diversified, so we will continue to explore other types of workshops to expand online access. Text messaging services can no longer be utilized due to utilization of Lone Star Alert. We will request additional funds to secure marketing across social media platforms. Unit Objectives for the Cycle Based on the Previous Cycle s Recommendations Unit Objective Complete the design, development and integration of common marketing collateral and website for CDI's national marketing campaign. Unit Objective Continue to increase and diversify the campus workshops to educate students about online services and how to utilize the services. Unit Objective Launch an enhanced social media campaign to increase student-college communication and participation in on-campus events with the expected outcome of 10% increase in student interaction on Facebook. LSC IE (ACE) Handbook 21

25 Assessment Tool: Compliance Assist Lone Star College (LSC) uses Compliance Assist as the assessment tool. All ACE, SLO, and PLO assessment plans and reports are stored in this tool. Utilizing this tool, LSC maintains consistent framework for assessment across the multiple colleges. All ACE units use the same ACE form with the same planning and reporting components. The web link to the Compliance Assist, ACE Form Guide, and the Compliance Assist navigation tool are listed below: Web Link to Compliance Assist ACE Form Guide to Data Entry - Internal Users Only Compliance Assist Navigation Tool - Internal Users Only Worksheets for creating the ACE assessment plans and reports are listed in the Appendices section as Appendix E and F on p. 31 and p. 32. LSC IE (ACE) Handbook 22

26 Glossary of Terms Action An event or action taken to accomplish an outcome (i.e.: summer camp, workshop, and marketing). Activities Things you do-activities you plan to conduct in your program. Assessment Schedule The frequency with which assessment tools are used to measure progress toward an outcome (i.e.: Monthly, Semester, Annually). Assessment Plan A plan with clearly defined strategies or actions, outcomes, assessment tools, assessment schedules, and performance targets; outcomes must be aligned with the System and Campus objectives. Assessment Instrument The specific method selected to measure progress toward achievement of an outcome (CCSEE, Pre/Post Perceptions Survey, specific rubrics, specific portfolio assignments, etc.). Assessment Results The results of the outcomes measures (i.e.: Was the performance target met? How will you use the results in the Campus Planning process [Academic, Administrative, Budget, Facilities]?). Continuous Quality Improvement (CQI) A management philosophy that addresses the progress of an institution systemically and systematically to improve the quality of its processes and outcomes. CQI stresses a strong culture of assessment-based evidence for decision making in a cooperative, cross-functional way throughout the institution. Department A department reports to a Dean within a Division unit (i.e.: CIT, Advising, Financial Aid). Division Second largest operational unit at the College (i.e.: Division 1, Division 2, and Student Services). Formative Evaluation Any evaluation that takes place before or during a project s implementation with the aim of improving the project s design and performance. Formative evaluation complements summative evaluation and is essential for trying to understand why a project or a program works or doesn t. Information or data gathered from the formative evaluation can be used to provide timely intervention so as to improve the effectiveness of a project or a program. Goal A broad statement that describes what is to be achieved to fulfill the program mission. Institutional Effectiveness refers to "the systematic, explicit, and documented process of measuring performance against mission in all aspects of an institution (SACS/COC Resource Manual, 2012, p. 115). It is a continuous process of planning, assessing, and reviewing results, aimed at ongoing improvement. LSC IE (ACE) Handbook 23

27 Measure A clearly definable performance level that can be measured. Mission A statement that describes who is served by the program, the philosophy of service, and what will be achieved; should be two to four sentences. Needs Assessment A systematic study of institutional problems that interfere with achieving goals and that require solutions. Objective Statements of aims or desired ends whose achievement can be assessed to demonstrate progress toward or achievement of their goals. Outcome End results an organization seeks to achieve as a consequence of customers receiving or experiencing the organization s outputs, or changes in knowledge, skills, or behavior due to services or intervention programs provided. Output - Deliverables from an organization s key work processes experienced by customers, or products, services, and information designed to meet customers needs Outcome Result - What happened? Was the achievement target accomplished? Personnel Individuals currently employed in the program; program leadership should be identified. Person Responsible One person who will be charged with facilitating the assessment cycle for the outcome listed. Process Assessment Production of information concerning how close actual processes are to desired processes. Program/Committee A program; the smallest operational unit within the college responsible for conducting periodic program reviews (i.e.: CISCO, Honors, and Scholarships). Program Evaluation Determination of the adequacy of the program in fulfilling its mission, goals, and objectives. Qualitative Method Qualitative method of assessment collects data that are narrative. Data are analyzed by looking for recurring themes. Examples of this method include interviews, observations, focus group study, etc. Quantitative Method Quantitative method of assessment collects data that can be summarized into meaningful numbers and can be analyzed statistically. Examples include test score comparison, analysis of survey ratings, and number of events comparison. Related Goal/Objective Lone Star College System Goal/Objective to which a program outcome must be aligned. LSC IE (ACE) Handbook 24

28 Strategic Planning Long-term, often three-five year, planning at the level of the whole institution or unit that focuses specifically on adaptation to the organization s external environment and the future. Guided by a vision of the organization in the future, strategic planning attempts to position the organization favorably with respect to needed resources. Strategy An event or action taken to accomplish an outcome (i.e.: summer camp, workshop, and marketing). Summative Evaluation Judgment at the end of an activity of its effectiveness according to standards. SWOT Analysis An activity that engages the stakeholders in brainstorming on the Strengths and Weaknesses within the organization, as well as the external factors that pose as Opportunities and Threats to the organization. Information gathered from this activity can be used for strategic planning or intervention design, etc. Unit Largest operational unit at the College (i.e.: Student Learning, Student Success, and Administrative Services). Resources Internal Websites: LSC Office of Strategic Planning and Assessment (SPA) Website LSC Institutional Effectiveness (IE) Website LSC Analytics and Institutional Reporting (AIR) Website LSC Dashboard and Scorecards -- internal users only External Websites: Richland College Assessment Website American Association of Community Colleges (AACC) American Evaluation Association Association for Institutional Research (AIR) Community College Research Center Council for the Advancement of Standards in Higher Education Integrated Postsecondary Education System (IPEDS) National Center for Education Statistics (NCES) LSC IE (ACE) Handbook 25

29 SACS/COC Society for College and University Planning (SCUP) Books: Alfred, R., Shults, C., and Seybert, J. (2007). Core indicators of effectiveness for community colleges (3 rd ed.). Lanham, MD: Rowman & Littlefield Publishers Banta, T. W. (2004). Community college assessment (Ed.). San Francisco, CA: Jossey- Bass. Dean, L. A. (2012). CAS professional standards for higher education (8th ed.). Washington, DC: Council for the Advancement of Standards in Higher Education. Miller, B. A. (2007). Assessing organizational performance in higher education. San Francisco: Jossey-Bass. Southern Association of Colleges and Schools (2012). Principles of accreditation: Foundations for quality enhancement (2012 ed.). Decatur, GA: Commission on Colleges of the Southern Association of Colleges and Schools. Selim, B. R., J. Pet-Armacost, A. Alber, and P. S. Krist (2005). The administrative unit assessment handbook: Guidelines for planning and implementing. Retrieved March 13, 2006 from LSC IE (ACE) Handbook 26

30 Appendices Appendix A: LSC Strategic Plan ( ) See Lone Star College s Strategic Plan link below: LSC IE (ACE) Handbook 27

31 Appendix B: Worksheet for SWOT Analysis Strengths Weaknesses External Internal Opportunities Threats LSC IE (ACE) Handbook 28

32 Appendix C: Worksheet for Developing a Vision Statement Directions: Please use the following worksheet to generate a Vision Statement for your unit. You will need to work with your team to brainstorm for ideas, using questions laid out in the worksheet to prompt discussion. After you have ideas, you can then synthesize these ideas and weave them into a Vision Statement. What would you like your unit to become? (the best in or a leader in or nationally recognized for ) What would you like your unit to strive for? (What reputation? What level of excellence?) What would you like your unit to look like in the future? Integrate the above information and compose a Vision Statement: LSC IE (ACE) Handbook 29

33 Appendix D: Worksheet for Developing a Mission Statement Directions: Please use the following worksheet to generate a Mission Statement for your unit. You will need to work with your team to brainstorm for ideas, using questions laid out in the worksheet to prompt discussion. After you have ideas, you can then synthesize these ideas and weave them into a Mission Statement. What are the primary functions and activities that your unit performs? Why do you perform these activities/what's the purpose of your unit? For whom does the unit conduct the activities? Integrate the above information and compose a Mission Statement: LSC IE (ACE) Handbook 30

34 Appendix E: Worksheet for Creating an ACE Assessment Plan Campus Academic Year LSC Strategic Goal & Objective Alignment Unit Objective Action Steps Start Date Finish Date Methods of Assessment Achievement Targets Resources Needed A broad statement that describes what is to be achieved to fulfill the institution's mission Statements of aims or desired ends whose achievement can be assessed to demonstrate progress toward or achievement of their goals Steps or strategies need to be accomplished in order for you to attain your objective? When will initiative/acti on plan begin? When will initiative/acti on plan end? Method & Data selected to measure progress of an outcome. Who will be collecting the data and what will they collect? Intended outcomes, such as % increase over the benchmark data, or expected success rate, or targeted output, etc. Fiscal Impact/Personnel Resources needed to implement plan LSC IE (ACE) Handbook 31

35 Appendix F: Worksheet for Creating an ACE Progress Report and Closing-the-Loop Report Campus Academic Year Information from the ACE Assessment Plan Implementation Stage Reporting Closing-the-Loop Reporting Unit Objective Action Steps Methods of Assessment Achievement Targets Mid-Year Progress Final Results Data Interpretation & Recommendations Statements of aims or desired ends whose achievement can be assessed to demonstrate progress toward or achievement of their goals Steps or strategies need to be accomplished in order for you to attain your objective? Method & Data selected to measure progress of an outcome. Who will be collecting the data and what will they collect? Intended outcomes, such as % increase over the benchmark data, or expected success rate, or targeted output, etc. What action steps were you able to complete? What adjustment to your objectives or action steps do you recommend? Present the actual data and draw objective conclusions on the results. Address if the targets are met or not. Identify what factors impacted the results & recommend further actions to improve the outcomes LSC IE (ACE) Handbook 32

36 Appendix G: Self-Assessment Rubric for Completing ACE Assessment Plans and Reports Assessment Elements Beginning 1 Developing 2 Acceptable 3 Exemplary 4 Unit Objectives Unit objectives are not laid out. Unit objectives are partially completed and are not specific enough. Unit objectives are completed; the objectives are measurable and relevant to the unit's mission and the institution's mission. Unit objectives are clearly laid out, using action verbs; the objectives are specific, measurable, realistic, and relevant to the unit's mission and the institution's mission. P l a n Action Steps Assessment Methods No action steps are laid out. Methods for data collection and data analysis are not identified Action steps are laid out but some are not relevant to the accomplishment of the Unit Objectives. Methods for data collection and data analysis are identified but are not relevant to the measurement of the expected outcomes. Action steps are laid out and are relevant to the accomplishment of the Unit Objectives. One method for data collection and data analysis is clearly laid out and is relevant to the expected outcomes Action steps are laid out with owners and timeline clearly identified; they are also relevant to the accomplishment of the Unit Objectives. Mixed methods for data collection and data analyses are identified and clearly laid out to adequately gauge the fulfillment of the expected outcomes. Achievement Targets No target is set. General statements rather than specific description of the target are presented. Specific targets are set; targets align well with the method of assessment and the unit objectives. Specific targets and baseline data are set based on faculty input and researched information; targets align well with the method of assessment and the unit objectives. LSC IE (ACE) Handbook 33