Systems Appraisal Feedback Report. Completed in Response to a Systems Portfolio Submitted by

Transcription

1 Systems Appraisal Feedback Report Completed in Response to a Systems Portfolio Submitted by SOUTHERN ARKANSAS UNIVERSITY August 29, 2017 TEAM CHAIR PETER BEMSKI TEAM MEMBERS ELIZABETH DAVIES LINDA LUJAN BRUCE MASSIS SUE SYDOW Higher Learning Commission

2 1 Contents I. Reflective Overview... 2 II. Strategic Challenges Analysis... 4 III. AQIP Category Feedback... 4 IV. Criteria for Accreditation Evidence Screening... 7 V. Quality of the Systems Portfolio... 8 VI. Using the Systems Appraisal Feedback Report... 9 APPENDIX A: Stages in Systems Maturity APPENDIX B: AQIP Category Feedback APPENDIX C: Criteria for Accreditation & Core Component Evidence Screening... 56

3 2 I. Reflective Overview Upon completing its review of the Institutional Overview and Category Introductions included in the Systems Portfolio, the Systems Appraisal team formulated its understanding of the institution, the institution s mission, and the constituents served. This understanding is conveyed in the following Consensus Reflective Statement. Additional team insights are also summarized here in relation to the six AQIP Pathway categories. Reflective Overview Statement Southern Arkansas University (SAU) is a comprehensive, regional university. SAU entered the AQIP Pathway in June 2010 and has engaged in a number of Action Projects since then, including those focusing on making data-informed decisions, assessing student learning, and improving student success for retention, persistence, and graduation rates. SAU provides four-year undergraduate baccalaureate degrees, including more than seventy majors in four distinct colleges. The University also provides associate degrees in four areas. Graduate study is offered through all four colleges, with master s degrees in 26 different fields. In Fall 2016, SAU offered 243 online courses and 56 hybrid classes. In fall 2016, 4771 students were enrolled at the University, 2,700 of whom were enrolled in online sections. SAU entered the AQIP Pathway in June Throughout the Portfolio reported levels of maturity vary and SAU acknowledges opportunities for growth. Category Summary Statements 1. Helping Students Learn: SAU has worked over the past several years to improve its general education assessment and program improvement processes. The processes require annual reporting through electronic software to aggregate data and create reports for better decision-making. The College Assessment Team (CAT) and Assessment Review Council (ARC) analyze the reports for decision-making. Processes vary in their level of maturity, and SAU has undertaken several improvement projects intended to mature processes. 2. Meeting Student & Other Key Stakeholder Needs: SAU analyzes student surveys that the Academic Advising and Assistance Center (AAAC) and other departments use to assist in identifying services for student success in academic and non-academic needs. Retention, persistence, and completion data from IPEDS reporting assists in the development of initiatives focused on improving these rates. The University identifies external stakeholder groups and builds its relationships with partners that align with its mission and the needs of the community. 3. Valuing Employees:

4 3 SAU has undertaken several projects that have helped to mature processes and improve the institution s commitment to hiring, developing, and evaluating employees. The University has made changes to its evaluation processes, including adjusting the timeline to fit the academic calendar and revising the peer evaluation system. SAU s Academy for Professional Development (a result of two Action Projects) offers a comprehensive solution to a common higher education challenge by offering a wide array of professional development opportunities for faculty and staff, including a Mule Kick option for topics and issues of concern campus-wide. 4. Planning and Leading: The Mission is the foundation of SAU s strategic planning and was reviewed in 2015, and the new mission statement, focusing on the Four P s (People, Planning, Programs and Philanthropy), was approved in Additionally, the Office of Institutional Effectiveness (OIE) was created in 2016 to coordinate assessment and planning processes. SAU follows industry conventions related to integrity and transparency. 5. Knowledge Management & Resource Stewardship: SAU has made a significant commitment to improving data utility through staffing enhancements and reorganization. The creation of the Office of Institutional Research and associated positions, in response to feedback from the previous systems appraisal, appears to have led to increasingly visible and effective processes regarding data collection, retention, analysis and use. The University has created both a cost-containment task force and a cost containment oversight committee, which are tracking costsavings efforts (e.g. the new HVAC system installed to contain rising utility costs. The University has also created a more transparent budgeting process. 6. Quality Overview: SAU conducts quality improvement initiatives through the Action Project (AP) process. Various stakeholders under the guidance of the Quality Leadership Team (QLT) administer the AP process, with oversight from the Quality Executive Council (QEC). SAU has an explicit process for proposing for selecting, enacting and evaluating quality improvement initiatives, and appears to have made effective use of that process to improve several areas of activity, including providing professional development, decreasing student attrition, and increasing the availability and use of data.

5 4 II. Strategic Challenges Analysis In reviewing the entire Systems Portfolio, the Systems Appraisal team was able to discern what may be several overarching strategic challenges or potential issues that could affect the institution s ability to succeed in reaching its mission, planning, and overall quality improvement goals. These judgments are based exclusively on information available in the Systems Portfolio and thus may be limited. Each item should be revisited in subsequent AQIP Pathway reviews, such as during the comprehensive evaluation in Year 8. Strategic Challenge: Throughout the portfolio the team noted that Southern Arkansas University reports data from recent applications of the College Employee Satisfaction survey (CESS) and the Student Satisfaction inventory (SSI). Ensuring the appropriate use of instruments specific to a particular category is an opportunity to provide greater perspective, more robust data, and enhance data informed decision-making. Strategic Challenge: The Team noted that throughout the appraisal few internal targets and external benchmarks are reported. Better identification of internal targets and external benchmarks could provide better context for analyzing results. Strategic Challenge: Many processes are recent and the institution has not yet had the opportunity to evaluate them. As these processes mature there will be an opportunity not only to obtain results, but also to review the processes themselves. III. AQIP Category Feedback As the Systems Appraisal team reviewed the Systems Portfolio, it determined the stages of maturity of the institution s processes and results. These stages range from Reacting to Integrated and are described in Appendix A. Through use of the maturity stages and its analysis of the institution s reported improvements, the team offers below summary feedback for each AQIP Pathway category. This section identifies areas for further improvement and also possible improvement strategies. In addition to the summary information presented here, Appendix B conveys the team s specific feedback for all Process, Results, and Improvement items included in the institution s Systems Portfolio. The summary feedback below, and the detailed feedback offered in Appendix B, is based only upon evidence conveyed in the Systems Portfolio. It is possible that the institution has additional information on specific processes,

6 5 results and improvements that was not included in the Systems Portfolio. In such instances, the institution should plan to provide this evidence in a future AQIP Pathway review process. Category 1: Helping Students Learn SAU is at a systematic level of maturity for most of its processes for Category One: Helping Students Learn. The University collects and reports assessment data using an electronic software package. SAU has completed several Action Projects to enhance data collection and analysis. As these processes mature, using more internal targets and external benchmarks may assist SAU improve both processes and results. SAU has made progress on diversity and multiculturalism since its most recent Systems Appraisal, however the processes and results described in this category do not demonstrate the maturity level or results the University seeks. Continued focus on ensuring all faculty are engaged in the development, review/revision, and assessment of institutional, program, and course level learning outcomes may strengthen SAU s processes and results. Strategic issues: SAU should look at various direct and indirect measures of data, drilling down into subcategories of data in order to make better informed decisions. Setting internal and external benchmarks may assist SAU in improving both processes and results. Many of the processes are relatively new, and as these mature the use of longitudinal data may improve the interpretation of results. Category 2: Meeting Student & Other Key Stakeholder Needs SAU acknowledges it is still reacting to the analysis and interpretation of results for meeting key stakeholder needs, complaint processes, and building collaborations and partnerships. There is an opportunity to better align and integrate this diverse set of processes across the organization, which may help the institution to effectively use what it learns. SAU s has processes in place for meeting the needs of students as well as retention, persistence, and completion and recognizes the need to ensure buy-in from all appropriate stakeholders. Action Projects are being considered that may result in formal processes in this category. A Retention Task Force recommendation will be considered for adoption. While there is evidence that many partnerships are effective, and that SAU is working to increase its maturity level with regard to a number of processes (e.g. student complaints), areas of growth for SAU include processes for selection of tools by which to measure effectiveness and use of internal and external benchmarks. Strategic issues:

7 6 The University is aware that many of its processes are new and that systematic data collection and analysis of results are still in the reacting level of maturity. There is a need to set targets and identify benchmarks in several areas. Category 3: Valuing Employees The University acknowledges opportunities for improvements in data aggregation in the area of professional development. Software is being piloted for faculty annual summaries, which may result in better collection and reporting of faculty accomplishments. The Academy for Professional Development, the result of two AQIP Action Projects, has provided professional development opportunities for SAU s staff, however; a number of employees do not participate. The President s task force on retention and graduation has proposed that faculty development, especially in the area of teaching, should become a strategic priority for the institution. Strategic Issues: There is a need for targets and benchmarks, as well goals and strategies for measuring results. Category 4: Planning and Leading SAU s processes in this category are generally systematic and there is good involvement from internal and external stakeholders. The university relies in large part on data collection from the College Employee Satisfaction Survey (CESS) and the Student Satisfaction Inventory (SSI) and has an opportunity to better identify how well it is achieving results through diversifying the selection of tools, methods, and instruments to help directly measure outcomes. Other areas of opportunity include integrating the strategic planning process and strengthening the academic integrity process. Strategic Issues: The university has an opportunity to identify measures and metrics designed for aggregation and analysis to enable the creation of internal targets and external benchmarks. This data may also provide better insight and interpretation toward data informed decision-making. Category 5: Knowledge Management & Resource Stewardship SAU s processes in the areas of knowledge management and resource stewardship, by its own acknowledgement, are relatively young and therefore vary in their level of maturity. Its pilot of a transparent and predictable budget process could become a strength. The institution has the opportunity to better utilize internal targets and external benchmarks to determine results. As an example, the physical plant audit could be useful as a direct measure of the physical plant

8 7 assessment. The CESS provides useful information, but there are other measures that may be identified to yield more direct measures in this category. Clearly defining processes and measure results for all resource areas (fiscal, physical and technological infrastructures) may improve maturity in this area. Strategic issues: Better use of a variety of measures, internal targets, and external benchmarks Category 6: Quality Overview Since 2010 SAU has been committed to the AQIP Pathway. SAU describes a culture of continuous quality improvement that is evolving, expanding, and becoming much more the norm. Organizational structure, communication processes, and use of AQIP elements, including Action Projects, have helped the institution continue to move to a more aligned level of maturity. The university provides a number of Action Project results as evidence. SAU s greatest opportunity lies in being more systematic in identifying internal targets, external benchmarks, and appropriate measures for determining progress and improvement. Results of these measures could continue to be shared, aggregated, analyzed, and used in decision-making. IV. Criteria for Accreditation Evidence Screening The Systems Appraisal team screened the institution s Systems Portfolio evidence in relation to the Criteria for Accreditation and the Core Components. This step is designed to position the institution for success during its comprehensive evaluation in Year 8. In order to accomplish this task, HLC has established linkages between the Systems Portfolio s Process and Results items and the Criteria s Core Components. Systems Appraisal teams have been trained to conduct a soft review of the Criteria and Core Components for Systems Portfolios completed in the third year of the AQIP Pathway cycle and a more robust review for Systems Portfolios completed in the seventh year. The formal review of the Criteria and Core Components for purposes of reaffirming the institution s accreditation occurs only in the eighth year of the cycle and is completed through the comprehensive evaluation, unless serious problems are identified earlier in the cycle. As part of this Systems Appraisal screening process, teams indicate whether each Core Component is Strong, clear, and well-presented ; Adequate but could be improved ; or Unclear or incomplete. When the Criteria and Core Components are reviewed formally for reaffirmation of accreditation, peer reviewers must determine whether each is Met, Met with concerns, or Not met.

9 8 Appendix C of this report documents in detail the Appraisal team s best judgment as to the current strength of the institution s evidence for each Core Component and thus for each Criterion. Institutions are encouraged to review Appendix C carefully in order to guide improvement work relative to the Criteria and Core Components. Immediately below, the team provides summary statements that convey broadly its observations regarding the institution s present ability to satisfy each Criterion, as well as any suggestions for improvement. Again, this feedback is based only upon information contained in the institution s Systems Portfolio and thus may be limited. Criterion 1. Mission: Strong, clear and well presented. Criterion 2. Integrity: Ethical and Responsible Conduct Strong, clear and well presented. Teaching and Learning: Quality, Resources, and Support For the most part, evidence is strong, clear and well presented. In 3.C.1 clear evidence as to how adjunct faculty are evaluated is lacking. In 3.C.6 evidence is lacking as to how adjunct faculty are trained. These result in feedback of Adequate, but could be improved. Criterion 4. Teaching and Learning: Evaluation and Improvement For the most part evidence is strong, clear and well presented. There are two exceptions: 4.A.6: There is no specific evidence provided reporting on employment rates, admission rates to advanced degree programs, and participation rates in fellowships, internships, and special programs. 4.C.3: Nothing is reported. As the recommendations from the retention Task Force are implemented evidence should be gathered and reported. These two exceptions result in feedback of Unclear or Incomplete. Criterion 5. Resources, Planning, and Institutional Effectiveness Strong, clear and well presented. V. Quality of the Systems Portfolio [Here the Systems Appraisal team is invited to share its impression of the overall quality of the Systems Portfolio by considering such factors as the portfolio s coherence, consistency, honesty, readability and

10 9 perceived usefulness to the institution in helping with its own improvement efforts. Appraisal teams are encouraged to review the Tips for Writing Feedback section of the Systems Appraisal procedure document when writing this section so as to provide honest but professional feedback.] The team found the Southern Arkansas University Systems Portfolio to be well written, and consistent with HLC guidelines. Categories are addressed directly and shortcomings are readily acknowledged, which gives the team confidence that they will be addressed. Evidence for the Core Components is easily discerned, and the Portfolio reads clearly and consistently throughout. VI. Using the Systems Appraisal Feedback Report The Systems Appraisal process is intended to foster action for institutional improvement. Although decisions about specific next steps rest with the institution, HLC expects every institution to use its Feedback Report to stimulate improvement and to inform future processes. If this Appraisal is being completed in the institution s third year in the AQIP Pathway cycle, the results may inform future Action Projects and also provide the focus for the institution s next Strategy Forum. In rare cases, the Appraisal completed in the third year may suggest either to the institution itself or to the Commission the need for a mid-cycle (fourth year) Comprehensive Quality Review. If this Appraisal is being completed in the institution s seventh year in the cycle, again the results may inform future Action Projects and Strategy Forums, but more immediately they should inform institutional preparation for its comprehensive evaluation in the eighth year of the cycle when the institution s continuing accredited status will be determined along with future Pathway eligibility. Institutions are encouraged to contact their staff liaison with questions.

11 10 APPENDIX A Stages in Systems Maturity: Processes Reacting Aligned Integrated The institution focuses on activities and initiatives that respond to immediate needs or problems rather than anticipating future requirements, capacities, or changes. Goals are implicit and poorly defined. Informal procedures and habits account for all but the most formal aspects of institutional operations. The institution is beginning to operate via generally understood, repeatable, and often documented processes and is prone to make the goal of most activities explicit, measurable, and subject to improvement. Institutional silos are eroding and signs of coordination and the implementation of effective practices across units are evident. Institutional goals are generally understood. The institution operates according to processes that are explicit, repeatable and periodically evaluated for improvement. Processes address key goals and strategies, and lessons learned are shared among institutional units. Coordination and communication among units is emphasized so stakeholders relate what they do to institutional goals and strategies. Operations are characterized by explicit, predictable processes that are repeatable and regularly evaluated for optimum effectiveness. Efficiencies across units are achieved through analysis, transparency, innovation, and sharing. Processes and measures track progress on key strategic and operational goals. Outsiders request permission to visit and study why the institution is so successful. Stages in Systems Maturity: Results Reacting Aligned Integrated Activities, initiatives, and operational processes may not generate data or the data is not collected, aggregated, or analyzed. Institutional goals lack measures, metrics, and/or benchmarks for evaluating progress. The monitoring of quality of operational practices and procedures may be based on assumptions about quality. Data collected may not be segmented or distributed effectively to inform decision-making. Data and information are collected and archived for use, available to evaluate progress, and are analyzed at various levels. The results are shared and begin to erode institutional silos and foster improvement initiatives across institutional units. The tracking of performance on institutional goals has begun in a manner that yields trend data and lends itself to comparative measures in some areas. Measures, metrics and benchmarks are understood and used by all relevant stakeholders. Good performance levels are reported with beneficial trends sustained over time in many areas of importance. Results are segmented and distributed to all responsible institutional units in a manner that supports effective decision-making, planning and collaboration on improvement initiatives. Measures and metrics are designed to enable the aggregation and analysis of results at an institutional level. Data and information are analyzed and used to optimize operations on an ongoing basis. Performance levels are monitored using appropriate benchmarks. Trend data has been accrued and analyzed for most areas of performance. Results are shared, aggregated, segmented and analyzed in a manner that supports transparency, efficiency, collaboration and progress on organizational goals. Measures and metrics for strategic and operational goals yield results that are used in decision-making and resource allocations.

12 11 APPENDIX B AQIP Category Feedback Category 1: Helping Students Learn Category 1 focuses on the design, deployment and effectiveness of teaching-learning processes (and the processes required to support them) that underlie the institution s credit and non-credit programs and courses. 1.1: Common Learning Outcomes 1P1 Process Aligning common outcomes to the mission, educational offerings and degree levels of the institution Determining common outcomes Articulating the purposes, content and level of achievement of the outcomes Incorporating into the curriculum opportunities for all students to achieve the outcomes Ensuring the outcomes remain relevant and aligned with student, workplace and societal needs Designing, aligning and delivering cocurricular activities to support learning Comments on Process Maturity and Improvement Aligned SAU s 2012 revision to the University Learning Goals, promulgated by the General Education Committee, is evidence of a process that functions to align the university s educational offerings to its mission and degree levels. SAU complies as well with Arkansas mandated general education requirements. Aligned An established process is in place for determining common outcomes. The academic community is well represented on the GEC (General Education Committee) and an improvement initiative led to significant revisions in SAU communicates the purpose, content, and level of achievement of the outcomes through their catalogs and implementation of a standard course syllabus template. It is unclear, to what extent the common syllabus is being used. Increasing its use and putting a process in place for review of that use may assist the University to better serve its stakeholders. Aligned All candidates for undergraduate degrees are required to satisfy General Education requirements, (Table 1-2). The process of assessment is regular and repeatable, and results are used to improve outcomes. The mandates of the Arkansas Department of Higher Education are reflected in the framework for general education. The Board of Trustees provides input as do faculty. It is not clear that a formal process is in place to review and ensure relevant outcomes. Having such a process in place may help SAU to become aligned in this category. Aligned SAU has aligned co-curricular offerings to the Council for Advancement of Standards in Higher Education (CAS) standards. Abundant co-curricular opportunities exist for SAU students. New student organizations follow a process for proposal, approval, and assessment to ensure alignment with both the University s learning

13 12 goals and mission. Selecting tools, methods and instruments used to assess attainment of common learning outcomes Assessing common learning outcomes Other identified processes Processes are in place for faculty, working with the Office of Institutional Effectiveness, to select appropriate tools, methods and instruments. It is not clear how comprehensive faculty participation is, and it is also unclear how consistent this is across the University. Clearly defining and communicating the extent of these processes and the expectations of departments and faculty may provide an opportunity for better alignment. Aligned Explicit, repeatable processes are used to assess general education learning outcomes, guided by a number of relevant constituencies (faculty, Office of Institutional Effectiveness, Gen. Ed. Comm.). These constituencies guide assessment of common learning outcomes. 1R1 Results Summary results of measures (including tables and figures when possible) Comparison of results with internal targets and external benchmarks Interpretation of results and insights gained Evaluation of Results and Systems Improvement Data are collected, archived, and analyzed at various levels, as exemplified by the tables and charts presenting Undergraduate Learning Goals results and the AAC&U Value Rubrics. Trend data are presented, and, when appropriate, further analyzed to determine how to make improvements (e.g. Ethics). Results suggest that performance levels and trend data are mixed, and it is unclear how the data are obtained i.e. how courses that are sampled are picked. A review of all levels of education offered may help SAU become better aligned in this category. Internal targets have been set and trend data and results are being used to determine student success of the outcomes. While reference is made to widely used external measures (e.g. AACU Value Rubrics), it is unclear how SAU is comparing its results to those of other institutions using the same measures, which would strengthen the analysis. Analysis of the results takes place through faculty discussions, resulting in increased general education strength. Similar initiatives addressing all levels of degrees and a more formal process for analysis may help SAU to strengthen curriculum throughout the programs offered. 1I1

14 13 Evaluation of Improvement Efforts SAU is prepared to begin considering how targets are set, and to broaden the use of the assessment tool, engaging more faculty in the process, which will lead to more robust insights and make possible clearer interpretations of data by subcategory (e.g. online class, dual credit classes, etc.) In the context of its relatively newly established practice of assessing General Education, SAU is working to refine its processes of assessment. 1.2: Program Learning Outcomes 1P2 Process Aligning program learning outcomes to the mission, educational offerings and degree levels of the institution Determining program outcomes Articulating the purposes, content and level of achievement of these outcomes Ensuring the outcomes remain relevant and aligned with student, workplace and societal needs Designing, aligning and delivering cocurricular activities to support learning Comments on Process Maturity and Improvement The Assessment Review Council (ARC) reviews program specific learning goals and objectives for congruence with ULGs and SAU mission. A recent Action Project asked programs to examine their learning outcomes and develop a process for regular review. Completing several cycles and engaging more faculty and programs will provide SAU an opportunity to assess its success in using this process. Aligned Faculty and department chairs at SAU develop program goals and objectives that are specific to the discipline, discipline specific state mandates, and specialized accreditation standards, which align to the University Learning Goals. Updates as necessary were provided as a result of a recent Action Plan. Continuing a focus on this work may ensure the processes are understood, effective, and widely used. The University website posts program outcomes and a standardized syllabus provides the opportunity to do the same. It is not clear how extensively the template is used. An assessment of the degree to which templates are actually used across the University, and wider dissemination of program outcome expectations and data may elevate this process. SAU s University Learning Goals are designed to support societal needs and are reviewed by faculty working within each program. External review is conducted by programs regulated by specialized accreditation. Programs without specialized accreditation participate in a state mandated self-study and review process. Implementing an explicit, repeatable review process for evaluating improvement across all units may move this to a higher level of maturity. Programming and selection of co-curricular activities is done in alignment with program learning goals. The activities do not appear to be aligned between different programs or colleges at the University, but rather at the level of program learning goals. A formal

15 14 Selecting tools, methods and instruments used to assess attainment of program learning outcomes Assessing program learning outcomes Other identified processes process for identifying and measuring the impact of cocurricular activities may better serve the needs of students across the University. SAU s use of the OIE and Assessment Coordinator, the CAT process, electronic software, its work to develop assessment rubrics and plans, and its discussion of what is working, what is not working, and what is still to be accomplished shows a understanding of how its processes for selecting tools, methods, and instruments will work. Given time and continued focus, SAU appears to have the capacity to move this process to a higher level. At this time the process seems to be on hold due to the suspension of the College Assessment Team (CAT). Faculty annually collect program assessment data following their program s assessment plan and store data using electronic software. The University acknowledges that it has suspended its College Assessment Teams (CAT) review for the past year to revise its process through an Action Project. Once the Action Plan is completed and more programs take part in the process, SAU may be better aligned in this category. 1R2 Results Overall levels of deployment of the program assessment processes within the institution (i.e., how many programs are/not assessing program goals) Summary results of measures (including tables and figures when possible) Comparison of results with internal targets and external benchmarks Evaluation of Results and Systems Improvement SAU has made progress in engaging more programs in the assessment process and recognizes the need for all programs to participate. A recent Action Project assisted two programs that had not completed an assessment plan to do so. Ensuring participation throughout the university may result in better alignment of results. Programs having more mature processes may serve as exemplars for replication. Table 1-10 provides access codes for electronic software reports of program outcomes by program. Participation in program assessment as reported in Table 1-6 is difficult to assess without knowing whether enrollment in each College has increased or declined (i.e. data are presented as raw numbers rather than percentages). Having measures, metrics and benchmarks understood and used by all relevant stakeholders could move this to a higher level of maturity. The University data shows that some programs lack clear targets and/or fail to collect and report data for all required objectives. Data indicates that approximately 80% of programs regularly submit reports and as improvement in meeting performance thresholds is achieved, alignment may improve.

16 15 Interpretation of assessment results and insights gained SAU provides some interpretation of its results and discusses insights, such as the need to increase program review compliance beyond 80% and the need to improve the rubric dimensions and benchmarks for program performance targets. As the University collects more data and looks at external benchmarks, it has potential to learn much about the effectiveness of it processes for program assessment. Evaluation of Improvement Efforts Action Projects have used effectively in bringing all programs into compliance in identifying program learning outcomes; and SAU is working to bring all programs into compliance with the Program Assessment Plan Cycle and to improve the quality of assessments, which it has determined should be the next step in the ongoing review process. Past, current, and future Action Projects are intended to strengthen both processes and results in this category, allowing for ongoing improvement. 1.3: Academic Program Design 1P3 Process Identifying student stakeholder groups and determining their educational needs Identifying other key stakeholder groups and determining their needs Developing and improving responsive programming to meet all stakeholders needs Comments on Process Maturity and Improvement Student stakeholder groups are identified by using external measures (NSSE, SSI), internal measures (student surveys, advisor surveys), and by examining previous student success data. In addition, for already identified stakeholder groups (e.g. multicultural students), existing offices design programs to determine needs. Describing the coordination and communication among units to improve processes may move this to a higher level of maturity. SAU describes how its program advisory groups and internal SAU stakeholders provide input. It is not clear that the University takes advantage of the extensive amount of information available through government and other agencies to better identify such groups and determine their needs. Creating repeatable processes that are periodically evaluated for improvement may move this to a higher level of maturity. The University conducts analyses of market needs and career opportunities that align with its mission in developing and improving programming and delivery modalities to meet its stakeholders needs. Less evident are the processes involved in doing this and how these processes are systematically used across the university. Clearly defining processes may assist SAU in increasing its

17 16 responsiveness to its stakeholders and assuring alignment to mission. Selecting the tools, methods and instruments used to assess the currency and effectiveness of academic programs Reviewing the viability of courses and programs and changing or discontinuing when necessary Other identified processes SAU s policy requires programs to produce annual reports along with seven-year reports. SAU uses the Student Satisfaction Inventory (SSI) and the College Employee Satisfaction Survey (CESS) to verify that the needs of its diverse stakeholders are being met. A process for selecting and integrating those and other tools may be useful to improve the maturity of this category. Aligned State-required viability reports insure that programs are reviewed on a regular basis. There is an explicit, repeatable process for proposing and approving needed curricular changes, which involves all stakeholders, including faculty, Board of Trustees and the Arkansas Department of Higher Education, as well as accrediting bodies. 1R3 Results Summary results of assessments (including tables and figures when possible) Comparison of results with internal targets and external benchmarks Interpretation of results and insights gained Evaluation of Results and Systems Improvement SAU presents data on selected questions from the SSI and CESS external measures, as well as results of an external consultant s study regarding the feasibility of a capital campaign and the uses made of enrollment data/viability studies. Additional analysis of internal and external measures, drilling down into the sub-category data, and looking at data from additional stakeholders, may elevate this process. Results are compared with external (SSI and CESS) benchmarks nationally, but a more nuanced comparison within and between stakeholder groups at the University may also prove informative. Insights appear to be limited by the choice of measures used; results are interpreted soundly but limited in their generalizability to specific groups and stakeholders. The use of a variety of data points from a number of sources may assist to drive data based decisionmaking. 1I3 Evaluation of Improvement Efforts

18 17 SAU reports that the final recommendations of a presidential task force will not be reported to faculty until Fall 2017, but the panel has tentatively indicated that changes in program design will be recommended as part of an Action Project for improving retention and completion rates. It is likely, for example, that the task force will endorse a comprehensive review of all eight-semester degree plans to address barriers to student success and to ensure that departmental plans properly reflect course rotations. 1.4: Academic Program Quality 1P4 Process Determining and communicating the preparation required of students for the specific curricula, programs, courses and learning they will pursue Evaluating and ensuring program rigor for all modalities, locations, consortia and dual-credit programs Awarding prior learning and transfer credits Selecting, implementing and maintaining specialized accreditation(s) Assessing the level of outcomes attainment by graduates at all levels Comments on Process Maturity and Improvement Aligned SAU informs students of prerequisites through their Graduate and Undergraduate Catalogs and the SAU website. Prerequisites are set by state mandates along with faculty, department chairs, and deans. Advisors and course syllabi also provide students with information about prerequisites and other pertinent information. Aligned Programs in all forms and places of delivery are evaluated on an ongoing basis, using the same criteria. Online courses have selfevaluation tools and primers not mentioned as available in equivalent formats for non-online courses, some programs utilize externally normed exams and/or admissions criteria while others do not, and some courses (e.g. concurrent credit HS/college courses) are certified by a national organization (NACEP). Aligned Explicit and replicable processes administered by appropriate university, state, or private agencies and codified in some cases by articulation agreements ensure that prior learning is assessed reliably and validly within each domain of learning (transfer credit, military, international education, etc.). A number of programs have specialized accreditation. The process for initiating and maintaining accreditation appears to vary in part by college but is also integrated into the strategic planning process through an unspecified process. Sharing processes among units could provide best practices for other units to follow. SAU annually reviews program assessment reports that include measures of student learning. All undergraduates must achieve a minimum GPA of 2.0 to graduate and graduate students must achieve a minimum GPA of 3.0. An opportunity exists for the University to aggregate data and implement effective practices across all departments to move to the aligned level of maturity.

19 18 Selecting the tools, methods and instruments used to assess program rigor across all modalities Other identified processes SAU describes the tools it has used in its assessment processes along with some minimal rationale for selecting them. This section may be strengthened by more intentionally explaining not only the process for selecting tools, methods and instruments but the rationale for selecting them. 1R4 Results Summary results of assessments (including tables and figures when possible) Comparison of results with internal targets and external benchmarks Interpretation of results and insights gained Evaluation of Results and Systems Improvement SAU provided results in Table 1-16 from the 2015 NSSE and from the fall 2015 Student Satisfaction Inventory (SSI), Table The University compared SSI results from 2012 and 2015 to establish trends in Table Additional research conducted for the capital campaign feasibility study indicated a decrease in excellent ratings but a comparable increase in very good ratings. The University may want to determine other direct measures to support effective decision-making, planning and collaboration to improve the quality of its programs. Results on NSSE and SSI are compared to results from other institutions, but direct measures of program quality were not compared to internal targets or external benchmarks. Limitations in data collection appear to limit the possibility of data interpretation and in turn limit the insights gained from that interpretation. General conclusions are drawn from focus group, interview and online survey participants responses. 1I4 Evaluation of Improvement Efforts The University acknowledges that program assessment compliance needs to reach 100% to strengthen program design and quality to enhance student success. A Presidential task force on retention and completion has met and recommendations will be available in fall The anticipated recommendations may include: (1) ensuring a minimum standard of quality in our online courses and fully online programs (including the possibility of pursuing Quality Matters certification); (2) reviewing academic programs to ensure that requirements are clear, and that neither prerequisites nor lack of course availability hamper students progress through their major; and (3) building in regular checkpoints to monitor students who may be at risk of failure. The task force is also expected to recommend ideas for enhancing student success in general education courses without diminishing academic quality.

20 19 1.5: Academic Integrity 1P5 Process Ensuring freedom of expression and the integrity of research and scholarly practice Ensuring ethical learning and research practices of students Ensuring ethical teaching and research practices of faculty Selecting the tools, methods and instruments used to evaluate the effectiveness and comprehensiveness of supporting academic integrity Other identified processes Comments on Process Maturity and Improvement Aligned The AAUP Standards of Ethics are followed. The Faculty Handbook commits SAU to academic freedom and policy requires that faculty sit on the IRB. A process is in place for research related grievance. Academic integrity guidelines are explicitly taught to students during orientation, articulated in the catalog and website, and if violated, lead to an explicit and replicable disciplinary and reporting process. SAU may want to consider adding that students need to have a grievance procedure for student due process. It is unclear whether SAU has a procedure for due process of student grievances. The University has a Code of Ethics that is published in the University Handbook. The University conveys its expectations for ethical behavior in faculty scholarship in the Faculty Handbook. In addition, review boards ensure that all research involving human and animal subjects conforms to the highest standards of ethical behavior. Less evident is the process for ensuring ethical teaching. The University may want to offer explicit training for faculty regarding ethics in teaching with follow-up evaluation of the behaviors of those trained. The tools, methods, and instruments used include reports on academic integrity violations, number of IRB applications, response time on IRB requests, and NIH certifications; the CESS; and the SSI. Of all these tools, the CESS and SSI may provide the least objective information as it s based on opinions of those willing to take the surveys. SAU may wish to discuss other ways to engage faculty and students in evaluating academic integrity. This could strengthen both its processes and its results. 1R5 Results Summary results of measures (including tables and figures when possible) Evaluation of Results and Systems Improvement Student participation in remedial academic integrity workshops at the library is used as a measure, despite questions regarding the validity of this measure (e.g. faculty may not report all violators, TurnItIn may be leading to increased detection of violators, or more students may be cheating). Assessment of IRB performance relies on length of time to approval, number of submissions, and percent of IRB members with NIH certification, each of which is improving. It is not clear that the information is useful in and of itself or for

21 20 measuring effectiveness. Comparison of results with internal targets and external benchmarks Reacting No internal targets are provided; external benchmarks are provided only for the indirect measures of the SSI and CESS. Interpretation of results and insights gained Analysis of IRB data appears to indicate a positive trend, but the insights gained seem unlikely to be of use in other areas. Interpretation of other results is less clear, with a resulting limitation on insights. An opportunity exists for SAU to follow up on the increased numbers of reported academic violations to determine trends. 1I5 Based on 1R5, what process improvements have been implemented or will be implemented in the next one to three years? Evaluation of Improvement Efforts SAU continues to explore ways to address academic dishonesty issues. An Action Project is planned this year to benchmark IRB policies at comparable institutions to address issues around recertification of committee members. Category 2: Meeting Student and Other Key Stakeholder Needs Category 2 focuses on determining, understanding and meeting needs of current and prospective students and other key stakeholders, such as alumni and community partners. Category 2: Meeting Student and Other Key Stakeholder Needs 2.1: Current and Prospective Student Needs 2P1 Process Identifying underprepared and at-risk students, and determining their academic support needs Comments on Process Maturity and Improvement Aligned Criteria for identifying underprepared students are explicit; at-risk students are identified by faculty or staff as well as through an early-alert system. Each college has an embedded professional academic advisor to support students and provides accurate academic advising and targets specific populations for specialized services. Placement scores, number

22 21 Deploying academic support services to help students select and successfully complete courses and programs Ensuring faculty are available for student inquiry Determining and addressing the learning support needs (tutoring, advising, library, laboratories, research, etc.) of students and. faculty Determining new student groups to target for educational offerings and services Meeting changing student needs Identifying and supporting student subgroups with distinctive needs (e.g., seniors, commuters, distance learners, military veterans) Deploying non-academic support services to help students be successful Ensuring staff members who provide academic and nonacademic student support services are qualified, trained, and supported of hours completed, as well as faculty and staff all assist in identifying atrisk students and guiding them to the appropriate services. Students are placed in developmental courses in reading, writing, and/or mathematics based on established state placement requirements, high school GPA, and ACT (or equivalent) test scores. Coordinated by the Office of Transitional Studies, programs are in place to address the needs of academically underprepared students. Disability Student Services provides assistance for students seeking academic accommodations. It is not clear how these processes are linked and how they align. Better linkage may result in less duplication of effort and better services to students. All faculty are required to post each semester on their office doors a schedule indicating their class hours and at least eight hours per week of regularly scheduled office hours. An opportunity exists to survey students to determine their level of satisfaction with the process. SAU offers a wide array of student and faculty support services that include advising, disability service support, tutoring, counseling, supplemental instruction, an Early Alert System, workshops on research and library use, travel awards for faculty, professional laboratories, etc. for students and faculty development. It is not clear that t these processes align with each other. Processes could be strengthened by explicitly articulating the integration between them in order to ensure coordination. SAU uses direct and indirect measures to identify new student groups to serve; the results are incorporated into the strategic planning process. SAU seeks input from prospective students, current students, and alumni through surveys, industry feedback, advisory councils, and monitoring of best practices to drive some new educational offerings. It is unclear how this information is processed or turned into strategic action. SAU meets the changing needs of its students by collecting and analyzing input from various stakeholder groups reported through surveys and faculty evaluations. It is unclear how this information is processed or turned into strategic action. Describing how data is communicated, evaluated, and analyzed across all units may move this to a higher level of maturity. Student subgroups are identified through demographic data. Students can self-identify as part of a group with distinctive needs. Processes by which appropriate support services are designed for target student groups are unclear. Better identifying needed support, and establishing measures to determine the effectiveness of these services may move this to a higher level of maturity. Reacting The response provided does not address this. Aligned SAU s hiring processes ensure properly qualified personnel are selected to fill non-academic student support positions. Hiring processes are consistent with best practices and Council for the Advancement of Standards in Higher Education guidelines. Training and professional development opportunities are made available to staff members.