How to Conduct a Mock Site Visit

Transcription

1 How to Conduct a Mock Site Visit I. Introduction A. About This Course B. Learning Objectives II. Overview A. Meet Tom B. What Happens during the Site Visit? C. More about the Site Visit C. What Is a Mock Site Visit? D. Why Conduct a Mock Site Visit? E. When Should You Conduct a Mock Site Visit? F. Who Should Conduct a Mock Site Visit? G. Use Available Tools H. Let s Practice III. Understanding COA's Rating System A. COA's Rating Scale B. Mandatory Standards C. Let s Review IV. Implementation A. Evaluating Implementation B. Prepare Your Staff and Leaders to Be Interviewed C. More about Preparing Stakeholders to Be Interviewed V. Assigning a Rating A. Think Like a Reviewer B. What You Need in Order to Assign a Rating C. Rating the Standards D. Assigning a Rating to a Specific Standard E. Let s Review F. Let s Practice G. Next Steps H. Remember Tom? IV. Conclusion A. Congratulations B. Resources C. Post-Training Survey

2 Section One: Introduction About This Course Welcome to this training course titled, "How to Conduct a Mock Site Visit." The purpose of this course is to provide you with an overview of conducting a Mock Site Visit, including COA s Rating System. After completing this course, you will be better prepared to conduct a Mock Site Visit, which will ultimately help to prepare you for the actual Site Visit. Please note that links to tools and other resources mentioned throughout this training can be found on the Resources link on the top right of the course viewer. This course is designed for your organization or program's primary contact. It assumes that you have a solid understanding of COA's standards and accreditation process. Therefore, before taking this course, we recommend that you take the following pre-requisite courses: 1. The Format and Structure of COA's Accreditation Standards 2. Assessing Practices against COA's Standards 3. Evidence and Implementation 4. Preparing for the Site Visit 5. Overview of the Site Visit Learning Objectives After completing this course, you should be able to: 1. Identify three benefits to conducting a Mock Site Visit. 2. Understand the basics of COA s rating system. 3. Summarize the steps and tools involved with rating a standard. Please keep in mind that this is a general training. There may be differences in terminology and approach that are specific to your organization or program. You can discuss any questions you may have with your COA Accreditation Coordinator. Section Two: Overview Meet Tom Tom is his organization s primary contact. It is Tom s first time going through the accreditation process. Tom is nervous. He does not know what to expect. What can Tom do to prepare himself and his staff to ensure success? Keep reading. In this course, you will learn about conducting a Mock Site Visit, which is one of the most important steps Tom can take to set himself and his organization or program up for success. What Happens during the Site Visit? Before we get into the topic of how to conduct a Mock Site Visit, let's first take a moment and review what happens

3 during a Site Visit. The following is a quick review of a typical Site Visit: With the exception of COA's Military and Family Readiness (M&FR) and the Child and Youth Development (CYD) program accreditation processes, the Review Team will conduct document reviews on the Sunday afternoon prior to the official start of the Site Visit on Monday morning. More about the Site Visit First thing Monday morning is the Entrance Meeting. During the Entrance Meeting the Review Team meets with your staff and leadership and if you are a nonprofit organization, this includes members of your board. Among other things, the Entrance Meeting provides an outline of how the Site Visit will unfold over the coming days. For M&FR and CYD programs, the Entrance meeting is usually followed by the document review that occurs on Sunday afternoon for other organizations. The bulk of the Site Visit consists of interviews with staff and other stakeholders and tours of your administrative and service-delivery sites where the Reviewers will observe your programs in operation including observing staff/client interactions. The Site Visit ends with a formal Exit Meeting where the Review Team, in broad terms, will describe your organization or program's strengths and opportunities for improvement. What Is a Mock Site Visit? A mock visit is a trail run that you can conduct in preparation for your accreditation Site Visit. It should include an honest assessment of the following: - Your strengths - Your weaknesses - Any areas in need of improvement Why Conduct a Mock Site Visit? A Mock Site Visit can be helpful in a number of ways. First and foremost, a Mock Site Visit can help answer the question: Are we ready for the Site Visit? The answer to this question relies on your readiness in three areas: 1. Standards Implementation Standards implementation is the primary issue at hand. Are you satisfied that you can demonstrate to the Review Team that the standards have been adequately implemented? 2. Staff Preparation A Mock Site Visit can help staff and Board Members/leadership understand the types of questions they may be asked. A Mock Site Visit can help create a level of comfort about the upcoming visit and help lower anxiety. 3. Testing Site Visit Logistics and Ensuring a Smooth Visit A Mock Site Visit can be very useful to test Site Visit logistics: - Do staff know their roles? - Is the on-site evidence properly organized? - Does the room set aside for the Review Team have a working Internet connection? - Is the IT person ready to give the Review team a demonstration of the Management Information System

4 (MIS)? For more information about Site Visit logistics, view the self-paced training titled, Preparing for the Site Visit. When Should You Conduct a Mock Site Visit? There is no simple rule for when an organization or program should conduct a Mock Site Visit because a Mock Site Visit can be conducted at any time. Some accredited organizations perform a Mock Site Visit annually in order to help them maintain their accreditation. However, for this course, we will assume that you will conduct the Mock Site Visit(s) within the last couple of months leading up to your actual Site Visit. You can conduct multiple mock visits. Consider one that focuses on standards implementation around the time that your Self-Study Evidence is due, and possibly earlier if a lot of potential problems were discovered while you were assessing your practices. Closer to the Site Visit, you also should conduct a run-through of Site Visit logistics. Who Should Conduct a Mock Site Visit? If you are undergoing reaccreditation, you probably have staff who are both knowledgeable about your organization or program s practices AND who have been through the accreditation process before. You may even have a trained COA Reviewer on your staff. If you are getting accredited for the first time, focus on staff who are knowledgeable about the standards. There is a good chance that some of these people are on your accreditation coordinating committee. In the end, no one is better qualified to conduct a Mock Site Visit than a trained COA Reviewer. If you do not have a trained COA Reviewer on staff, consider borrowing one from an already accredited organization or program. Go to the "Who Is Accredited" section of the COA website and search for accredited organizations in your area. Then, contact them and ask if they have a COA Reviewer on staff. If you contact an organization that is a member of one of COA's Sponsoring Organizations (like the Alliance for Children and Families, or Catholic Charities USA), the likelihood of finding an organization with a COA Review is greater. For a list of COA's Sponsor Organizations, see the link on the Resources page. Even if you are unable to locate a COA Reviewer in your immediate area, there still is a lot a Reviewer can do over the telephone, the most important being to help you prepare for the staff and stakeholder interviews, which comprise a significant portion of the Site Visit. You also can contact your COA Coordinator for help locating a trained Reviewer. Use Available Tools Many of the tools you may have used when assessing your practices during the Self-Study phase of the accreditation process can be useful for evaluating implementation during a Mock Site Visit. Such tools include: Personnel File Checklist Documentation of Training Worksheet Facility Checklist Standards Assessment and Tracking Worksheet, and others See the Recommended Tools & Resources document on the Resources page.

5 Let s Practice 1. True or False: The benefits to conducting a Mock Site Visit include: 1) verification of your ability to demonstrate to the Review Team that the standards have been adequately implemented, 2) helping staff and Board Members/leadership understand the types of questions they may be asked, and 3) an opportunity to test Site Visit logistics. A. True (That s correct! These are several good reasons to conduct a Mock Site Visit.) B. False (Incorrect. These are the primary benefits to conducting a Mock Site Visit.) 2. True/False: If you do not have a COA Reviewer on staff to help with a Mock Site Visit, it is appropriate to reach out to one from an already accredited organization to see if they have a trained reviewer on staff. You can use the Who is Accredited section of the COA website if necessary. A. True (That s correct! There are many positive benefits to having a trained COA Reviewer to assist with a Mock Site Visit.) B. False (Incorrect. It is certainly appropriate to reach out to an already accredited organization to see if they have a trained reviewer on staff. Also, if you have trouble finding a trained reviewer, you can contact your COA Coordinator for assistance.) Section Three: Understanding COA's Rating System It is important that you understand COA's Rating System, but for the purposes of a Mock Site Visit, unless you have access to a trained COA Reviewer, it's probably sufficient for you to take a "pass/fail" approach. COA's Rating Scale Standards are rated against a four point scale. A rating of 1 is the highest possible rating, and a rating of 4 is the lowest. Ratings of 1 and 2 are considered "passing," and ratings of 3 and 4 are "not passing." Let's look more closely at the four possible ratings. Rating "1": Full Implementation A rating of 1 means that practices fully meet the standard. All elements or requirements outlined in the standard are evident in practice, with few or no exceptions, and any exceptions do not impact service quality or organizational performance. Rating "2": Substantial Implementation A rating of 2 means that infrastructure and practices are basically sound but there is room for improvement. The majority of the standard s elements have been met and the basic framework required by the standard has been implemented. Parts of the standard that are not fully implemented do not significantly impact service quality or

6 organizational performance. Rating "3": Partial Implementation A rating of 3 means that significant aspects of the observed infrastructure and/or practices outlined in the standard require significant improvement. Only part of the standard has been implemented. Service quality or organizational functioning, as it relates to the specific standard, may be compromised. Rating "4": Unsatisfactory Implementation A rating of 4 indicates that implementation of the standard is minimal or there is no evidence of implementation at all. The organization or program's observed infrastructure and practices with respect to the specific standard are weak, non-existent, or show signs of neglect, stagnation, or deterioration. Mandatory Standards To achieve COA accreditation you must receive a rating of 1 or 2 on the following: A. All Purpose Standards B. All Core Concept Standards C. All Fundamental Practice Standards D. Most Practice Standards While it is common for organizations to achieve accreditation with 3 and 4 ratings on some standards, the great majority of Practice Standards must be rated 1 or 2. For an explanation of the different types of standards see How to Read the Standards and Tables of Evidence on the Resources page. Let s Review Choose the best answer. To achieve COA Accreditation, you must receive a rating of 1 or 2 in which of the following: A. All Purpose Standards, all Core Concept Standards, and all Practice Standards (Not quite. To achieve COA Accreditation, you must receive a rating of 1 or 2 in all Purpose Standards, all Core Concept Standards, all Fundamental Practice Standards, and most Practice Standards.) B. All Fundamental Practice Standards and all Core Concept Standards (Not quite. To achieve COA Accreditation, you must receive a rating of 1 or 2 in all Purpose Standards, all Core Concept Standards, all Fundamental Practice Standards, and most Practice Standards.) C. All Purpose Standards, all Core Concept Standards, all Fundamental Practice Standards, and most Practice Standards (That s correct! To achieve COA Accreditation, you must receive a rating of 1 or 2 in all Purpose Standards, all Core Concept Standards, all Fundamental Practice Standards, and most Practice Standards.) D. All Practice Standards (Not quite. To achieve COA Accreditation, you must receive a rating of 1 or 2 in all Purpose Standards, all Core Concept Standards, all Fundamental Practice Standards, and most Practice Standards.)

7 Section Four: Implementation Evaluating Implementation When the Review Team arrives on site they will be looking for evidence that the standards have been implemented. In other words, you will need to demonstrate that the standards have been incorporated into the way that you do business and provide services on a day-to-day basis. Thus, evaluating implementation is one of the central activities when conducting a Mock Site Visit. There are many types of standards and many types of evidence associated with the standards. For an in-depth discussion about evaluating standards implementation, see the self-paced training titled, Evidence and Implementation, which can be accessed through the Training Calendar on the COA website. Preparing Your Staff and Leaders to Be Interviewed Of all the advantages a mock COA Reviewer can bring to your Site Visit preparations, conducting preparatory interviews with your staff and stakeholders may be the most important. The types and depth of questions a Reviewer asks depends on any number of factors. These include: The quality and thoroughness of your Self-Study Evidence and On-Site Evidence The results of the COA Stakeholder Surveys Interviews already conducted with other staff and leaders during the Site Visit Programs already visited If you are undergoing reaccreditation, the Reviewer will have looked at any recommendations for improvement on the Final Accreditation Report from the last time you were accredited. More about Preparing Stakeholders to Be Interviewed For the purposes of a Mock Site Visit we recommend you start by asking a general question. For example, for ETH 2, Conflicts of Interest, you may ask: How does your program address possible conflicts of interest? Then, with your standards at hand, listen for responses that address the related practice standards and ask follow-up questions. You should tailor questions to the level of responsibility of the interviewee. For example, your HR Director will be expected to know the details of your staff grievance process, whereas a staff member may simply need to know that such a procedure exists and how to gain access to the procedure in a way that ensures confidentiality. Section Five: Assigning a Rating Think Like a Reviewer Accreditation has been described as "a model of evaluation that relies on professional judgment to assess the merit of a program." This is why knowledge and experience are crucial qualifications for becoming a COA Reviewer. Remember that COA's Reviewers are your peers. Like you, they are professionals currently working at, or retired from, organizations and programs like yours. They are qualified by experience, knowledge, and training in COA's standards and process.

8 When it comes down to it, assigning a rating is rarely a black or white, yes/no exercise. Modern service-provider organizations and programs are complex and vary greatly according to their size, staffing, geographic location, the services provided, the clients served, and many other variables. COA's standards, especially the Administration and Management (AM) and Service Delivery Administration (SDA) Standards, must apply to many different types of organizations and programs. The standards are written so that they are specific enough to clearly communicate their intent, and general enough to be broadly applicable to the full range of services that COA accredits. Thus, the role of a COA Reviewer is to objectively review the available evidence, then call upon their knowledge and experience to form a professional judgment about whether or not the organization has successfully implemented a standard within the context of COA's Rating System. What You Need in Order to Assign a Rating Once you are done reviewing the evidence, it's time to assign a rating. In order to rate a standard you will need the following: 1. Access to the standards Either online or hard copy. 2. Notes from your review of the evidence We'll assume that you've already looked for implementation (i.e., interviewed stakeholders, done a facility walkthrough, read a procedure, etc.). So at this point, evidence means: Any notes you took during interviews Any tools you may have completed (such as the Facility Checklist) A hard copy of a procedure that you marked up while reviewing it, etc. 3. The Rating Indicators The Rating Indicators define each of the four ratings and provide examples that can be used as guidance when trying to determine what rating to assign. Links to the Rating Indicators can be found on the Resources page. 4. A rating sheet A rating sheet is a document for tracking the ratings you are assigning. We recommend you either: 1. Print a hard copy of the standards to track ratings. This serves the dual purpose of also having a copy of the standards at hand as you assign ratings. OR 2. If you have put a system in place for tracking progress toward implementation of specific standards (use of the Standards Assessment and Tracking Worksheet comes to mind) then it can be very useful to add a "Ratings" column. One of the advantages of this method is that you'll be able to see the history of your work with each standard at the time when you rate it.

9 Rating the Standards When rating the standards, the Review Team will first rate all the Practice Standards under a given Core Concept Standard. Then, when all the Practice Standards have been rated, the Reviewer can assign a rating to the Core Concept Standard. For example, in the Ethical Practices (ETH) section, the first Core Concept Standard is ETH 1, which has five Practice standards (ETH 1.01, ETH 1.02, etc.). When all five Practice Standards in ETH 1 have been rated, the Reviewer can assign a rating to the Core Concept ETH 1 as a whole. The Reviewer will then move on to the next Core Concept Standard ETH 2 and repeat until all six Core Concepts in ETH have been rated. When all six ETH Core Concepts have been rated, the Reviewer can assign a rating to the ETH Purpose standard. Assigning a Rating to a Specific Standard The basic steps for assigning a rating to a standard are as follows: 1. Read the standard. Read the entire standard carefully every word. Sometimes, even the last two words can contain significant content. Don't forget to read any Notes and Interpretations that are associated with the standard as well. 2. Determine its intent. Ask yourself: What exactly is the standard asking us to do? Keep in mind that there are different ways to meet the same standard, even between different programs being delivered by the same organization. 3. Evaluate the evidence. Does the evidence demonstrate that you are meeting the standard? 4. Assign a rating. Assigning a rating of 1 or 4 is usually pretty easy. Differentiating between a rating of 2 and 3 can be more of a challenge, even for the most experienced Reviewers. This where a trained Reviewer brings his or her professional judgment to bear. Additionally, you should know that during the Site Visit, all ratings of 3 and 4 are discussed amongst the Team before a final rating is assigned. Therefore, you may want to take a "pass/fail" approach as was mentioned earlier. Whatever process you use, for the purposes of a Mock Site Visit, be strict with your ratings, especially when rating a Fundamental Practice Standard. If you are satisfied that you will receive a rating of at least 2, then move on. At the end of the day a rating of 2 is as good as a rating of 1. Between now and the Site Visit, you will want to focus your attention on standards you are sure need additional work. Let s Review Choose the best answer. The recommended steps for assigning a rating to a standard are: A. Read the standard, determine its intent, evaluate the evidence, assign a rating (Correct! This is the proper order for assigning a rating.) B. Evaluate evidence, read the standard, assign a rating

10 (Not quite. The proper order is: Read the standard, determine its intent, evaluate the evidence, and assign a rating.) C. Evaluate evidence then assign a rating (Not quite. The proper order is: Read the standard, determine its intent, evaluate the evidence, and assign a rating.) Let s Practice Choose the best answer. As part of conducting your Mock Site Visit, once you are done reviewing the evidence and are ready to assign ratings, COA recommends you have these resources available: A. Access to the standards: Either on-line or hard copy (That s partially correct, but what other resources should you have available?) B. Notes from your review of the evidence (That s partially correct, but what other resources should you have available?) C. Rating Indicators (That s partially correct, but what other resources should you have available?) D. A rating sheet (That s partially correct, but what other resources should you have available?) E. All of the above (That s correct! You should have all of the above-listed resources available once you are done reviewing the evidence and are ready to assign ratings.) Let s Practice True/False: COA recommends rating the standards in this order: Purpose standards, then Core Concepts, then Practice Standards. A. True (Incorrect. When rating the standards, the Review Team will first rate all the Practice Standards under a given Core Concept Standard. Then, when all the Practice Standards have been rated, the Reviewer can assign a rating to the Core Concept Standard. When all the Core Concept Standards have been rated, the Reviewer can assign a rating to the Purpose Standard.) B. False (That s right! When rating the standards, the Review Team will first rate all the Practice Standards under a given Core Concept Standard. Then, when all the Practice Standards have been rated, the Reviewer can assign a rating to the Core Concept Standard. When all the Core Concept Standards have been rated, the Reviewer can assign a rating to the Purpose Standard.) Next Steps If yours is like most organizations, you will have areas you need to follow up on after your Mock Site Visit. Be systematic about how you approach follow-up.

11 Be clear about what specifically still needs to be done (e.g., revise the conflict of interest policy, train staff on the new consent form, install window guards in the new group living facility, etc.), and then assign responsibility for completing all identified tasks to appropriate staff. Remember Tom? As noted above, Tom is his organization s primary contact. It s Tom s first time undergoing accreditation, and he was nervous. He did not know what to expect. Now that he has completed this training, Tom now understands the benefits of conducting a Mock Site Visit, as well as how to go about conducting one. Section Six: Conclusion Congratulations! You have come to the end of this course. We hope you have found this training to be useful and informative. Now that you have completed this course, you should be able to: 1. Identify three benefits to conducting a Mock Site Visit. 2. Understand the basics of COA s rating system. 3. Summarize the steps and tools involved with rating a standard. Resources Click here to access a list of helpful resources. We urge you to visit the Training Calendar and Tools Search pages on the COAnet.org website for more useful accreditation tools and resources. In addition, here is the link to a list of COA's Sponsor Organizations. If at any time you have questions, please contact your coordinator or use the contact us link at Post-Training Survey Thank you for participating in this training. This course was developed in collaboration with Relias Learning, online training partner to the Council on Accreditation. Please click on the survey link in the confirmation you received for this course to share your feedback about this training. We greatly appreciate your input.