Regist ry Vendor Assessm ent

Regist ry Vendor Assessm ent April, 2014 Completed by the (NQRN ) The PCPI Foundation and the disclaim any liability for use or non-use of this document. The PCPI does not provide medical, legal, financial, or other professional advice and users are encouraged to consult a professional for such advice. This document includes samples of a Request for Proposal, Cost Estimator Worksheet and Vendor Evaluation Scorecard. These are provided as examples only. Users should develop their own documents specific to their needs. If you have any questions, please contact Seth Blumenthal at 312-464-4049 or seth.blumenthal@ama-assn.org. 2016 The PCPI Foundation. All Rights Reserved. Page 1 of 16

Introduction Given the mission-critical importance of a registry, time spent evaluating the capabilities of vendors can go a long way towards a successful project 1. This document describes a process and a set of tools that can help registry stewards select vendors that best meet their needs. The Vendor Selection Process Vendor selection can be more effective when a process is followed (Table 1). Table 1: The Vendor Selection Process Step Description Tool Build a project team Include representatives from organizations that will use the registry, as they can give valuable feedback. Additionally their early participation in the project can help develop them as registry champions useful during the initial build up of participation. Define needs Describe the primary activities of the registry with sufficient detail to determine what activities can be supported in house, and what may require outsourcing to a vendor. Create a list of vendors Narrow down to a short list and contact short listed vendors with initial expression of interest Review vendors websites to help determine maturity, such as years in business, number of clients and installation, level of professionalism of the website, and if there are physician owners or advisers on staff. Create a short list of vendors to contact by looking for evidence that the vendor has successfully supported multiple registry installations and clients in the past. Ask for customer references from the vendors and from colleagues. Through the NQRN and other professional societies and organizations, discuss with others their experience with particular vendors and their implementation progress Use an intake form to keep track of basic information about the project and on a prospective vendor, including contact information from both parties 2. Intake form Eliminate vendors that haven t been around long enough or that lack the necessary expertise. Ensure that vendors are conversant in the clinical areas that the registry is expected to cover, and that they can scale as the registry grows. Issues uncovered here may influence either the choice of vendor or the contractual provisions needed to ameliorate the issues. 2016 The PCPI Foundation. All Rights Reserved. Page 2 of 16

Evaluate vendors Make a recommendation Intake Form Engage prospective vendors with a Request for Proposal that describes detailed requirements and how the vendor should respond. Invite the vendors with the best responses in for a detailed demonstration. With advance preparation, demos can be more impactful and useful for evaluation. Estimate initial and ongoing costs Use a scorecard to evaluate multiple vendors against technical and non-technical criteria. Present results in a visually impactful way, showing how vendors compared against requirements. Request for proposal (RFP) Demonstration checklist Cost worksheet Scorecard Recommendations Report An intake form can be used with prospective vendors on initial contact. It contains basic information about the solicitation, as well as contact information for key personnel, both within the registry team and at the prospective vendor. Sample content for an intake form: Registry team contact information Vendor contact information A description of the products or services being obtained Subject matter requiring legal attention (eg storage and collection of certain data, privacy issues, HIPAA, need for business associate agreements) Contact information for people who will need to use the software Contractual provisions planned for inclusion in the contract (eg acceptance testing, service level agreement) Specific individuals that need to be contacted regarding any aspect of this transaction (eg approvals) Request for Proposal (RFP) If practical, going through the contract process with three or more potential vendors will provide additional options, create awareness of the competitive nature of the bid and increase the chance that contract provisions will be of benefit. A Request for Proposal (RFP) is a solicitation to potential suppliers to submit business proposals. Using an RFP process can help specify important contractual issues, and address them early in a competitive selection process, rather than during a protracted negotiation (Table 3). 2016 The PCPI Foundation. All Rights Reserved. Page 3 of 16

Table 3: Elements of a Request for Proposal document Element Description Introduction Explains the steps the prospective vender will need to take to respond. May include a scope and purpose statement as well as any submission deadlines. Use Cases A use case is a list of steps that define the interactions between the registry and the people, organizations and other health IT systems that interact with it 3. Use cases are a method for describing the primary operations of a registry, along with their supporting steps, to a level of detail that allows a prospective vendor to understand and respond how they can support. Contractual terms and conditions Additional Information Requests Measures or Standards Information Contract Additional Considerations Common use cases for a registry might include: Connecting the registry to health IT system from multiple organizations Registering a health care professional to submit data Submitting data through a secure electronic link from an EHR Submitting data through a secure website Providing feedback reports to submitters Providing data quality reports Querying the registry and reporting the results of the query Specific contractual terms and conditions, and a request that the vendor agree to these with specific language directly in the vendor s contract documents. Specific language about confidentiality may also be included if desired. Language could be added instructing recipients who have decided not to respond, to notify on or before the due date and to destroy received documents Long form questions for the vendor to: Explain its level of experience with clinical data registries Discuss overall how it intends to support the use cases Describe its unique technologies, methodologies, and resources not mentioned elsewhere to support the registry Discuss its current business and financial models, with an eye toward long term sustainability Describe cost structure, including anticipated costs to support the expected increasing user base size as the registry develops Describe measures and standards recommended or required for the vendor to support. Obtain a copy of the vendor s standard contract Additional content or sections that can be added to an RFP as needed: A request to explain pricing and payment terms An explanation of how vendors responses will be evaluated Contact information for questions Proposal submission instructions Summary implementation schedule, with a note that it is subject to 2016 The PCPI Foundation. All Rights Reserved. Page 4 of 16

See Appendix A for a sample RFP. Vendor Demonstrations change A statement that news releases, release of details or the use of organization name & trademark are not allowed without prior permission A statement that the RFP is a definition of requirements, not an offer to award business A statement that the RFP can be amended, modified, withdrawn, cancelled or terminated at any time, and how notification will take place should any of these occur. A statement that the RFP does not create or be construed to create a contractual relationship RFP respondents may be invited in for a live product demo. It is important to prepare for the demo and to take control to ensure that what is demonstrated is the most relevant, and that critical questions are answered. Checklist for ensuring a successful demo: Plan approximately two hours Bring a scorecard for note taking during the demo Think ahead of time about what the focus of the demo should be, and tell the vendor in advance. Work to actively control the content of the conversation and ensure appropriate time is spent on what matters Ask to see a demonstration of the software from the perspective of all stakeholders who will be using the software Ask if it is possible to keep a demo copy of the software, or retain login access to a demo system if available via the Web, to allow for further investigation and the generation of detailed follow up questions If practical provide de-identified data for use during the demo. One way to do this is to work with the vendor ahead of time to agree on a typical clinical scenario, and then have the vendor demo how to enter those data Arrange to retain access to the demo system for further experimentation and generation of additional detailed questions Note that demos are normally arranged after responses have been received from the RFP. However if it is desired initially to gain a better high level understanding, a more basic demo may be requested of vendors before responses have been received. 2016 The PCPI Foundation. All Rights Reserved. Page 5 of 16

Estimating Cost Registries face initial and then ongoing costs for the expected life of the registry. Cost has many complexities, but basic estimation is possible with a simple worksheet, filled out for each prospective vendor. Ongoing costs should be expected to grow as the registry develops and serves a larger population, storing greater amounts of data and performing more complex analysis and reporting on those data 4. Table 4 shows a list of typical cost categories and descriptions. Table 4: Cost Categories Cost Category Hardware & Software Hosting & Data Storage Registry Design Implementation Professional support Your own staff time Description These costs will apply if the registry is to be hosted in house. This category may also include the cost of additional workstations for data entry. As the registry develops, software costs may grow to include analytics applications as well as ongoing costs related to connecting the registry with other health IT systems. Ongoing hardware costs include maintenance and upgrades when greater storage capacity or processing power is needed. These costs apply if the vendor hosts the registry on their equipment. Expect this to grow as the registry develops. Ask prospective vendors to describe their cost structure for handling increased storage, number of submitters and data transaction volume. Costs for registry design work anywhere along the continuum from study question through implementation. Costs for bringing the registry up and running, as well as connecting it to other health IT systems Costs for technical and aftersales support plus training for before, during, and after go live. While the time local staff spends on the registry project may or may not be directly billable, it is valuable time taken away from other work so it is helpful to account for it here. In negotiating with vendors it is important to understand clearly how they expect to be paid (amount and timing) and to ensure that the contract spells out favorable payment terms explicitly. It is also useful to estimate the registry s expected growth. The contract should specify how costs will change with different levels of increased volume and/or scope. See Appendix B for a sample cost estimator worksheet. 2016 The PCPI Foundation. All Rights Reserved. Page 6 of 16

A Scorecard for Vendor Evaluation A scorecard is helpful in creating a comprehensive, prioritized evaluation of RFP responses. By developing rating categories in both technical and non-technical domains, and weighting each criterion according to its overall importance, vendors can be easily compared (Table 5). Table 5: Vendor Evaluation Criteria Criterion Description Experience with Determining the number of new registries that the vendor has implemented registries or supported in the past two years, as well as talking to references both from colleagues and those supplied by the vendor, can help with understanding a vendor s level of registry experience. Pricing Includes cost structure for initial and ongoing support, how costs change as the registry scales up, and payment terms. Professional support This includes training and technical and professional support before, during and after registry go-live. Look for vendors that are willing to train for selfsufficiency in terms of day to day operations and handling common issues. Legal, Ethical & The ability of the vendor to instill confidence that they will handle registry Oversight Data Quality Data Entry & Transmission Data Type, Measurement & Standardization See Appendix C for a sample scorecard. data safely. Capabilities the vendor can provide to help assure a high level of data quality at any time, both at submission and ongoing. What are their data quality reporting capabilities? Can they help with preventing data submission errors and notifying submitters of a problem they need to fix? Here the focus is on the capabilities, ease and cost of connecting the registry with other health IT systems. Possible connections can include EHRs from multiple vendors, specialty departmental IT systems, reference data sets, data systems from evaluating organizations and payers, and other registries. How will a prospective vendor help create these connections for better interoperability? Are they fluent in available interoperability standards to use? Evaluate vendor capabilities to handle the specific kinds of data the registry will be using, especially from the perspective of applicable data definitions, standardized data sets and standardized measure sets. 2016 The PCPI Foundation. All Rights Reserved. Page 7 of 16

Reporting Recommendations After reviewing and evaluating the vendors that have responded to the RFP, the results can then be presented with appropriate context and visual impact. Show scorecard results graphically as well as numerically. Visually emphasize how vendors did or did not meet the standards set in the RFP (Figure 1). Figure 1. A Sample Recommendations Report References 1. HIMSS, Chicago, IL, 2006; http://www.himss.org/files/himssorg/content/files/selectingemr_flyer2.pdf; accessed 12/12/13 2. Journal of Healthcare Information Management (HIMSS, Chicago, IL, Spring 2011); http://www.himss.org/files/himssorg/content/files/code%2093_negotiations%20when%20sel ecting%20a%20vendor_jhim.pdf; accessed 12/12/13 3. Alexander, Ian, and Ljerka Beus-Dukic. Discovering requirements: how to specify products and services. John Wiley & Sons, 2009. 4. Valancy, Jack. "How much will that EMR system really cost?." Family practice management 9.4 (2002): 57. 2016 The PCPI Foundation. All Rights Reserved. Page 8 of 16

Appendix A A Sample RFP for a Registry Vendor Request for Information and Preliminary Proposal for <ORGANIZATION NAME> Clinical Data Registry <DATE> DISCLAIMER: THIS DOCUMENT IS INTENDED TO GATHER INFORMATION ON POTENTIAL STRATEGIC PARTNERS FOR AN <ORGANIZATION NAME> CLINICAL DATA REGISTRY (CDR) AS PART OF THE DUE DILIGENCE PROCESS. A POTENTIAL PARTNER SHOULD ONLY SUBMIT A RESPONSE IF THEY ARE COMMITTED TO ENGAGE THE <ORGANIZATION NAME> IN THE CREATION OF A CDR. THIS DOCUMENT AND RESPONSE DOES NOT CONSTITUTE A CONTRACT BETWEEN THE <ORGANIZATION NAME>AND THE RESPONDER. THE <ORGANIZATION NAME>RESERVES THE RIGHT TO ENGAGE ONE, NONE, OR MULTIPLE RESPONDERS IN A FORMAL CONTRACTING PROCESS. THE <ORGANIZATION NAME>WILL USE THE SUBMITTED RESPONSE TO THIS REQUEST AS THE BASIS FOR SUCH CONTRACT. 2016 The PCPI Foundation. All Rights Reserved. Page 9 of 16

1 Introduction... 3 1.1 Explanation of document... 3 1.2 Scope & Purpose... 3 1.3 Process & Timeline... 3 2 Use Cases... 3 2.1 Use case: Population Registry and Reporting... 3 2.2 Use Case: Ad hoc query by <ORGANIZATION NAME>...5 3. Additional Information Requests... 6 4. Appendix A. <MEASURES INFO>... 7 2016 The PCPI Foundation. All Rights Reserved. Page 10 of 16

<ORGANIZATION NAME> Clinical Data Registry Request for Information and Preliminary Proposal January 3, 2014 1 Introduction 1.1 Explanation of document This document provides a list of specific information requests to be completed and returned to the <ORGANIZATION NAME> to inform its Clinical Data Registry initiative. At any time, the responder is invited to contact the <ORGANIZATION NAME> for further clarification or to answer any questions. To contact the <ORGANIZATION NAME>: <ORGANIZATION CONTACT INFO> The information submitted by the applicant will be considered confidential and not shared outside of the <ORGANIZATION NAME> and its <ORGANIZATION GOVERNING BODY>. 1.2 Scope & Purpose The scope of the <ORGANIZATION NAME> Clinical Data Registry is to accept patient data from practicing <ORGANIZATION SPECIALTY> on the care provided to patients with <INCLUSION CRITERIA> (see Appendix A). These data will inform the main goals of the <ORGANIZATION NAME> Clinical Data Registry, which are to: 1. Provide a unified method for <ORGANIZATION NAME> members to collect and submit Physician Quality Reporting System (PQRS) data and Maintenance of Certification (MOC) data to meet quality improvement and regulatory requirements. 2. Demonstrate the value of <ORGANIZATION SPECIALTY>. 3. Facilitate appropriate secondary uses of the aggregated data (e.g., research, benchmarking). 1.3 Process & Timeline Responses to this request for preliminary proposal are due by <DUE DATE AND TIME>. 2 Use Cases The following use cases present the core, high-level functionalities that must be supported by the <ORGANIZATION NAME> Clinical Data Registry in Phase One collecting and submitting MOC and PQRS data on behalf of users. The <ORGANIZATION NAME> recognizes that implementing a registry is a multi-phased project. Phase Two would include these functions as well as the capacity to benchmark performance, collect patient reported outcomes (e.g., pain control, quality of life, employment, social and recreational participation, caregiver QOL, etc.), and facilitate research on the efficacy and value of <ORGANIZATION SPECIALTY> care. 2.1 Use case: Population quality reporting to meet PQRS and MOC requirements This use case requires the vendor to a) register participants; b) accept clinical data from physicians on measures used by the CMS Physician s Quality Reporting System (PQRS) and submit these data to CMS; c) accept clinical data from physicians on measures used by the <MEASURE STEWARD ORGANIZATION> for Maintenance of Certification (MOC) and submit these data oversight organizations; and d) provide reports back to participants rating the quality of their care and actionable data to improve performance on these measures. Use Case key functions: Population quality reporting to meet PQRS and MOC requirements 2016 The PCPI Foundation. All Rights Reserved. Page 11 of 16

Core technical function Request for Information Performance Measures are Added to the Clinical Registry as identified Measures to include in registry are Describe plan to work with <ORGANIZATION identified. NAME> to refine performance measures that could be implemented in the registry. Discuss vendor capability to expand the number of measures and modules in the future. Discuss vendor capacity to include patient reported outcomes, now or in the future, directly from the patient Performance measure is converted to a formalized language. Performance measure is encoded into the registry into the registry. Responder would be responsible for encoding it into the registry for processing. Physician Signs up and Interacts with the Registry Practice/provider creates an account Describe process for individual registration and the vendor s capacity to manage several hundred to several thousand users. Practice adds providers to the account Discuss capabilities to report on an entire practice as well as on individual physicians. Describe the process to group providers and add providers to an existing account. Patient data are standardized across Describe how vendor will standardize and integrate data systems. Security and privacy disparate data sources (e.g. multiple EHRs, practice standards are in place. management systems, lab, and pharmacy systems). Discuss vendor capability to integrate additional data elements (i.e., patient demographics, severity ratings, patient reported outcomes) from other data sources. Describe ability to accept data from the top 10 EHR vendors and standardized data submitted to the Clinical Data Registry. Discuss HIPAA and security measures to protect patient and provider privacy, limiting access to the data to authorized individuals only, etc. Data are submitted to the registry Discuss the process physicians use to submit data Analysis is run to identify performance rates Data are submitted to CMS for PQRS participation and to <MOC VENDOR> for Maintenance of manually and by transferring data from their electronic health record and other electronic data sources. Describe the vendor s process for receiving data and any data quality and integrity processing conducted during and after data submission (validation testing and auditing). Discuss data analytics used to determine quality score. Describe process used transmit data to CMS and to upload data into <MOC SYSTEM>. 2016 The PCPI Foundation. All Rights Reserved. Page 12 of 16

Certification Describe operations required at the practice and provider level. Performance rate is reported to Describe how the vendor would report to physician physician and/or practice. and practice. Physician uses the report to improve Describe how their reports will enable physician to their performance metric identify groups of patients to target and links to quality improvement resources. 2.2 Use Case: Ad hoc query by <ORGANIZATION NAME> The use case allows the <ORGANIZATION NAME> to query the data for trends and care gaps. The <ORGANIZATION NAME> would use these data to direct resources to assist neurologists to improve quality and to promote public policy that facilitates the acceleration of this quality improvement. It also allows for secondary uses of the data by researchers and others. The <ORGANIZATION NAME> should be able to generate queries that result in both reports and exportable data sets. Use Case key functions: Ad-hoc query by <ORGANIZATION NAME> Core technical function Request for Information Ad hoc query by <ORGANIZATION NAME> <ORGANIZATION NAME> has Describe level of user technical skills required. centralized user interface to construct queries of the aggregate clinical data set A native query is created or a new query is Describe level of user technical skills to create created as a modification of an existing new queries. query Describe the ability to have a library of Query is analyzed for performance issues, trends <ORGANIZATION NAME> Access to Reports <ORGANIZATION NAME> reviews the performance metrics in aggregate for all participants in the registry and a report is generated on the statistics of the clinical data registry. saved queries. Describe how the system is able to analyze the query for performance characteristics and trends Discuss process <ORGANIZATION NAME> would use for querying and reporting data in aggregate. 2016 The PCPI Foundation. All Rights Reserved. Page 13 of 16

3. Additional Information Requests As part of your submission, please address each of the following points. 1. Provide your key contact s information for all communications about this proposal and the <ORGANIZATION NAME> registry initiative. 2. Provide a brief background of your company and experience in clinical data registries and working with professional associations. 3. Discuss how you can support Phase One Use Cases a. 2.1: Population quality reporting to meet PQRS and MOC requirements b. 2.2: Ad Hoc Query by <ORGANIZATION NAME> 4. Discuss how you can support Phase Two Use Cases a. Benchmarking (relative to national average and peers) b. Collecting Patient Reported Outcomes (on and off site data collection from patients or their care givers). 5. Describe your unique technologies, methodologies, and resources not mentioned elsewhere, to support the clinical data registry. 6. Discuss current business and financial models. 7. As part of the deployment of the clinical data registry, successively larger pilots are planned. Please discuss anticipated costs to support the following pilots. Including, vendor ability to host the clinical data registry and be responsible for scaling the hardware, software and technical assistance to support the registry. (Cost data submitted with this RFI will not be used to rank applicants, but rather provide the <ORGANIZATION NAME> and its <GOVERNING BODY> with an approximate cost range. A formal contracting process would be used to formalize any cost data.) a. 500 physicians b. 2,000 physicians c. 5,000 physicians d. 10,000 physicians 8. Please provide any additional information you believe is relevant to this application and this clinical data registry initiative. 4. Appendix A. <MEASURE DESCRIPTION> <TABLE OF MEASURES> 2016 The PCPI Foundation. All Rights Reserved. Page 14 of 16

Appendix B A Sample Cost Estimator Worksheet Initial Cost Unit Initial Description cost Quantity cost Year 1 Annual and five-year costs Years 2-5 5-yr cost Computer workstations $ - $ - Registry server hardware $ - $ - Hardware for data storage $ - $ - Hardware for data backup $ - $ - Fees for data hosting/storage $ - $ - Printers $ - $ - Network installation $ - $ - Maintenance charges $ - $ - License fees $ - $ - Interface fees $ - $ - TOTAL: System $ - $ - $ - $ - $ - $ - Design & customization fees $ - $ - Installation fees $ - $ - Training fees $ - $ - TOTAL: Implementation $ - $ - $ - $ - $ - $ - Project management $ - $ - Legal $ - $ - IT support $ - $ - Building & electrical $ - $ - Data entry/abstraction $ - $ - TOTAL: Professional Support $ - $ - $ - $ - TOTAL $ - $ - $ - $ - $ - $ - Total cost Five year cost $ - Net present value $ - Five-year cost, discount rate: 2% Initial Year 1 Year 2 Year 3 Year 4 Year 5 Date paid 1/1/2014 7/1/2014 7/1/2015 7/1/2016 7/1/2017 7/1/2018 Amount paid $ - $ - $ - $ - $ - $ - Appendix C A Sample Vendor Evaluation Scorecard 2016 The PCPI Foundation. All Rights Reserved. Page 15 of 16

Weight (1-3, 3=most imp.) Rating (1-5, 5=fully meets this requirement) Score (weight * rating) Total 0 0 0 0 0 0 Vendor Basics Number of installations in past two yrs. 0 0 0 Customer references check 0 0 0 Pricing 0 0 0 Payment terms 0 0 0 Training & tech support capability, cost 0 0 0 Registry Functionality Reporting functionality meets needs 0 0 0 Needed measures capability 0 0 0 Submit data to evaluating orgs (ie CMS) 0 0 0 System can scale to expected size 0 0 0 Legal, Ethical & Oversight Security of data transmission & storage 0 0 0 Data Quality Capabilities Data validation capabilities 0 0 0 Data quality reporting 0 0 0 Data Entry/Transmission Capabilities Data entry capabilities meet needs 0 0 0 Connects to needed EHRs/other health IT 0 0 0 Uses encryption standards 0 0 0 Uses classification standards (ie ICD-10) 0 0 0 Data Types Supported Supports required data types 0 0 0 Lab/imaging results 0 0 0 Product/device 0 0 0 Pharmaceutical 0 0 0 Patient reported 0 0 0 Charge 0 0 0 Reference data sets (ie Soc. Sec. index) 0 0 0 Vendor1 Vendor2 Vendor3 Vendor1 Vendor2 Vendor3 Notes/comments 2016 The PCPI Foundation. All Rights Reserved. Page 16 of 16