KLAS EMR Toolkit AN INDEPENDENT REPORT FROM HEALTHCARE EXECUTIVES AND PROFESSIONALS MARCH 2009

Transcription

1 Non-Provider Price: $7,850 AN INDEPENDENT REPORT FROM HEALTHCARE EXECUTIVES AND PROFESSIONALS KLAS EMR Toolkit A Healthcare Provider Resource for Stimulus-Related Technology Decisions ACCURATE. HONEST. IMPARTIAL. MARCH

2 KLAS EMR Toolkit T A B L E O F C O N T E N T S PAGE SECTION 1 INTRODUCTION 5 ACUTE CARE EMR Cerner Millennium McKesson Horizon Eclipsys SCM Meditech C/S Epic EpicCare QuadraMed CPR GE Centricity Enterprise Siemens Soarian 15 ACUTE CARE EMR (COMMUNITY) Cerner Millennium McKesson Paragon CPSI Meditech C/S Healthland QuadraMed Affinity HMS Siemens MedSeries4 Keane InSight 26 AMBULATORY EMR (OVER 26 PHYSICIANS) Allscripts Enterprise Epic EpicCare Cerner Millennium GE Centricity eclinicalworks NextGen 34 AMBULATORY EMR (2-25 PHYSICIANS) Allscripts Professional Greenway CHARTCARE McKesson Practice Partner e-mds NextGen eclinicalworks Sage Intergy GE Centricity KLAS EMR Toolkit 2009 I 1

3 T A B L E O F C O N T E N T S PAGE SECTION 45 CLINICAL IMPLEMENTATION PRINCIPAL ACS Epic Cerner IBM CSC (FCG) McKesson Deloitte Consulting Perot Systems Eclipsys Siemens 57 ADDENDUM Additional KLAS Research KLAS Performance Evaluation About KLAS 2 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

4 Introduction KLAS EMR TOOLKIT: A HEALTHCARE PROVIDER RESOURCE FOR STIMULUS-RELATED TECHNOLOGY DECISIONS Figure 1: Where KLAS Data Comes From Respondent Titles A CHANGING CLIMATE While a host of questions remain unanswered about the impact of the recently signed American Recovery and Reinvestment Act (ARRA) of 2009 and its effect on healthcare IT (HIT) spending, many healthcare providers are already closely examining what the new stimulus package might mean to their facilities. Though some CIOs remain reticent to discuss the prospect of any definitive impact, others are looking at the package which includes more than $19 billion in potential funding for electronic medical record (EMR) systems and other technologies with marked optimism. Consider these comments from healthcare CIOs: I think the details are still to come on the package, but generally what we ve seen suggests a lot of support for our initiatives to get medication management, CPOE and eprescribing done as soon as possible. Dept Manager 27% Dept Director 23% CNO/Nurse 2% CEO/CFO/COO 6% CIO 8% CMO/Physician 5% IT Manager 19% IT Director 10% [The stimulus package] does accelerate our discussions about pushing out EMR to our affiliated doctors. We ve been perplexed by the fact that the major chunk of what s in the package is targeted more for groups or hospitals that haven t [implemented technology] as well as we have. Right now we are trying to find the difference between the real world and fantasy land. Assuming that money really does flow our way, we will accelerate a lot of items. At this time, we do not plan to change our IT strategy. It is too early to tell where this is going to go since there are too many terms that are still undefined, such as the meaningful use of IT. The ARRA is going to change how we look at [EMR implementation]. There is a minimum of $2.5 million on the table for our facility to move to EMR. It may help us move quicker, but we still need to get into the details. The carrot is certainly encouraging and the stick in is something we cannot afford as we already have a very thin margin. We are actively pursuing a new EMR for our inpatient and outpatient setting. We feel our timing is excellent, as the funding from the stimulus package becomes available to us. KLAS EMR Toolkit 2009 I 3

5 3309 Number of EMR evaluations in the KLAS database WHAT S THE REAL IMPACT While the impact of the ARRA varies by facility based on a number of factors, some CIOs have already used the stimulus act as a catalyst for renewed discussion regarding HIT implementation plans, with an eye toward expected reimbursement payments. Below is a sample of three hospitals from one organization that have used Medicare data and estimates from the National Association of Public Hospitals to project EMR usage incentives: Hospital A $ 5,100,000 Hospital B $ 3,300,000 Hospital C $ 7,500,000 Even with many unanswered questions, estimates of that magnitude are provoking conversation and debate among healthcare CIOs. PROCEEDING WITH CAUTION Despite the potential of increased funding for HIT investments, some in the industry are also warning CIOs against moving too quickly, without careful consideration to the long-term impact of those purchases. As John Glasser, CIO at Partners HealthCare System and chairman of the CIO SmartBrief Advisory Board, wrote recently: now is the time when we have to be thoughtful. We have to be focused, move quickly and be thoughtful. Providers have to be careful that they are not so focused on the money that they make poor vendor choices and slam in systems in a way that fails to make necessary process changes and meaningfully engage clinicians. With that thoughtful approach in mind, KLAS has created this EMR Toolkit to help providers make informed choices regarding stimulus-related technology investments. KLAS performance ratings for HIT products and services are based on the evaluations of actual healthcare customers, who answer questions such as would you buy the product again, has the vendor kept all of its promises, does the product have the needed functionality and many others. Taken together, the answers to these questions comprise an overall performance rating, which allows providers to quickly compare and contrast the performance of a wide range of healthcare product and service vendors. This KLAS EMR Toolkit focuses specifically on the performance of vendors who deliver EMR products, as well as those services firms that offer implementation expertise. The following pages include performance ratings and analyses for vendors in these key market segments: Acute Care EMR (>200 beds) Acute Care EMR (1-200 beds) Ambulatory EMR (>25 physicians) Ambulatory EMR (2-25 physicians) Clinical Implementation Principal For other questions, or to speak with a KLAS researcher, call or klasinfo@klasresearch.com. 4 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

6 Acute Care EMR Acute Care EMR KLAS EMR Toolkit 2009 I 5

7 Acute Care EMR Below is a composite view of the KLAS performance ratings for acute care EMR vendors* serving large hospitals (>200 beds) over the past three years. Individual vendor performance is broken out and analyzed on the subsequent pages. Three-Year Performance Trend Cerner Millennium PwrCht/PwrOrders/CareNet Eclipsys Sunrise Clinical Manager Epic EpicCare Inpatient GE Centricity Enterpise Clinicals McKesson Horizon Expert Orders/Doc. Meditech C/S Enterprise Medical Record QuadraMed CPR Siemens Soarian Clinicals *The vendors profiled in this KLAS EMR Toolkit are those that were fully rated and ranked as part of the 2008 Top 20 Best in KLAS Awards: Software and Professional Services report (December 2008). To see other vendors in this space for which KLAS has initial ratings, providers can access the KLAS online performance database by going to and filling out a KLAS evaluation. 6 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

8 Acute Care EMR Cerner Millennium PowerChart/PowerOrders/CareNet Cerner Corporation Kansas City, MO Percent of Cerner s Own Client Base Doing CPOE (U.S. hospitals over 200 beds) 26% A v e r a g e o f a l l v e n d o r s : 2 2 % Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 42% 36% Would you buy this product again? 91% 88% 85% Cerner places fourth behind McKesson, Epic and Siemens in number of new contracts, selling an average of 13 new hospitals annually for the past several years. Cerner continues to be one of two CPOE leaders - with solid progress in nursing and ICU though support, implementation challenges and complexity continue to frustrate customers. Cerner is now looking for international opportunities to sustain revenue growth like it did in the early clinical days in the United States. KLAS EMR Toolkit I 7

9 Eclipsys Sunrise Clinical Manager Eclipsys Corporation Atlanta, GA Percent of Eclipsys Own Client Base Doing CPOE (U.S. hospitals over 200 beds)* 51% A v e r a g e o f a l l v e n d o r s : 2 2 % *Includes both TDS and SCM, with most of the CPOE usage on SCM Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 35% 36% Would you buy this product again? 85% 90% 93% Eclipsys continues to surprise many in the HIT arena as its products continue to gain more traction and improve. The company continues to sell at a steady, although not breathtaking, pace; and customers comment on a renewed focus from senior management. A great deal is riding on Eclipsys ability to successfully deliver an integrated pharmacy solution to more complex sites this year. Clients will then focus on the ambulatory EMR, testing the limits on the products Eclipsys has built and acquired. 8 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

10 Acute Care EMR Epic EpicCare Inpatient Epic Systems Corporation Verona, WI Percent of Epic s Own Client Base Doing CPOE (U.S. hospitals over 200 beds) 28% A v e r a g e o f a l l v e n d o r s : 2 2 % Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 57% 54% Would you buy this product again? 100% 98% 97% Epic is getting attention from all corners of the market and has replaced products from nearly every other EMR vendor (from Meditech to Cerner and GE). Epic continues to sell and pick its customers, resulting in high profit, high profile clients moving its way. Performance scores have dropped slightly since Epic s initial jump onto the scene - as more and more EpicCare Inpatient clients go live and report their experiences to KLAS - but this vendor continues to score well above the competition in most areas. KLAS EMR Toolkit I 9

11 GE Centricity Enterprise Clinicals GE Healthcare South Burlington, VT Percent of GE s Own Client Base Doing CPOE (U.S. hospitals over 200 beds)* 31% A v e r a g e o f a l l v e n d o r s : 2 2 % *Includes a split between Centricity Enterprise and legacy solutions Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 50% 3% Would you buy this product again? 56% 59% 40% GE s partnership with Intermountain Healthcare (IHC) presents an incredible vision of how IT can help physicians practice and truly improve patient care. When the GE development at IHC is finished, the company will have a unique story to tell the market. Until then, however, it remains unknown whether existing customers will stay the course; many lack the patience to wait and wonder if GE can truly deliver on the vision. While GE s customer satisfaction has dropped steadily in recent years, over the past quarter, the company s performance scores have begun to improve. 10 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

12 Acute Care EMR McKesson Horizon Expert Orders/Documentation McKesson Alpharetta, GA Percent of McKesson s Own Client Base Doing CPOE (U.S. hospitals over 200 beds) 9% A v e r a g e o f a l l v e n d o r s : 2 2 % Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 36% 27% Would you buy this product again? 92% 92% 85% McKesson led all vendors in the number of contracts sold in 2007, and client satisfaction scores remain competitive. However, performance scores are down over time and McKesson has lost more clients than any other vendor in recent years because of integration issues and ongoing challenges with Horizon Ambulatory Care, Emergency Care and other challenging products. The company continues to focus on integration needs as they bring Horizon Expert Orders, Horizon Expert Documentation and Horizon Meds Manager more closely together in version KLAS EMR Toolkit I 11

13 Meditech C/S Enterprise Medical Record Meditech Westwood, MA Percent of Meditech s Own Client Base Doing CPOE (U.S. hospitals over 200 beds)* 10% A v e r a g e o f a l l v e n d o r s : 2 2 % *Includes currently marketed and legacy products Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 23% 35% Would you buy this product again? 93% 92% 80% Meditech continues to deliver at a low price point and loses few customers. Between the integration and the cost, it is difficult for a provider to leave Meditech, even when physician adoption is one of the lowest in the industry. Some serious changes are in store for Meditech clients as they consider a move to v.6. This will cause some to move forward because of the new user interface and will cause others to look at different options if v.6 does not solve the clinician adoption issues. 12 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

14 Acute Care EMR QuadraMed CPR QuadraMed Reston, VA Percent of QuadraMed s Own Client Base Doing CPOE (U.S. hospitals over 200 beds)* 36% A v e r a g e o f a l l v e n d o r s : 2 2 % *Most clients are doing CPOE with CPR Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 100% 36% Would you buy this product again? 79% 90% 72% QuadraMed is encouraging their existing customers to move to the new CPR platform, which has more robust clinical functionality than Affinity. This may be a win for Affinity customers who have the opportunity to move to a more robust clinical solution at a discount. The technology of Affinity and CPR is reported to be similar, allowing QuadraMed to tie the products more closely together than with just a standard interface. KLAS EMR Toolkit I 13

15 Siemens Soarian Clinicals Siemens Medical Solutions USA Malvern, PA Percent of Siemens Own Client Base Doing CPOE (U.S. hospitals over 200 beds)* 23% A v e r a g e o f a l l v e n d o r s : 2 2 % *Most CPOE is being done with Invision, with 3 organizations live with Soarian CPOE Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 0% 15% Would you buy this product again? 92% 93% 80% Though early delays of the Soarian product created a number of challenges for Siemens, actual deployment of Soarian is much broader than most would suspect, with over 30 organizations live on results review and over 20 live on nursing documentation. Clients rave about the user interface, the vision of Siemens and the potential with workflow. If Soarian continues to progress, KLAS expects a number of additional Invision clients to sign in the near future. Providers will know it is ready for prime time when users of other current EMR products, like Cerner, Eclipsys, Epic, McKesson and Meditech, move to Soarian. 14 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

16 Acute Care EMR (Community) KLAS EMR Toolkit I 15

17 Acute Care EMR (Community) Below is a composite view of the KLAS performance ratings for acute care EMR vendors* in the community hospital space (1-200 beds) over the past three years. Individual vendor performance is broken out and analyzed on the subsequent pages Cerner Millennium PwrCht/PwrOrders/CareNet CPSI Clinicals Healthland Clinicals HMS Clinicals Keane InSight Clinicals McKesson Paragon Clinicals Meditech C/S Enterprise Medical Record QuadraMed Affinity Clinicals Siemens MedSeries4 Clinicals *The vendors profiled in this KLAS EMR Toolkit are those that were fully rated and ranked as part of the 2008 Top 20 Best in KLAS Awards: Software and Professional Services report (December 2008). To see other vendors in this space for which KLAS has initial ratings, providers can access the KLAS online performance database by going to and filling out a KLAS evaluation. KLAS offers early ratings in this market for: Eclipsys Sunrise Clinical Manager M2 Emphasis McKesson Horizon Expert Orders/Documentation Meditech MAGIC Siemens Soarian Clinicals 16 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

18 Acute Care EMR (Community) Cerner Millennium PowerChart/PowerOrders/CareNet Cerner Corporation Kansas City, MO Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 14% 45% Would you buy this product again? 89% 74% 78% Cerner has created a version of its EMR product for community hospitals that can be implemented more quickly and hosted remotely. Though the product s clinical functionality is robust, it takes more IT muscle than most community hospitals have to run it. KLAS EMR Toolkit I 17

19 CPSI Clinicals Computer Programs and Systems, Inc. Mobile, AL Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 77% 48% Would you buy this product again? 75% 74% 77% CPSI has a significant number of new sites doing CPOE and barcoding at the bedside, leading most of the community hospital vendors. However, most of those sites are in a pilot phase and physicians have not been thrilled with the CPSI user interface, which some describe as antiquated and not easy to use. 18 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

20 Acute Care EMR (Community) Healthland Clinicals Healthland Glenwood, MN Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 21% 36% Would you buy this product again? 83% 89% 75% Healthland s primary focus is on critical access hospitals. The company s solution tends to be stronger on the financial side than the clinical, though the company is working to integrate and upgrade its clinical feature set. Healthland only has one site live with CPOE. KLAS EMR Toolkit I 19

21 HMS Clinicals Healthcare Management Systems Nashville, TN Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 100% 50% Would you buy this product again? 86% 71% 72% HMS s strength is traditionally in patient accounting, and the company is playing catch up when it comes to clinical functionality. Many clinical pieces exist but are immature or not widely deployed. HMS is starting to get some customers live on CPOE and barcoding at the bedside. 20 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

22 Acute Care EMR (Community) Keane InSight Clinicals Keane, Inc. Melville, NY Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 50% 17% Would you buy this product again? 56% 42% 28% Keane is writing a new version of its EMR software, but the company is also sunsetting a number of older products, so many customers are leaving for other clinical solutions. Only a quarter of Keane customers would buy its EMR again. KLAS EMR Toolkit I 21

23 McKesson Paragon Clinicals McKesson Alpharetta, GA Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 31% 30% Would you buy this product again? 100% 95% 88% McKesson has enjoyed a fairly dramatic turnaround in customer satisfaction in the community hospital EMR market in recent years, going from one of the worst-performing vendors to Best in KLAS for the community hospital space. Paragon s clinical functionality continues to expand as McKesson rolls out early versions of pharmacy and emergency department modules, with CPOE set to launch soon. More and more hospitals seem to be looking at Paragon in the last 12 months. 22 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

24 Acute Care EMR (Community) Meditech C/S Enterprise Medical Record Meditech Westwood, MA Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 33% 39% Would you buy this product again? 88% 91% 88% Meditech continues to deliver at a low price point and loses few customers. Between the integration and the cost, it is difficult for a provider to leave Meditech, and physician adoption in the community hospital market is just starting to gain traction. Some serious changes are in store for Meditech clients as they consider a move to v.6. KLAS EMR Toolkit I 23

25 QuadraMed Affinity Clinicals QuadraMed Reston, VA Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 47% 33% Would you buy this product again? 89% 86% 66% As QuadraMed encourages clients to move away from the Affinity product to CPR, it remains to be seen how the CPR platform - originally intended for larger hospital use - will map to the needs of community hospital providers. 24 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

26 Acute Care EMR (Community) Siemens MedSeries4 Clinicals Siemens Medical Solutions USA Malvern, PA Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data No Data 6.1 Does the product have all the functionality you need today? No Data No Data 43% Would you buy this product again? 96% 80% 71% The MedSeries 4 product was a strong financial product purchased by Siemens for use in community hospitals, but the company has seemingly invested few resources in the clinical side. A few clients are live with the new version of clinicals, but none are live yet with CPOE. Most sites are still using an older, less-robust version of the clinical software. KLAS EMR Toolkit I 25

27 Ambulatory EMR (Over 25 Physicians) 26 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

28 Ambulatory EMR (Over 25 Physicians) Ambulatory EMR (Over 25 Physicians) Below is a composite view of the KLAS performance ratings for ambulatory EMR vendors* serving larger physician practices (>25 physicians) over the past three years. Individual vendor performance is broken out and analyzed on the subsequent pages Allscripts Enterprise EHR Cerner Millennium PwrCht Office/PwrWorks EMR eclinicalworks EMR Epic EpicCare Ambulatory GE Centricity EMR NextGen EMR 60.0 *The vendors profiled in this KLAS EMR Toolkit are those that were fully rated and ranked as part of the 2008 Top 20 Best in KLAS Awards: Software and Professional Services report (December 2008). To see other vendors in this space for which KLAS has initial ratings, providers can access the KLAS online performance database by going to and filling out a KLAS evaluation. KLAS offers early ratings in this market for: Allscripts Misys EMR Allscripts Professional EHR McKesson Horizon Ambulatory Care MED3000 IC Chart LSS (Meditech) MPM Sage Intergy EHR KLAS EMR Toolkit I 27

29 Allscripts Enterprise EHR Allscripts Chicago, IL Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 50% 42% Would you buy this product again? 96% 87% 91% Allscripts is the most considered ambulatory EMR vendor overall, and one of the few that offers a solution across all practice sizes. However, within those market segments, Allscripts currently markets three different EMR products, which can be confusing for customers wondering which would best fit their practice. The company performs relatively well across most segments, though performance has been trending downward in recent years with the challenging release of v I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

30 Ambulatory EMR (Over 25 Physicians) Cerner Millennium PowerChart Office/PowerWorks EMR Cerner Corporation Kansas City, MO Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 17% 23% Would you buy this product again? 72% 62% 60% Cerner is often included in ambulatory EMR purchase discussions because of the company s large installed base on the inpatient side. As a sole-source vendor, this integration between inpatient and outpatient solutions is one of the company s greatest strengths. Cerner s performance ratings are generally on the lower end but have been trending upward recently, in part because of the company s emphasis on PowerWorks and the improved performance of its hosted solution. KLAS EMR Toolkit I 29

31 eclinicalworks EMR eclinicalworks LLC Westboro, MA Indicator Has the vendor lived up to expectations? No Data Rating for this vendor s product quality? No Data Quality of post-implementation support? No Data Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 38% 32% Would you buy this product again? 100% 87% 85% eclinicalworks enjoys high physician adoption with an EMR product that is considered cost-effective and easy to use. The product also scales to all practice sizes and works well at both ends of the spectrum. eclinicalworks has been growing at a rapid pace over the past four years, though that rapid growth has also led to struggles with support and implementation. Some providers question whether eclinicalworks has the resources to continue meeting customers needs as the company continues to grow so quickly. 30 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

32 Ambulatory EMR (Over 25 Physicians) Epic EpicCare Ambulatory Epic Systems Verona, WI Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 79% 50% Would you buy this product again? 100% 96% 98% Epic leads the ambulatory EMR market among large group practices (those with more than 100 physicians) and is by far the most considered vendor in that space. Providers who install Epic typically feel they are getting their money s worth, and the company performs very well when integrating inpatient and outpatient facilities. However, Epic is selective about which organizations they will engage, and smaller practices have to go through an IDN if they want Epic. KLAS EMR Toolkit I 31

33 GE Centricity EMR GE Healthcare South Burlington, VT Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 44% 35% Would you buy this product again? 94% 89% 82% GE is one of the most considered vendors in the ambulatory EMR market, but also the second highest on the list of vendors that providers would avoid. GE s traction is somewhat based on brand recognition and its perception as a safe choice. Physicians generally like the GE EMR, noting that it is easy to use, and the product scales to every practice size. Support for the GE solution has been problematic, however, and the technology is older with little innovation or development recently. 32 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

34 Ambulatory EMR (Over 25 Physicians) NextGen EMR NextGen Healthcare Information Systems, Inc. Horsham, PA Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 50% 39% Would you buy this product again? 86% 96% 80% NextGen EMR is the most considered product in the ambulatory EMR space. (Allscripts is the most-considered vendor, but that interest is divided among several products.) The NextGen EMR is a robust product with deep functionality, and one that is not easily replaced. However, NextGen has largely disappeared from smaller practices. The company has a relationship with Siemens on the inpatient side, but that integration is still unproven. KLAS EMR Toolkit I 33

35 Ambulatory EMR (2 25 Physicians) 34 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

36 Ambulatory EMR (2 25 Physicians) Ambulatory EMR (2 25 Physicians) Below is a composite view of the KLAS performance ratings for ambulatory EMR vendors* serving small- to mid-sized physician practices (2-25 physicians) over the past three years. Individual vendor performance is broken out and analyzed on the subsequent pages Allscripts Professional EHR CHARTCARE EMR e-mds Chart eclinicalworks EMR GE Centricity EMR Greenway Medical PrimeSuite Chart McKesson Practice Partner EMR NextGen EMR Sage Intergy EHR *The vendors profiled in this KLAS EMR Toolkit are those that were fully rated and ranked as part of the 2008 Top 20 Best in KLAS Awards: Software and Professional Services report (December 2008). To see other vendors in this space for which KLAS has initial ratings, providers can access the KLAS online performance database by going to and filling out a KLAS evaluation. KLAS offers early ratings in this market for: Allscripts Enterprise EHR Allscripts Misys EHR Eclipsys PeakPractice EMR HealthPort EMR imedica PRM EMR MedcomSoft Record PracticeOne e-medsys EHR Praxis EMR Amazing Charts EMR Eclipsys MediNotes e KLAS EMR Toolkit I 35

37 Allscripts Professional EHR Allscripts Chicago, IL Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 73% 63% Would you buy this product again? 97% 96% 93% Allscripts is the most considered ambulatory EMR vendor overall, and one of the few that offers a solution across all practice sizes. However, within those market segments, Allscripts currently markets three different EMR products, which can be confusing for customers wondering which would best fit their practice. The company performs relatively well across most segments, though performance has been trending downward in recent years with the challenging release of v I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

38 Ambulatory EMR (2 25 Physicians) CHARTCARE EMR CHARTCARE, Inc. New York, NY Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 75% 57% Would you buy this product again? 100% 90% 100% CHARTCARE offers a fairly robust EMR product with an average user interface that is targeted primarily at mid-sized physician practices. The company does a good job supporting its client base and rolling out new technology. The lack of an effective billing solution for the United States market (CHARTCARE s parent company is based in Canada) has hampered adoption of the EMR in the U.S. KLAS EMR Toolkit I 37

39 e-mds Chart e-mds Austin, TX Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 57% 70% Would you buy this product again? 94% 97% 99% e-mds offers an extremely physician-friendly EMR that has experienced reasonably strong growth in recent years. The company is among the top-tier of vendors when it comes to product functionality but has yet been unable to translate its success among small practices into significant large-practice deployments. The company s recent growth has also led to some provider concerns about the availability and timeliness of support resources. 38 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

40 Ambulatory EMR (2 25 Physicians) eclinicalworks EMR eclinicalworks LLC Westboro, MA Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 57% 55% Would you buy this product again? 97% 93% 91% eclinicalworks enjoys high physician adoption with an EMR product that is considered cost-effective and easy to use. The product also scales to all practice sizes and works well at both ends of the spectrum. eclinicalworks has been growing at a rapid pace over the past four years, though that rapid growth has also led to struggles with support and implementation. Some providers question whether eclinicalworks has the resources to continue meeting customers needs as the company continues to grow so quickly. KLAS EMR Toolkit I 39

41 GE Centricity EMR GE Healthcare South Burlington, VT Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 75% 43% Would you buy this product again? 98% 89% 80% GE is one of the most considered vendors in the ambulatory EMR market, but also the second highest on the list of vendors that providers would avoid. GE s traction is somewhat based on brand recognition and its perception as a safe choice. Physicians generally like the GE EMR, noting that it is easy to use, and the product scales to every practice size. Support for the GE solution has been problematic, however, and the technology is older with little innovation or development recently. 40 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

42 Ambulatory EMR (2 25 Physicians) Greenway Medical PrimeSuite Chart Greenway Medical Technologies Inc. Carrollton, GA Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 63% 61% Would you buy this product again? 96% 94% 95% Greenway is a consistent top performer in the KLAS ratings and is seen by many providers as a company with great vision and delivery. The Greenway product offers good functionality and is generally seen as a cost-effective investment, even though it is not inexpensive. Greenway is attentive to its customer base, offering great support and a strong development track record. One weakness of the Greenway EMR is its lack of integration with any inpatient solution. KLAS EMR Toolkit I 41

43 McKesson Practice Partner EMR McKesson Alpharetta, GA Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 73% 60% Would you buy this product again? 84% 86% 91% McKesson markets two products in the ambulatory EMR space: Practice Partner for smaller practices and Horizon Ambulatory Care (HAC) for larger. HAC purchases are coming primarily from McKesson sites that are looking to have an integrated inpatient-outpatient solution; and Practice Partner is supposed to offer integration with Paragon Clinicals, McKesson s EMR solution for community hospitals. For Practice Partner, however, providers have yet to see the level of integration that McKesson has promoted. Like GE, McKesson is also considered to be a safe purchase, and its products work relatively well with reasonably good support. 42 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

44 Ambulatory EMR (2 25 Physicians) NextGen EMR NextGen Healthcare Information Systems, Inc. Horsham, PA Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 71% 63% Would you buy this product again? 93% 91% 92% NextGen EMR is the most considered product in the ambulatory EMR space. (Allscripts is the most-considered vendor, but that interest is divided among several products.) The NextGen EMR is a robust product with deep functionality, and one that is not easily replaced. However, NextGen has largely disappeared from smaller practices (under five physicians) but still has good traction in larger practices. KLAS EMR Toolkit I 43

45 Sage Intergy EHR Sage Software Healthcare, Inc. Tampa, FL Indicator Has the vendor lived up to expectations? Rating for this vendor s product quality? Quality of post-implementation support? Quality and effectiveness of implementation services? How well does the product support integration and interoperability goals? No Data Does the product have all the functionality you need today? No Data 57% 45% Would you buy this product again? 86% 91% 92% The Sage EMR product offers good functionality and provider satisfaction is improving. A new executive team at Sage is bringing renewed energy and direction. Sage is currently a small- and medium-sized practice solution and is generally considered favorably by providers. Sage does not offer an inpatient solution or integration with an acute care offering, so it will take significant work before the company is considered at large practices. 44 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

46 Clinical Implementation Principal KLAS EMR Toolkit I 45

47 Clinical Implementation Principal Below is a composite view of the KLAS performance ratings for professional services firms* doing clinical implementations. Individual vendor performance is broken out and analyzed on the subsequent pages. One-Year Overall Performance Trend ACS Cerner CSC (FCG) Deloitte Consulting Eclipsys Epic IBM McKesson Perot Systems Siemens Jan 08 Feb 08 Mar 08 Apr 08 May 08 Jun 08 Jul 08 Aug 08 Sep 08 Oct 08 Nov 08 Dec 08 *The firms profiled in this KLAS EMR Toolkit are those that were fully rated and ranked as part of the 2008 Top 20 Best in KLAS Awards: Software and Professional Services report (December 2008). To see other vendors in this space for which KLAS has initial ratings, providers can access the KLAS online performance database by going to and filling out a KLAS evaluation. KLAS offers early ratings in this market for: Accenture Coastal Healthcare Consulting CTGHS Cumberland GE Global Works 46 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

48 Clinical Implementation Principal ACS ACS Healthcare Solutions Dearborn, MI One-Year Overall Performance Trend Jan 08 Feb 08 Mar 08 Apr 08 May 08 Jun 08 Jul 08 Aug 08 Sep 08 Oct 08 Nov 08 Dec 08 Indicator Has the vendor lived up to expectations? Have you received your money s worth? Would you buy this service again? 91% 91% Was the service delivered as sold in the proposal, sales presentation, demonstration and contract? Was the implementation staff knowledgeable of the product, industry and your needs? Considering everything, rate the overall satisfaction with this vendor and service? Would you recommend this service to a peer or close friend? 89% 82% ACS has experienced a lot of change since it acquired Superior Consulting. The company seems to have worked through a number of leadership changes with only a modest level of distraction. ACS is able to provide a wide range of resources that can perform highlevel work and do a lot of heavy lifting. Because ACS is not focused solely on healthcare, it is at the mercy of its own large organization and priorities at times. The company has strong PM and technical strength, but it is more focused on blocking and tackling than on strategic or advisory services. KLAS EMR Toolkit I 47

49 Cerner Cerner Corporation Kansas City, MO One-Year Overall Performance Trend Jan 08 Feb 08 Mar 08 Apr 08 May 08 Jun 08 Jul 08 Aug 08 Sep 08 Oct 08 Nov 08 Dec 08 Indicator Has the vendor lived up to expectations? Have you received your money s worth? Would you buy this service again? 100% 100% Was the service delivered as sold in the proposal, sales presentation, demonstration and contract? Was the implementation staff knowledgeable of the product, industry and your needs? Considering everything, rate the overall satisfaction with this vendor and service? Would you recommend this service to a peer or close friend? 88% 100% Cerner has a lot of experience implementing its own system. With a reproducible implementation methodology that focuses on workflow and process redesign, there is every reason to expect the noted improvements to continue. Client ratings are the highest they have been on Cerner s services. Resources, however, are a mix of experienced implementers with strong clinical backgrounds and young resources fresh out of school. Cerner s challenge is to get the balance right for every installation. 48 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

50 Clinical Implementation Principal CSC (FCG) Computer Sciences Corporation Healthcare Technologies Farmington Hills, MI One-Year Overall Performance Trend Jan 08 Feb 08 Mar 08 Apr 08 May 08 Jun 08 Jul 08 Aug 08 Sep 08 Oct 08 Nov 08 Dec 08 Indicator Has the vendor lived up to expectations? Have you received your money s worth? Would you buy this service again? 77% 77% Was the service delivered as sold in the proposal, sales presentation, demonstration and contract? Was the implementation staff knowledgeable of the product, industry and your needs? Considering everything, rate the overall satisfaction with this vendor and service? Would you recommend this service to a peer or close friend? 81% 73% The acquisition of FCG by CSC and the leadership changes that occurred prior to the acquisition have had an impact on this highly respected firm. Its scores have dropped by 10 points over the past year, and many have expressed concern and dissatisfaction regarding the merger and the loss of key talent. CSC is one of the top three options for consulting firms in terms of breadth and depth of capabilities. The company is experienced with a number of vendor applications and yet has a lot of clinical expertise. CSC s leadership appears to be committed to the healthcare provider space, but only time will tell. KLAS EMR Toolkit I 49

51 Deloitte Consulting Deloitte Consulting Cincinnati, OH One-Year Overall Performance Trend Jan 08 Feb 08 Mar 08 Apr 08 May 08 Jun 08 Jul 08 Aug 08 Sep 08 Oct 08 Nov 08 Dec 08 Indicator Has the vendor lived up to expectations? Have you received your money s worth? Would you buy this service again? 100% 100% Was the service delivered as sold in the proposal, sales presentation, demonstration and contract? Was the implementation staff knowledgeable of the product, industry and your needs? Considering everything, rate the overall satisfaction with this vendor and service? Would you recommend this service to a peer or close friend? 100% 100% Deloitte Consulting is the only third-party consulting firm that has not gone through some consolidation in the past four years. In fact, the company has been the beneficiary of all the consolidation, hiring a number of very talented resources from other major players in this space. The company s partner model appears to be working, and it gets high marks regarding executive involvement. Deloitte Consulting is expensive; however, many recognized that they got a lot of value for the money paid, which is reflected in the fact that 100 percent of clients would both recommend Deloitte Consulting to others and buy from the company again. 50 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

52 Clinical Implementation Principal Eclipsys Eclipsys Corporation Atlanta, GA One-Year Overall Performance Trend Jan 08 Feb 08 Mar 08 Apr 08 May 08 Jun 08 Jul 08 Aug 08 Sep 08 Oct 08 Nov 08 Dec 08 Indicator Has the vendor lived up to expectations? Have you received your money s worth? Would you buy this service again? 83% 83% Was the service delivered as sold in the proposal, sales presentation, demonstration and contract? Was the implementation staff knowledgeable of the product, industry and your needs? Considering everything, rate the overall satisfaction with this vendor and service? Would you recommend this service to a peer or close friend? 83% 84% Implementations continue to be a challenge for Eclipsys; however, those implementing the Eclipsys product will probably need to use Eclipsys for a lot of the work. Until noticeable improvements occur, external project management and project oversight should be considered. KLAS EMR Toolkit I 51

53 Epic Epic Systems Corporation Verona, WI One-Year Overall Performance Trend Jan 08 Feb 08 Mar 08 Apr 08 May 08 Jun 08 Jul 08 Aug 08 Sep 08 Oct 08 Nov 08 Dec 08 Indicator Has the vendor lived up to expectations? Have you received your money s worth? Would you buy this service again? 100% 100% Was the service delivered as sold in the proposal, sales presentation, demonstration and contract? Was the implementation staff knowledgeable of the product, industry and your needs? Considering everything, rate the overall satisfaction with this vendor and service? Would you recommend this service to a peer or close friend? 100% 92% Partner is the term most providers use to describe their relationship with Epic. The company continually gathers experience from previous implementations and fashions it into a roadmap for the provider to follow. Knowledge transfer is measured to ensure success. While communication is very structured, access to the CEO is always possible. 52 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

54 Clinical Implementation Principal IBM IBM Global Healthcare Industry Atlanta, GA One-Year Overall Performance Trend Jan 08 Feb 08 Mar 08 Apr 08 May 08 Jun 08 Jul 08 Aug 08 Sep 08 Oct 08 Nov 08 Dec 08 Indicator Has the vendor lived up to expectations? Have you received your money s worth? Would you buy this service again? 70% 70% Was the service delivered as sold in the proposal, sales presentation, demonstration and contract? Was the implementation staff knowledgeable of the product, industry and your needs? Considering everything, rate the overall satisfaction with this vendor and service? Would you recommend this service to a peer or close friend? 100% 75% This once top-performing consulting firm ranks dead last when it comes to client satisfaction. IBM is not as personable or flexible as Healthlink. There is more red tape, the prices are higher and the focus is different. While on paper this acquisition looked like a win-win deal, it has not turned out that way. While a few providers are happy, they are in the minority. It is now up to IBM to right this ship. KLAS EMR Toolkit I 53

55 McKesson McKesson Alpharetta, GA One-Year Overall Performance Trend Jan 08 Feb 08 Mar 08 Apr 08 May 08 Jun 08 Jul 08 Aug 08 Sep 08 Oct 08 Nov 08 Dec 08 Indicator Has the vendor lived up to expectations? Have you received your money s worth? Would you buy this service again? 78% 78% Was the service delivered as sold in the proposal, sales presentation, demonstration and contract? Was the implementation staff knowledgeable of the product, industry and your needs? Considering everything, rate the overall satisfaction with this vendor and service? Would you recommend this service to a peer or close friend? 80% 81% McKesson continues to do most of the McKesson implementation work in the industry. Many clients indicate that McKesson is easy to work with and does a good job at keeping projects on track, hitting deadlines and making its management available to resolve issues. That said, providers need to be wary of the resources McKesson has available and ensure up front that qualified, experienced McKesson staff will be working on the project when they need them. Otherwise, the quality decreases significantly and issues are not resolved in a timely manner. 54 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

56 Clinical Implementation Principal Perot Systems Perot Systems Corporation Plano, TX One-Year Overall Performance Trend Jan 08 Feb 08 Mar 08 Apr 08 May 08 Jun 08 Jul 08 Aug 08 Sep 08 Oct 08 Nov 08 Dec 08 Indicator Has the vendor lived up to expectations? Have you received your money s worth? Would you buy this service again? 80% 80% Was the service delivered as sold in the proposal, sales presentation, demonstration and contract? Was the implementation staff knowledgeable of the product, industry and your needs? Considering everything, rate the overall satisfaction with this vendor and service? Would you recommend this service to a peer or close friend? 100% 78% Even the market leader finds it difficult to fully meet clients expectations, which is an indication of just how difficult this is to do. Perot Systems sweet spot is in handling program management, step-by-step implementations, and the nuts and bolts of hardware, networking and configuration. The company shows weakness in providing the valued-added consultant industry expertise and insight; some providers want more than just a configuration company. KLAS EMR Toolkit I 55

57 Siemens Siemens Medical Solutions USA Malvern, PA One-Year Overall Performance Trend Jan 08 Feb 08 Mar 08 Apr 08 May 08 Jun 08 Jul 08 Aug 08 Sep 08 Oct 08 Nov 08 Dec 08 Indicator Has the vendor lived up to expectations? Have you received your money s worth? Would you buy this service again? 94% 94% Was the service delivered as sold in the proposal, sales presentation, demonstration and contract? Was the implementation staff knowledgeable of the product, industry and your needs? Considering everything, rate the overall satisfaction with this vendor and service? Would you recommend this service to a peer or close friend? 79% 100% Siemens is improving. Its bench strength is getting better, there is greater executive involvement, and the company is doing a better job of sharing best practices and properly setting client expectations. Siemens continues to struggle with project management, particularly in getting its product implemented on time, within budget and living up to expectations. There are signs that the lack of product knowledge is improving as the Soarian product matures. 56 I KLAS EMR Toolkit 2009 This material is copyrighted. Any organization gaining unauthorized access to this report will be liable to compensate KLAS for the retail price.

58 Additional KLAS Research KLAS Performance Evaluation About KLAS Addendum KLAS EMR Toolkit I 57