Error rates in breast imaging reports generated with automatic speech recognition technology versus traditional transcriptionist-generated reports

Transcription

1 Error rates in breast imaging reports generated with automatic speech recognition technology versus traditional transcriptionist-generated reports Poster No.: B-866 Congress: ECR 2011 Type: Scientific Paper Topic: Computer Applications Authors: A. M. Scaranelo, S. Basma, B. Lord, L. M. Jacks, M. Rizki; Toronto, ON/CA Keywords: Computer applications, Management, Breast, RIS, Neural networks, Technology assessment, Audit and standards DOI: /ecr2011/B-866 Any information contained in this pdf file is automatically generated from digital material submitted to EPOS by third parties in the form of scientific presentations. References to any names, marks, products, or services of third parties or hypertext links to thirdparty sites or information are provided solely as a convenience to you and do not in any way constitute or imply ECR's endorsement, sponsorship or recommendation of the third party, information, product or service. ECR is not responsible for the content of these pages and does not make any representations regarding the content or accuracy of material in this file. As per copyright regulations, any unauthorised use of the material or parts thereof as well as commercial reproduction or multiple distribution by any traditional or electronically based reproduction/publication method ist strictly prohibited. You agree to defend, indemnify, and hold ECR harmless from and against any and all claims, damages, costs, and expenses, including attorneys' fees, arising from or related to your use of these pages. Please note: Links to movies, ppt slideshows and any other multimedia files are not available in the pdf version of presentations. Page 1 of 16

2 Purpose 1. Breast imaging (BI) reports have more standard (MQSA, BI-RADS) criteria than other sub areas in Medical Imaging. 2. The literature reviewed showed increased error rate in reports using digital voice recognition software, however little is known about BI reports. 3. The purpose of this study it to compare the error rate of breast imaging reports generated with automatic speech recognition (ASR) versus conventional dictation transcriptionist service (CDT). Methods and Materials Institutional ethical approval was obtained for this study. A retrospective review of breast imaging reports was felt to be the optimal assessment method for this study. It was believed that this would exclude bias introduced by the knowledge that the reports would be scrutinized for errors. Data Selection: Breast imaging reports from January 2009 to April 2010 for patients presented at consecutive weekly, multidisciplinary, tumour-board rounds at 2 separate hospitals were included in this study. The data collected included reports prior to implementation of a voice recognition system in one hospital and post implementation of a voice recognition system in the other hospital, both tertiary hospitals with busy breast clinic (over 250 breast patients a day, for screening and diagnostic multi-modalities). The training period for voice recognition end-users to become comfortable with the new technology was not rd included in the study. A total of 14 staff members, 8 fellows and 11 residents (3 -year radiology residents or post-graduate year 4 residents) were included in the analysis. Imaging reports included in the study were of: mammograms, breast ultrasounds, breast MRIs, interventional breast procedures (ultrasound-guided large core and fine needle biopsies, pre-operative needle localizations, MRI-guided core biopsies, stereotacticguided core biopsies, ductograms, and clip placements), consultation reports and specimen radiographs. Most of the breasts imaging reports were diagnostic exams for patients with suspected or proven breast cancer. Overall, the reports included in this study were of a higher complexity compared to screening exams reports, which are usually generated using report templates. Page 2 of 16

3 Dictation Method and work-flow: The voice recognition software used was Nuance Speech Magic, version 6.1, service pack 2. Reports were verified and signed off by the author as they were generated. If the speaker was a fellow or a resident, a staff member was responsible for reviewing the case prior to dictation of the report. Dictation was completed via a handheld Philips speech microphone (Figure 1 on page 4). Conventional dictation transcription was undertaken using a standard dictaphone system. Transcription was completed by transcriptionists experienced in breast imaging reporting. Once reports were transcribed, they were sent to the original speaker for electronic amendment and verification (Figure 2 on page 4). Reports dictated by attending radiologists or trainees were reviewed on the radiology information system (RIS) at an electronic picture archiving and communication systems (PACS) workstation, corrected for errors, and verified, making these reports immediately available on the hospital clinical information system. The speaker assumed complete responsibility for report production, including correcting typographic errors generated by the voice recognition software or the transcriptionist. Data Collection 615 reports were scrutinized for errors: 308 reports generated with ASR (data from the hospital where ASR has been used for 2 years) and 307 reports generated with CDT (data from the hospital which still relies on transcriptionists for report generation). Reports were reviewed for the presence of errors and the errors were classified into 12 types (Figure 3 on page 5). Reports containing an error were reviewed independently by two observers in order to determine the errors' impact: significant or minor. Significant errors: Errors that had an impact on the understanding of the report (Figure 4 on page 5) or may have had impacting patient management (Figure 5 on page 6) were considered significant errors. Minor errors: Errors without impact on report understanding or patient management were labelled as "minor errors" (Figure 6 on page 6). When a disagreement on the impact of the errors was found between the two observers, the errors were re-evaluated case by case until a consensus was reached. Page 3 of 16

4 Structured and non structured reports by speaker and language The section of the report (i.e. clinical information, comparison exams, findings, impression) where the error was located as well as the academic ranking of the speakers (faculty or staff members, fellows and resident) and the speakers' native language (English or non English) were recorded. Unstructured reports (free text) were not excluded from the study. Figure 7 on page 7 shows a flow chart of the overall Methods and Materials in this study. Data Statistical Analysis 1. Logistic regression analysis: for clustering of reports within speaker. 2. Multivariable logistic regression analysis: to determine independent predictors of the occurrence of errors in reports. Images for this section: Fig. 1: ASR work-flow. Page 4 of 16

5 Fig. 2: CDT work-flow. Fig. 3: Classification of error types. Page 5 of 16

6 Fig. 4: Significant error example. Fig. 5: Significant error example. Page 6 of 16

7 Fig. 6: Minor error example. Fig. 7: Methods and Materials flow chart. Page 7 of 16

8 Results The distribution ( on page 9Figure 1 on page 9) of modality, speaker's academic ranking, and speaker native language, were all statistically different between the two groups (p<0.0001). ASR reports were more commonly dictated by faculty compared to CDT reports (88% vs. 69%) and ASR reports were more often dictated by speakers with English as their first language compared to CDT reports (33% vs. 16%). A total of 33 speakers dictated the 615 reports, with 11 speakers dictating both ASR and CDT reports. The Figure 2 on page 11 shows the distribution of total number of errors found in the 308 reports generated with ASR and the 307 reports generated with CDT. Reports generated with ASR were also more likely to contain at least one "significant error" compared to reports generated with CDT (23% vs. 4%, p<0.01). A total of 230 errors were found in 159 ASR reports. The most common error types were: 1- Added Word (N= 46, 20% of total errors); 2- Word Omission (N= 43, 19% of total errors); 3- Word Substitution (N=39, 17% of total errors); 4- Punctuation Error (N=49, 21% of total errors). A total of 77 errors were found in 68 CDT reports. The most common error types were: 1- Word Substitution (N=15, 19% of total errors) ; 2- Word Omission (N=13, 17% of total errors); 3- Added Word (N=11, 14% of total errors); 4- Punctuation errors (N=14, 18% of total errors). Error rates differed significantly by modality for reports generated with ASR (Figure 3 on page 11). Significant errors found with ASR were more common in MRI (35%) and combined mammography and US reports (31%), with the lowest error rates found in Page 8 of 16

9 interventional procedure and mammography reports (p<0.01). No substantial difference in significant error rates was seen between reports generated by staff and reports generated by resident/fellows using either ASR or CDT. Minor errors were more common in ASR reports dictated by staff compared to reports dictated by residents or fellows (37% vs. 22%, p<0.01). A multivariable analysis was used to determine independent predictors of the occurrence of errors in reports (Figure 4 on page 12). After adjusting for the speaker's academic ranking, native language, and for the modality, the reports generated with ASR were more than twice as likely to contain minor errors (adjusted odds ratio= 2.2, p<0.01) and more than 8 times as likely to contain significant errors (adjusted odds ratio= 8.4, p<0.01) than reports generated with CDT. The academic ranking and the native language of the speakers were not found to be independent predictors of the occurrence of significant errors. By contrast, modality was found to be an independent predictor of the occurrence of significant errors (p<0.01); MRI and Mammography plus Ultrasound (combined) reports were more likely to contain a significant error (odds ratio = 4.4 and 3.4 respectively) when compared to mammography reports alone. Images for this section: Page 9 of 16

10 Page 10 of 16

11 Fig. 1: Population sample showing the distribution of BI report modality, category of speaker (academic position and native language) and number of reports per radiologist by automatic speech recognition (ASR) software and conventional dictation transcriptionist service (CDT). Fig. 2: Out of the 308 reports generated with ASR, 159 reports (52%) contained at least one error whereas only 68 (22%) of the 307 reports generated with CDT contained at least one error (p Page 11 of 16

12 Fig. 3: Distribution of error rates (significant and minor errors) related to breast imaging report modality Page 12 of 16

13 Fig. 4: Multivariable analysis to determine independent predictors of the occurrence of errors. Page 13 of 16

14 Conclusion LIMITATIONS: 1. ASR has been used for 24 months & CDT has been used for a decade in our institution. 2. Complex breast cases scenario (MDTM): Bias of selection excluding screening exams (template) 3. Academic medical environment with several levels of trainees, so it is difficult to extrapolate the findings to private practice groups. CONCLUSIONS: 1. Breast imaging reports generated with ASR are 8 times more likely to contain significant errors than those generated with CDT, after adjusting for native language, academic ranking of the speaker, and breast imaging modality. 2. Careful editing of reports generated using ASR is crucial in order to minimize errors rates in complex cases breast imaging reports. References 1. Robbins AH, Horowitz DM, Srinivasan MK, Vincent ME, Shaffer K, Sadowsky NL et al. Speech-controlled generation of radiology reports. Radiology Aug;164(2): Gale B, Safriel Y, Lukban A, Kalowitz J, Fleischer J, Gordon D. Radiology report production times: voice recognition vs. transcription. Radiol Manage MarApr;23(2): Sferrella SM. Success with voice recognition. Radiol Manage MayJun;25(3): Marquez LO. Improving medical imaging report turnaround times. Radiol Manage Jan-Feb;27(1): Krishnaraj A, Lee JK, Laws SA, Crawford TJ. Voice recognition software: effect on radiology report turnaround time at an academic medical center. AJR Am J Roentgenol Jul;195(1): Page 14 of 16

15 6. Kanal KM, Hangiandreou NJ, Sykes AM, Eklund HE, Araoz PA, Leon JA et al. Initial evaluation of a continuous speech recognition program for radiology. J Digit Imaging Mar;14(1): Quint LE, Quint DJ, Myles JD. Frequency and spectrum of errors in final radiology reports generated with automatic speech recognition technology. J Am Coll Radiol Dec;5(12): Zeger SL, Liang KY. Longitudinal data analysis for discrete and continuous outcomes. Biometrics 1986; 42: McGurk S, Brauer K, Macfarlane TV, Duncan KA. The effect of voice recognition software on comparative error rates in radiology reports. Br J Radiol Oct;81(970): Rana DS, Hurst G, Shepstone L, Pilling J, Cockburn J, Crawford M. Voice recognition for radiology reporting: is it good enough? Clin Radiol Nov;60(11): White KS. Speech recognition implementation in radiology. Pediatr Radiol Sep;35(9): White GM. Speech recognition: a tutorial overview. IEEE Computer, 1976; 9: American College of Radiology. Breast Imaging Reporting and Data System (BIRADS). 3rd ed. Reston, Va: American College of Radiology, Reiner BI, Knight N, Siegel EL. Radiology reporting, past, present, and future: the radiologist's perspective. J Am Coll Radiol May;4(5): Personal Information Anabel Medeiros Scaranelo, MD, PhD. Breast Imaging Division, Department of Medical Imaging, Princess Margaret Hospital, University of Toronto, Toronto, ON. Canada. Page 15 of 16

16 Phone: ext 5468 Fax: or Princess Margaret Hospital 610 University Avenue, Rm Toronto, ON. Canada M5G 2M9 Page 16 of 16