Integrating natural language processing and machine learning algorithms to categorize oncologic response in radiology reports

Transcription

1 Integrating natural language processing and machine learning algorithms to categorize oncologic response in radiology reports Po-Hao Chen, MD MBA Hanna Zafar, MD Tessa S. Cook, MD PhD

2 Roadmap Background Methods Results Discussion

3 Roadmap Background Methods Results Discussion

4 Background Structured reporting templates found to improve quality and consistency of radiology reporting. Most radiology reports remain unstructured.

5 Oncologic Imaging Key outputs in oncologic imaging reports Progression Stable Disease Improvement Resolution No Cancer

6 Process Natural Language Processing Machine Learning Impression: 1. No evidence of new or increasing metastatic disease in the abdomen or pelvis. 2. Unchanged size of hepatic metastases since prior exam. 3. Resolved pneumobilia in keeping with improving post-procedure changes. Stable Disease

7 Background Hypothesis: Predictive ability of classification algorithm for oncologic reporting depends on the combination of feature engineering and machine learning.

8 Roadmap Background Methods Results Discussion

9 Methods Our institutional structured reporting: new lesions and existing oncologic lesions on abdominal/pelvic CT and MRI Progression Improvement Stable disease Resolution/no cancer Interval development of new lesion(s) OR progression of existing lesions No interval development of a new lesion AND improvement of existing lesions No interval development of a new lesion AND stable appearance of existing lesions Absence of any new lesion AND either no previously documented cancer OR complete response.

10 Feature Engineering Stop Word Removal (SWR) (+ / -) Stemming (+ / -) N-gram Tokenization (N=1 5) Text-mining Term Frequency (TF) Term Frequency * Inverse Document Frequency (TF-IDF) 16-Bit Feature Hashing Filter-Based Feature Selection 500, 750, 1000, 1500, 2500

11 Machine Learning Algorithms Logistic Regression Bayes Machine Support Vector Machine Random Decision Forest Neural Network

12 Combination Data Split 70% training 30% validation Training Hyperparameter optimization 5-Fold Cross-validation Performance measured using F 1 measure and accuracy

13 Roadmap Background Methods Results Discussion

14 Results 9418 Oncologic CT/MRI 804 Excluded 8614 Included 2498 Progression 2132 Stable Disease 1184 Improvement 2800 Complete Response No Cancer

15 Highest F 1 -Score Combination Support vector machine TF-IDF With stop word removal With stemming Unigrams + Bigrams Top 750 tokens

16 ML Performance

17 Results

18 Results

19 Effect of Language Processing On F-Measure

20 Roadmap Background Methods Results Discussion

21 Discussion Classification accuracy is likely a function of the classification task, the NLP techniques, and the ML algorithm. Stop word removal and stemming generally improve the F- measures of all ML algorithms except for Bayes point machine. Unigrams + Bigrams outperforms other N-Gram constructs on most ML algorithms. Support vector machine, logistic regression, and Bayes machines have an optimal number of features Random decision forest and neural network performance do not degrade with increased number of features

22 Thank you! Cai T, Giannopoulos AA, Yu S, Kelil T, Ripley B, Kumamaru KK, et al. Natural Language Processing Technologies in Radiology Research and Clinical Applications. Radiogr Rev Publ Radiol Soc N Am Inc Feb;36(1): Yim W-W, Yetisgen M, Harris WP, Kwan SW. Natural Language Processing in Oncology: A Review. JAMA Oncol Jun 1;2(6): Kocbek S, Cavedon L, Martinez D, Bain C, Mac Manus C, Haffari G, et al. Text mining electronic hospital records to automatically classify admissions against disease: Measuring the impact of linking data sources. J Biomed Inform Oct 11; Zafar HM, Chadalavada SC, Kahn CE, Cook TS, Sloan CE, Lalevic D, et al. Code Abdomen: An Assessment Coding Scheme for Abdominal Imaging Findings Possibly Representing Cancer. J Am Coll Radiol JACR Sep;12(9): Lipton ZC, Elkan C, Naryanaswamy B. Optimal Thresholding of Classifiers to Maximize F1 Measure. Mach Learn Knowl Discov Databases Eur Conf ECML PKDD Proc ECML PKDD Conf. 2014;8725: Bennasar M, Hicks Y, Setchi R. Feature selection using Joint Mutual Information Maximisation. Expert Syst Appl Dec;42(22): Hripcsak G, Rothschild AS. Agreement, the f-measure, and reliability in information retrieval. J Am Med Inform Assoc JAMIA Jun;12(3): Tong W, Xie Q, Hong H, Shi L, Fang H, Perkins R, et al. Using decision forest to classify prostate cancer samples on the basis of SELDI-TOF MS data: assessing chance correlation and prediction confidence. Environ Health Perspect Nov;112(16): Liu X, Song M, Tao D, Liu Z, Zhang L, Chen C, et al. Random forest construction with robust semisupervised node splitting. IEEE Trans Image Process Publ IEEE Signal Process Soc Jan;24(1): Wang J, Zhang J, An Y, Lin H, Yang Z, Zhang Y, et al. Biomedical event trigger detection by dependency-based word embedding. BMC Med Genomics Aug 10;9 Suppl 2:45. Wei W, Marmor R, Singh S, Wang S, Demner-Fushman D, Kuo T-T, et al. Finding Related Publications: Extending the Set of Terms Used to Assess Article Similarity. AMIA Jt Summits Transl Sci Proc AMIA Jt Summits Transl Sci. 2016;2016: