Capturing and Computer Reasoning with Quantitative and Semantic Information in Radiology Images

Size: px

Start display at page:

Download "Capturing and Computer Reasoning with Quantitative and Semantic Information in Radiology Images"

Moris Brooks
6 years ago
Views:

1 Capturing and Computer Reasoning with Quantitative and Semantic Information in Radiology s Daniel L. Rubin, MS, MD Assistant Professor of Radiology and Medicine Member, Cancer Center and Bio-X Stanford University Motivation: Data Deluge s in hospitals are exploding Clinical practice: Thousands per imaging study in clinical medicine Research: Hundreds per experiment, and hundreds of experiments per day on the Web are exploding s on Web pages s in image databases that are Webaccessible Physicians/scientists are challenged to use image data effectively Imaging Explosion CT Abdomen Pelvis Outline Motivating medical problem and approach Imaging ontologies Semantic image annotation and semantic statements based reasoning Outline Motivating medical problem and approach Imaging ontologies Semantic image annotation and semantic statements based reasoning Clinical Problem: Consider Cancer Cancer is a dynamic disease If treated well, tumor burden decreases over time If treatment fails, tumor increases over time Physicians use imaging to track the amount of cancer ( tumor burden ) to determine if it is responding to treatment ( treatment response ) Evaluation of repeated (serial) imaging is a key component of treatment response assessment Copyright 2011 Daniel L. Rubin 1

Disease changes over time C C c at t 1 c at t 2 Questions in temporal assessment of cancer Is an individual patient s disease responding to treatment ( response assessment )?

2 Disease changes over time C C c at t 1 c at t 2 Questions in temporal assessment of cancer Is an individual patient s disease responding to treatment ( response assessment )? Does the drug work in a cohort of patients ( treatment effectiveness assessment )? What is the best imaging biomarker for sensitive assessment of change in tumor burden in response to treatment? Cancer mass growing over time 1. Response assessment Assessment in individual patients Relies heavily on imaging to assess change in tumor burden Involves measuring an image biomarker on serial images and applying response criteria Response assessment Pre treatment imaging ( baseline ) Serial imaging post treatment (e.g., 6 week intervals) Identify and measure target lesions at each time point Measure the maximum linear dimension (for RECIST) Quantitative imaging biomarker: Sum of linear dimensions (SLD) L L3 L2 1.7 L3 1.0 RECIST Criteria Identify measurable disease (target and non target lesions Measure single longest dimension in each target lesion Calculate sum of longest dimension (SLD) for up to 10 target lesions RECIST CRITERIA SLD (Sum of the longest diameter) based on CR selection = complete response of target lesions 10mm, up to 5/organ and 10 total PR = partial response CR =Disappearance of all target lesions PD = progressive disease PR = 30% decrease in the SLD of target lesions SD = stable disease PD = 20% increase in the SLD of target lesions SD = Small changes not meeting above criteria Response assessment in single patients Sum of Maxim Lesion RECIST Score: Sum of Maximum Lesion Diameters Over Time 7/19/00 9/20/00 3/4/01 1/31/02 4/3/02 7/31/02 1/31/03 6/22/03 9/25/03 Study Date Tx PR CR PD SD CR PD PR biomarker = sum of the maximum length of each measurable lesion at each time point (SLD) Copyright 2011 Daniel L. Rubin 2

Treatment effectiveness assessment Assessment in cohort of patients Does the improve the imaging biomarker in a group of patients?

3 Questions in temporal assessment of cancer Is an individual patient s disease responding to treatment ( response assessment )? Does the drug work in a cohort of patients ( treatment effectiveness assessment )? What is the best imaging biomarker for sensitive assessment of change in tumor burden in response to treatment? Treatment effectiveness assessment Assessment in cohort of patients Does the improve the imaging biomarker in a group of patients? Involves measuring the best response in each patient and assessing in a waterfall plot Results depend on the imaging biomarker Patient response in patient cohorts: Is the drug working? Sum of Maxim Lesion Waterfal l Plot Sum of Maximum Lesion Diameters Over Time 7/19/00 9/20/00 3/4/01 1/31/02 4/3/02 7/31/02 1/31/03 6/22/03 9/25/03 Study Date Questions in temporal assessment of cancer Is an individual patient s disease responding to treatment ( response assessment )? Does the drug work in a cohort of patients ( treatment effectiveness assessment )? What is the best imaging biomarker for sensitive assessment of change in tumor burden in response to treatment? (each bar is a different patient) 3. Determining the best imaging biomarker There are different aspects of cancer that can be assessed with imaging Size of lesions Volume Metabolic activity Diffusion Etc The amount and sensitivity for detecting response depends on the imaging biomarker used This may differ in different types of cancer Treatment assessment depends on the imaging biomarker.. Drug is effective in about half of patients Drug is effective in MOST patients Exploration of alternative imaging biomarkers Copyright 2011 Daniel L. Rubin 3

Best imaging biomarker depends on disease DISEASE e.g., which image biomarker is best in cancer?

categorize and measure abnormalities on images Oncologists synthesize this information to determine the total tumor burden and whether it is changing GIST XX 1) Radiologist: Annotations and Reporting

each time point 2) Oncologist: Response Assessment & Oncologist reviews report & images Needs to locate lesions radiologist reports Needs to compare same lesions across studies Needs to assesses

4 Best imaging biomarker depends on disease DISEASE e.g., which image biomarker is best in cancer? WHO & Tumor PET SUV Disease RECIST Volume Mean 25-75% max NHL Panc CA Br CA XX IMAGE BIOMARKER XX DCE-MRI Ktrans RKtrans Upstroke XX XX XX DI-WI Evaluation of cancer in clinical workflow Radiologists categorize and measure abnormalities on images Oncologists synthesize this information to determine the total tumor burden and whether it is changing GIST XX 1) Radiologist: Annotations and Reporting annotations Mark & measure lesions Text Report Summarize observations Lesion Location Anatomic Description Number Lesion Dimension(s) Impression of disease status May not measure same/all lesions at each time point 2) Oncologist: Response Assessment & Oncologist reviews report & images Needs to locate lesions radiologist reports Needs to compare same lesions across studies Needs to assesses disease response based on lesion analysis Makes treatment decision based on response There is a Problem based criteria of disease response are complex Applying response criteria thwarted Laborious Error prone Variability in reporting the required data What we really want: Computing with image content Baseline 9/20/00 1/31/02 Copyright 2011 Daniel L. Rubin 4

Our goals Develop methods to make semantic and quantitative image contents explicit ( image metadata ) Develop tools for capturing image metadata during routine workflow Enable mining, integration,

extract image measurements 4 Decision support Viewer SLD RR RC CR PD PD SD 1 Automated Response Assessment Structured capture of measurements 3 Metadata 2 Analyze metadata at single timepoints Key

5 Our goals Develop methods to make semantic and quantitative image contents explicit ( image metadata ) Develop tools for capturing image metadata during routine workflow Enable mining, integration, and exchange of image metadata Automate response assessment and treatment effectiveness assessment Enable discovery of better imaging biomarkers Overview of Approach Data Disease Response viewer to extract image measurements 4 Decision support Viewer SLD RR RC CR PD PD SD 1 Automated Response Assessment Structured capture of measurements 3 Metadata 2 Analyze metadata at single timepoints Key Capabilities we Leverage Ontologies Specify meaningful distinctions in images (anatomic entities, observations, diseases) Formal relations and consistent semantics enables machine reasoning Semantic annotation Structure the inherently unstructured image content Enable image based reasoning Semantic Web Applications query and calculations Automated inference of treatment response Outline Motivating medical problem and approach Imaging ontologies Semantic image annotation and semantic statements based reasoning Ontologies Represent biomedical meaning ( semantics ) of entities and relations in the domain Human readable, computer accessible Enable computer reasoning with data (draw conclusions from known facts) RadLex An ontology for radiology research and practice Joint effort with professional organizations and standards bodies (RSNA, ACR, DICOM, CAP/SNOMED, subspecialty organizations) Imaging Methods Devices and acquisition parameters Will enable identification of similar studies Imaging Results Imaging indications, findings, anatomy, diagnoses Will enable semantic markup and data mining Copyright 2011 Daniel L. Rubin 5

Outline Motivating medical problem and approach Imaging ontologies Semantic image annotation

Anatomy Abnormalities and visual observations AKA semantic features Machine observations Pixel

images (e.g., pointing out lesions) Problem: metadata not explicit s intrinsically lack these

Annotation makes image meaning accessible to computers Current approach to recording image

5 cm in the right lobe of the liver, likely a metastasis.

(Synonyms, definitions, provenance) Radiology knowledge (Arterial supply to organs

Processing in image space Reasoning in knowledge space Dimension Information models are common

Biological and Biomedical Investigations (MIBBI) http://www.mibbi.org/index.

6 Outline Motivating medical problem and approach Imaging ontologies Semantic image annotation and semantic statements based reasoning contents = image metadata Radiologist observations Anatomy Abnormalities and visual observations AKA semantic features Machine observations Pixel based feature vectors Geometry and calculations markups Regions of interest (ROI) Labels on images (e.g., pointing out lesions) Problem: metadata not explicit s intrinsically lack these metadata information about their contents Machines cannot access/process them Semantic Annotation makes image meaning accessible to computers Current approach to recording image metadata There is a hypodense mass measuring 4.5 x 3.5 cm in the right lobe of the liver, likely a metastasis. Annotation Radiology Report Associating meaning with images Meaning Terminological knowledge (Synonyms, definitions, provenance) Radiology knowledge (Arterial supply to organs connectivity, typology of disease) Semantic Annotation Link semantics to regions of image Processing in image space Reasoning in knowledge space Dimension Information models are common in biology They are referred to as minimum information standards Minimum Information for Biological and Biomedical Investigations (MIBBI) MIAME (Minimum Information About a Microarray Experiment) MIASE (Minimum Information About a Simulation Experiment) MIGen (Minimum Information about a Genotyping Experiment) Copyright 2011 Daniel L. Rubin 6

Annotation and Markup (AIM) Markup ONTOLOGY BASED ANNOTATION ANDIMAGE MARKUP (AIM) Rubin DL, et.

Irregular mass in the right lobe of the liver, likely a metastasis.

image Annotation: Explanatory or descriptive information, generated by humans or machines, directly related to the

0) metadata are actually semantic statements about images Equipment User Calculation Results Finding Annotation

metastasis Probability = likely Radiology Report AIM encodes semantic statements about image content in XML

annotation) Referenced (DICOM_UID) AnatomicEntity (RadLex term) ImagingObservation (RadLex term)

7 Annotation and Markup (AIM) Markup ONTOLOGY BASED ANNOTATION ANDIMAGE MARKUP (AIM) Rubin DL, et. al: Medical Imaging on the Semantic Web: Annotation and Markup, AAAI Irregular mass in the right lobe of the liver, likely a metastasis. Annotation Markup: Graphical symbols associated with an image and optionally with one or more annotations of that same image Annotation: Explanatory or descriptive information, generated by humans or machines, directly related to the content of a referenced image AIM Information Model (v 3.0) metadata are actually semantic statements about images Equipment User Calculation Results Finding Annotation Annotation Of Annotation Annotation Role & References Inferences There is a hypodense mass measuring 4.5 x 3.5 cm in the right lobe of the liver, likely a metastasis. DICOM Web References Person Text Geometric Shapes (2D and 3D) Radiology Organ = liver Location = right lobe Diagnosis = metastasis Probability = likely Radiology Report AIM encodes semantic statements about image content in XML GeometricShape (line, x1, y1, x2, y2) ( ROI ) Name of ROI (string) Calculation (length of line) User (author of annotation) Referenced (DICOM_UID) AnatomicEntity (RadLex term) ImagingObservation (RadLex term) ImagingObservationCharacteristic(RadLex term) AIM incorporates ontologies To reduce ambiguity and to enable imagebased computer reasoning Represented as triple of (coding scheme, coding scheme designator, coded value) Coding scheme = name/identifier of ontology Coding scheme designator = identifier of ontology class Coded value = name corresponding to the identifier (RadLex, RID123, left lobe of liver ) Copyright 2011 Daniel L. Rubin 7

An An and A Markup Lung mass An, A Markup, and An Annotation Anatomic Entity:

Microlobulated margin (RID5712) Geometric Shape: Polyline 2D coordinates:

8 cm Annotations on lesions over time 2001 L 1 L 2 L1 L2 DICOM SR HL7 CDA XML

the arrow [coordinates (x,y)] in this [DICOM: 1.2.814.234543.

5 cm in the Right Lobe Text [SNOMED:A3310657] Report of the Liver

complex structure in a user-friendly annotation tool?

OsiriX open source workstation OsiriX provides Tools for visualizing and

8 An An and A Markup Lung mass An, A Markup, and An Annotation Anatomic Entity: Upper lobe of left lung (RID1327) Observation: Mass (RID3874) Characteristic: Microlobulated margin (RID5712) Geometric Shape: Polyline 2D coordinates: {(x,y), (x,y).} Calculation: Largest diameter result: 2.8 cm Annotations on lesions over time 2001 L 1 L 2 L1 L2 DICOM SR HL7 CDA XML 2003 L2 L1 L 1 L 2 Generating AIM Terminology Server The Pixel at the tip of the arrow [coordinates (x,y)] in this [DICOM: ] represents an Hypodense Mass [RID243, RID118] [2D measurement ] 4.5 x 3.5 cm in the Right Lobe Text [SNOMED:A ] Report of the Liver [SNOMED:A ] Likely [RID:392] a Metastasis [SNOMED:A ] There is a hypodense mass measuring 4.5 x 3.5 cm in the right lobe of the liver, How to create likely a this metastasis. complex structure in a user-friendly annotation tool? Semantic Annotation epad (electronic Physician Annotation Device) Plug in to OsiriX open source workstation OsiriX provides Tools for visualizing and annotating images Plug ins for image analysis ipad provides Template for collecting AIM compliant annotations Features for identifying and tracking lesions Automated assessment of treatment response Copyright 2011 Daniel L. Rubin 8

Feedback epad now a Rich Web Application

image Data Annotation Metadata See Lakeside

assessment of tumor burden BIMM: Searchable

9 ipad GUI in OsiriX Automated Lesion Summary/Tracking ROI Template Values Feedback epad now a Rich Web Application AIM/ePAD enable LINKING measurements to the image Data Annotation Metadata See Lakeside Computer Exhibit: LL-INE1190-WEA epad: A Cross-Platform Semantic Annotation Tool Software framework for quantitative imaging assessment of tumor burden BIMM: Searchable database of AIM and linked images Copyright 2011 Daniel L. Rubin 9

Outline Motivating medical problem and approach Imaging ontologies Semantic image annotation and semantic statements based reasoning The Semantic Web http://www.w3.

10 Outline Motivating medical problem and approach Imaging ontologies Semantic image annotation and semantic statements based reasoning The Semantic Web A new form of Web content that is meaningful to computers will unleash a revolution of new possibilities A web of resources that are machineunderstandable and accessible to automated processes. Moves the World Wide Web from a web of pages to a web of data, creating a global database Key technologies we use: RDF, OWL metadata are semantic statements about images Statement: Subject Predicate Object triples e.g.: DICOM UID ( ) Date (UID, 11/20/01) Modality (UID, CT ) Projection (UID, Axial ) 11/20/01 gs = GeometricShape (UID, line, x1, y1, x2, y2) AnatomicRegion (UID, gs, left lobe of liver ) Characteristics (UID, gs, [ hypodense, fuzzy margins, target lesion ]) Computer Reasoning with s AIM is translated to OWL to enable machine reasoning ( AIM Ontology ) Each image annotation is an instance/individual OWL assertions encode reasoning logic Automatic classifier processes AIM ontology to perform image based reasoning AIM Ontology annotations (AIMs) can be expressed as OWL individuals There is a hypodense mass measuring 4.5 cm in long axis in the right lobe of the liver, likely a metastasis. AIM: sopclassuid = hasimagingmodality = CT Abd/Pel SpatialCoordinate = [(x1,y1) (x2,y2)] CalculationData value = 4.5 cm LesionType = target lesion AnatomicEntity= right lobe of liver ImagingObservation = metastasis Annotations are instances from an ontology of image results (AIM ontology) Copyright 2011 Daniel L. Rubin 10

Find CT images that contain a cancer lesion in the liver PREFIX image: <http://protege.stanford.edu/rdf/image#> SELECT?annot FROM <http://protege.

?lesioncls rdfs:label "cancer lesion".?annot image:hasanatomiclocation?locationcls.?locationcls rdf_:part_of?livercls.?livercls rdfs:label "liver".

length) AS?SLD) FROM <http://protege.stanford.edu/ontologies/image.owl#> WHERE {?annot image:haslesiontype?lesiontype.?lesiontype rdfs:label target lesion".

length } AIM (Gets length of target lesions from image annotations) Annotations on image For example: Automatic disease classification Cancer response

it into OWL class declaration, restrictions, and class descriptions Apply automatic classifier Look for instance re classification Answer: The patient

11 Find CT images that contain a cancer lesion in the liver PREFIX image: < SELECT?annot FROM < WHERE {?annot image:haslesiontype?lesiontype.?lesiontype rdfs:subclassof?lesioncls.?lesioncls rdfs:label "cancer lesion".?annot image:hasanatomiclocation?locationcls.?locationcls rdf_:part_of?livercls.?livercls rdfs:label "liver". } AIM Annotation on image Find measurements of target lesions at each time point PREFIX image: < SELECT (SUM(?length) AS?SLD) FROM < WHERE {?annot image:haslesiontype?lesiontype.?lesiontype rdfs:label target lesion".?annot image:hasgeometricshapelength?length } AIM (Gets length of target lesions from image annotations) Annotations on image For example: Automatic disease classification Cancer response classes: CR, PR, PD, SD Start with English definition (Aristotelian) A PR is a RECIST classification which has a decrease in SLD greater then 30% Convert it into OWL class declaration, restrictions, and class descriptions Apply automatic classifier Look for instance re classification Answer: The patient has had partial response! A new, automated workflow OLD Workflow NEW Workflow Lesion ID Baseline Follow-up cm 1.2 cm cm 1.4 cm cm 1.0 cm SLD 6.5 cm 3.6 cm RR -44% Response PR Category Discovery and Decision Support Decision support Radiologist: Show lesions to be measured Oncologist: Automatically generate patient response graphs and cohort waterfall plots Discovery Mine annotated images to derive novel quantitative imaging biomarkers (e.g., volume, metabolic tumor burden, voxel histograms) Data mining to discovery imaging biomarkers more sensitive to response than RECIST Copyright 2011 Daniel L. Rubin 11

Discovery and Decision Support Decision support Radiologist: Show lesions to be measured Oncologist: Automatically generate patient response graphs and cohort waterfall plots Discovery

image reporting Objective image assessments at each time point Alerts to missing data; required assessments Decision Support: Oncologist assessment of patient treatment response

analysis of biomarkers response assessment ng

12 Discovery and Decision Support Decision support Radiologist: Show lesions to be measured Oncologist: Automatically generate patient response graphs and cohort waterfall plots Discovery Mine annotated images to derive novel quantitative imaging biomarkers (e.g., volume, metabolic tumor burden, voxel histograms) Data mining to discovery imaging biomarkers more sensitive to response than RECIST Decision support: Radiologist image reporting Objective image assessments at each time point Alerts to missing data; required assessments Decision Support: Oncologist assessment of patient treatment response Decision support: Patient response Baseline T1 T2 Automated lesion tracking Classification of lesions (measurable/non measurable) Calculation of quantitative imaging biomarkers Temporal analysis of biomarkers response assessment Discovery and Decision Support Decision support Radiologist: Show lesions to be measured Oncologist: Automatically generate patient response graphs and cohort waterfall plots Discovery Mine annotated images to derive novel quantitative imaging biomarkers (e.g., volume, metabolic tumor burden, voxel histograms) Data mining to discovery imaging biomarkers more sensitive to response than RECIST Discovering alternative imaging biomarkers from historical data Baseline T2 RECIST linear measurements AIM Store T1 Novel Quantitative Biomarkers Prior linear measurements as seed to automatic segmentation and volume estimation Copyright 2011 Daniel L. Rubin 12

Comparing alternative imaging biomarkers Discovery: Alternate biomarkers for drug effectiveness Role for qualifying novel

biomarker? Exploratory data mining for discovery e.g., which image biomarker is best in cancer?

information in images This may improve the medical utility of imaging in healthcare, specifically improving cancer treatment

13 Comparing alternative imaging biomarkers Discovery: Alternate biomarkers for drug effectiveness Role for qualifying novel quantitative imaging biomarkers Cross-sectional area shows response earlier than RECIST better quantitative imaging biomarker? Exploratory data mining for discovery e.g., which image biomarker is best in cancer? Conclusions s contain rich unstructured information Semantic Web methods enable automated inference about semantic information in images This may improve the medical utility of imaging in healthcare, specifically improving cancer treatment response assessment contents are now machineaccessible Acknowledgements Funding NCI QIN U01CA NCI cabig In vivo Imaging Workspace Copyright 2011 Daniel L. Rubin 13

14 AIM resources CDE Browser UML Model Browser AIM 1.0 & 2.0 standard C++ library AIM 1.0 & 2.0 referenced implementation ANIVATR Convert between AIM XML and AIM DICOM SR AIM 1.0 & 2.0 official XML schemas Thank you. Contact info: Copyright 2011 Daniel L. Rubin 14

QIN: Overview of Scientific Challenges

QIN: Overview of Scientific Challenges Goal: Robust methods for Imaging (QI) as a Biomarker for Response to Therapy RIDER: Public Resource to promote QI Methods and Standards QIN: Quantitative Imaging