M1741 Validating a Whole Slide Imaging System A Case Based Approach to the CAP Guidelines

Transcription

1 M1741 Validating a Whole Slide Imaging System A Case Based Approach to the CAP Guidelines Brent T. Tan, MD, PhD, FCAP

2 Objectives Design a validation study for a whole slide imaging system that meets CAP guidelines Determine the human resources required for a validation study Calculate the concordance rate between standard microscopy of glass slides versus whole slide scanned images Determine when a whole slide imaging system needs to be revalidated 2

3 Agenda Regulations relevant to WSI FDA CAP Designing a validation study Sample size and composition Study design and execution Calculating concordance rates When revalidation or additional validation is necessary Summary and Closing 3

4 16 pathologists at 4 study sites (Cleveland Clinic, University of Virginia, Miraca Life Sciences and Advanced Pathology Associates) performed ~16,000 reads over ~2,000 surgical pathology cases. Digital diagnoses were comparable to those made on glass. Since April 2017, only one manufacturer has been FDA-approved for primary diagnosis The approval process for other manufacturers will be easier. Without FDA approval, manufacturers can only market/advertise WSI systems as Research Use Only (RUO). Some manufacturers have received FDA 510(k) clearances for specific applications such as HER2/neu, ER/PR, Ki-67, and p53 immunostains on a computer monitor 4

5 College of American Pathologists (CAP) Digital image analysis section of anatomic pathology checklist Whole slide imaging section of the laboratory general checklist The CAP released 12 guideline statements regarding validation of WSI systems in For FDA-approved or non-fda approved instruments, a laboratory must verify/validate its performance prior to clinical use. 5

6 CAP guidelines 1. All pathology laboratories implementing WSI technology for clinical diagnostic purposes should carry out their own validation studies. 2. Validation should be appropriate for and applicable to the intended clinical use and clinical setting of the application in which WSI will be employed. Validation of WSI systems should involve specimen preparation types relevant to the intended use (eg, formalinfixed paraffin-embedded tissue, frozen tissue, cytology). 3. The validation study should closely emulate the real-world clinical environment in which the technology will be used. 4. The validation study should encompass the entire WSI system. 5. Revalidation is required whenever a significant change is made to any component of the WSI system. 6. A pathologist(s) adequately trained to use the WSI system must be involved in the validation process. 7. The validation process should include a sample set of at least 60 cases for one application (eg, H&E stained sections of fixed tissue, frozen sections, cytology, hematology) that reflects the spectrum and complexity of specimen types and diagnoses likely to be encountered during routine practice. 8. The validation study should establish diagnostic concordance between digital and glass slides for the same observer (ie, intraobserver variability). 9. Digital and glass slides can be evaluated in random or nonrandom order (as to which is examined first and second) during the validation process. 10. A washout period of at least 2 weeks should occur between viewing digital and glass slides. 11. The validation process should confirm that all of the material present on a glass slide to be scanned is included in the digital image. 12. Documentation should be maintained recording the method, measurements, and final approval of validation for the WSI system to be used in the clinical laboratory 6

7 Specimen Volume The OZpath practice 2500 hearts, 2500 neuropathology, 2500 lungs Locations OZ general hospital (includes ORs with intraoperative frozen sections) Emerald city (clinical laboratory and histology laboratory) Personnel 7

8 OZpath pathologists Years in practice Digital Pathology Experience Specialty Locations Dorothy 10 Extensive Generalist Kansas/OZ Scarecrow 3 None Neuropath OZ Tin Woodman 2 Moderate Hearts OZ Cowardly Lion 5 None Lungs Emerald city The Wizard 113 Extensive Generalist Emerald City Issues: 1. Pathologists no longer want to travel to OZ general hospital during off hours for frozen sections. 2. The wizard is getting old, often not wanting to leave Emerald City, but wants to be able to be consulted remotely. 8

9 Intended uses for WSI at OZpath OZ general hospital Emerald city 1. Viewing frozen sections (for surgeries at OZ general hospital) from Emerald City or Kansas 2. Showing routine cases to pathologists when they are offsite 9

10 Is validation necessary? Cowardly Lion: I m scared of WSI. We should probably validate it. Dorothy: I know were not in Kansas anymore, but I think there are recommendations for validation CAP #1: All pathology laboratories implementing WSI technology for clinical diagnostic purposes should carry out their own validation studies. The laboratory should validate and document the performance of its WSI system in its own specific laboratory environment prior to clinical use. 10

11 Inter- vs. intra-observer variability Scarecrow: What if we take 100 random cases and scan them. We can compare one pathologist s diagnosis on WSI to the Wizard s diagnosis on glass. After all, the Wizard should be the gold standard Years in practice Dorothy 10 Scarecrow 3 Tin Woodman 2 Cowardly Lion The Wizard Comparing the diagnosis made by one pathologist versus a different pathologist introduces an element of interobserver variability. CAP #8: The validation study should establish diagnostic concordance between digital and glass slides for the same observer (ie, intraobserver variability). 11

12 Order of viewing Wizard: Lets take random 100 cases that I ve already diagnosed on glass, and I ll re-review them with WSI. CAP #9: Digital and glass slides can be evaluated in random or nonrandom order (as to which is examined first and second) during the validation process. The order of viewing virtual versus glass slides has been shown in one study not to have had any effect on interpretation. 12

13 Sample size Cowardly Lion: I m not sure if 100 samples is going to be enough What is the minimum recommended number of samples? a) 100 with fixed tissue sections b) 100 with fixed tissue sections and 100 with frozen sections c) 50 with fixed tissue sections and 50 with frozen sections d) 60 with fixed tissue sections and 60 with frozen sections CAP #7: The validation process should include a sample set of at least 60 cases for one application (eg, H&E stained sections of fixed tissue, frozen sections, cytology, hematology) that reflects the spectrum and complexity of specimen types and diagnoses likely to be encountered during routine practice. 13

14 Validation study participants Years in practice Digital Pathology Experience Who should participate: a) Wizard only b) Wizard and Dorothy c) All pathologists who will make WSI diagnoses d) As many pathologists as reasonable Specialty Reads frozen sections Dorothy 10 Extensive Generalist Yes Scarecrow 3 None Neuropath Yes Tin Woodman 2 Moderate Hearts Yes Cowardly Lion 5 None Lungs No The Wizard 113 Extensive Generalist Yes CAP #6: A pathologist(s) adequately trained to use the WSI system must be involved in the validation process. 14

15 Study coordinator Dorothy: I ve got a lot of experience with digital pathology. What are benefits of having a pathologist study coordinator? a) Selection of appropriate study cases without biasing the results b) Work with IT to troubleshoot technical issues before validation c) Address questions by participants during the validation d) All of the above CAP #6: A pathologist(s) adequately trained to use the WSI system must be involved in the validation process. 15

16 Case numbers and composition 60 cases are required for each intended use. Dorothy decides on having a minimum of: 120 permanents 60 frozens Permanents Frozens Scarecrow Tin Woodman The cases should be composed of: a) All hearts b) Hearts and lungs c) Lungs and neuropathology d) Equal mix of hearts, lungs, and neuropathology Cowardly Lion 30 n/a The Wizard Totals CAP #7: The validation process should include a sample set of at least 60 cases for one application (eg, H&E stained sections of fixed tissue, frozen sections, cytology, hematology) that reflects the spectrum and complexity of specimen types and diagnoses likely to be encountered during routine practice. 16

17 Order of viewing: digital vs. glass first Permanents Frozens Total cases Scarecrow Tin Woodman Cowardly Lion 30 None The Wizard Dorothy decides that the cases will be viewed on glass first by using the diagnosis already rendered for the case. Is this approach acceptable? a) No, digital must be viewed first since glass is the gold standard b) No, the order of viewing must be randomized. c) Yes, the order can be in random or nonrandom order CAP #9: Digital and glass slides can be evaluated in random or nonrandom order (as to which is examined first and second) during the validation process. 17

18 Review #1: CAP guidelines 1. All pathology laboratories implementing WSI technology for clinical diagnostic purposes should carry out their own validation studies. 6. A pathologist(s) adequately trained to use the WSI system must be involved in the validation process. Having a study coordinator who is a pathologist is essential 7. The validation process should include a sample set of at least 60 cases for one application (eg, H&E stained sections of fixed tissue, frozen sections, cytology, hematology) that reflects the spectrum and complexity of specimen types and diagnoses likely to be encountered during routine practice. For OzPath, the validation study will require 60 fixed tissue and 60 frozen section cases 8. The validation study should establish diagnostic concordance between digital and glass slides for the same observer (ie, intraobserver variability). Validation is focused on determining the variability caused by methodology (digital vs. glass) Comparing diagnoses between different pathologists introduces interobserver variability. 9. Digital and glass slides can be evaluated in random or nonrandom order (as to which is examined first and second) during the validation process. The order, whether glass or digital first, does not matter, and may be random or non-random 18

19 Questions? 19

20 Study design Search LIS for cases signed out by participants Pull slides from storage Review slides for suitability Scan slides Pathologists review cases and enter diagnosis on spreadsheet Assign WSI cases to each original sign-out pathologist Create results spreadsheet for diagnoses Review scans for suitability Coordinator compares WSI diagnosis to prior glass diagnosis Coordinator scores cases as agree, major, or minor Calculate: % agreement, % major, % minor, Review discordant cases 20

21 Validation team Slide curation Slide scanning Digital slide management Servers, network and workstations User software interface Discrepancy review Fetch and organize slides Scanner superuser Scanner superuser IT/computer Pathologist participants Pathologist coordinator Adapted from Mohamed Salama, MD 21

22 Dorothy curates cases Case attrition Fixed (n) Frozen (n) Case attrition: Slides not found Unsuitable cases Cracked or scratched slides Cases requiring immunostains or special stains for diagnosis Cases with many slides Scanning issues Folded tissue Air bubbles Soft focus Selected from LIS Found from files Appropriate for study WSI of sufficient quality Needed for validation

23 Folded tissue Most WSI are captured in a single plane of focus (no Z-stacking), limiting the ability to view folded tissue 23

24 Air bubbles Bubbles can interfere with the plane of focus. 24

25 Slide scanner/workstation (at OZ Hospital) OZ internet service provider (VPN) OZ Hospital servers Emerald City (histology) OZ internet service provider (VPN) OZ Hospital network (behind Firewall) Dorothy s home computer Pathologist workstations at OZ Hospital Computers at Emerald City 25

26 Hardware/software summary Scanner Instrument Computer workstation and operating system Scanner software Network Hospital network Internet Service provider (ISP) and VPN Viewing workstations/monitors Do all of the above components need to be validated? Scanner Scanner workstation OZ Hospital Network Hospital servers CAP #4: The validation study should encompass the entire WSI system. A WSI system is comprised of a slide scanner, computer hardware, software, network, and viewing monitor 1. Each of these components may impact digital image quality and therefore interpretation 1. There is currently no substantial evidence to indicate that each individual component or step in the imaging process (eg, image acquisition, storage, viewing) needs to be validated separately 1. ISP VPN Viewing workstation (Emerald city) 26

27 Validation attempt #1 viewing hardware Which computers/monitors should be used to validate the system? a) Workstations on the OZ hospital network (highest performance) b) Workstations at Emerald city and Kansas without VPN c) Workstations at Emerald city and Kansas with VPN (lowest performance) d) Both a) and c) CAP #3: The validation study should closely emulate the real-world clinical environment in which the technology will be used. CAP #4: The validation study should encompass the entire WSI system. 27

28 Validation attempt #1 120 permanents Scarecrow takes over 2 hours for each of the first several cases Cowardly lion becomes frightened after finding a slide with no tissue. The wizard breezes through the first 10 cases, but recognizes case 11 as one from a case last week! The connection to Emerald city shows poor performance 28

29 Training Dorothy watches Scarecrow review WSI on the computer and discovers that he has a lot of difficulty navigating. This problem may be a reflection of: a. Inadequate training b. Scarecrow s inexperience with digital pathology c. Frustration with new technology d. Steep learning curve with WSI e. All of the above CAP #6: A pathologist(s) adequately trained to use the WSI system must be involved in the validation process. Trained Untrained Interpretation time of WSI (minutes) ± ±

30 Scanner tissue detection Cowardly Lion finds a WSI with no tissue. Lack of tissue on the WSI was the result of operator error: the wrong area on the slide was selected. CAP #11: The validation process should confirm that all of the material present on a glass slide to be scanned is included in the digital image. 30

31 Washout period The Wizard completed all 120 cases, but he remembered many of them that he had signed out from last week. The recommended amount of time between viewing a case on glass versus digital should be at least a. 2 weeks b. 4 weeks c. 3 months d. 6 months CAP #10: A washout period of at least 2 weeks should occur between viewing digital and glass slides. 31

32 Testing the complete system OZ ISP (VPN) Workstations at Emerald City Viewing from Emerald city and Kansas shows poor performance compared to viewing at OZ general hospital. The problem could be related to: a) Poor network performance by the OZ ISP b) A VPN problem c) Issues with computers at Emerald city d) All of the above CAP #3: The validation study should closely emulate the real-world clinical environment in which the technology will be used. CAP #4: The validation study should encompass the entire WSI system. 32

33 Issues addressed before validation attempt #2 Training Basic instruction for opening cases and navigation Participants practice on 10 permanent section cases from attempt #1 Participants track time-to-diagnosis for 10 practice frozen cases Scanner tissue selection A procedure was written for the scanner operator to QA all WSI for image quality and completeness. The validation study will include QA results by the scanner operator Issues with the complete system OZ hospital IT determined the poor remote viewing performance was caused by the VPN The Emerald City and Dorothy s house were included within the OZ hospital network 33

34 Review #2: CAP guidelines 3. The validation study should closely emulate the real-world clinical environment in which the technology will be used. Both local and remote workstations will be used for validation. 4. The validation study should encompass the entire WSI system. A WSI system is comprised of a slide scanner, computer hardware, software, network, workstation and viewing monitor. 6. A pathologist(s) adequately trained to use the WSI system must be involved in the validation process. Proper training is important for validation because of the steep learning curve. 10. A washout period of at least 2 weeks should occur between viewing digital and glass slides. Pathologists remember cases for a long time, particularly those with unique features. 11. The validation process should confirm that all of the material present on a glass slide to be scanned is included in the digital image. Scanned slides should be reviewed for quality and completeness by the scanner operator. 34

35 Questions? 35

36 Validation attempt #2 Dorothy curates a new set of cases that were finalized >3 months ago More than 60 frozen cases are selected The pathologist has the opportunity to defer interpretation. The reason for deferral will be recorded: Desire to show for second opinion, insufficient image quality, need for additional stains, etc. Any major on minor discrepancies will be rereviewed by the group to determine whether the original glass or WSI read is the better diagnosis. It is possible that WSI renders a better diagnosis Permanents Frozens Scarecrow Tin Woodman Cowardly Lion 35 n/a The Wizard Totals

37 Validation results (permanents) Cases n = 140 Deferred, No. (%) Major, WSI inferior, No. (%) Major, WSI superior, No (%) Minor, WSI inferior, No. (%) Minor, WSI superior, No (%) 2 (1.4%) 1 (0.7%) 1 (0.7%) 3 (2.1%) 0 Agree (%) 133 (95%) It is important to review deferrals Often cases that are already problematic with glass slides Grading follicular lymphoma, low versus high-grade epithelial lesions (dysplasia and cancer grading) The % major and % minor discrepancies are calculated separately In the case of a discrepancy, the WSI diagnosis can be superior to the original glass diagnosis Discrepancy review is an important process 37

38 Major discordance Glass: prostatic adenocarcinoma = 6 Digital: prostatic adenocarcinoma = 7 38

39 Minor discordance Glass: reactive (chemical) gastropathy Digital: mild chronic inflammation 39

40 Validation results (frozens) Deferred, No. (%) Major, WSI inferior No. (%) Major, WSI superior (%). Minor, WSI inferior, No. (%) Minor, WSI superior. Agree (%) Median TAT (range) Cases n = 75 3 (4.0%) 1 (1.3%) (94.7%) 8.5 min (2-21) For frozen sections, some cases may be deferred It is important to record the time to diagnosis (including the time to start the load the case into the software viewer and render a diagnosis) Prior to live use in intra-operative diagnosis, conduct studies to determine scanning time of freshly made frozen sections in the intra-operative consultation room. 40

41 Deferrals Air bubble prevented focus; r/o involvement by carcinoma 41

42 Deferrals Folded tissue; r/o metastatic carcinoma 42

43 Major discordance Glass: tissue with invasive fungus Digital: no organisms identified 43

44 Questions? 44

45 Validation for special stains Addition of interpretation of immunostains a) 20 total slides of various immunostains b) 20 slides for EACH specific immunostain c) 60 total slides of various immunostains d) 60 slides for EACH specific immunostain The validation process should include another 20 cases for each additional application (eg, immunohistochemistry, special stains). 45

46 Validation for cytology Addition of cytology interpretation a) 60 total slides that include a variety of different preparation methods (FNA direct smear, thin prep, cytopsin) b) 20 total slides that include a variety of different preparation methods c) 60 slides for each specific preparation method d) 20 slides for each specific preparation method It was the consensus opinion of the expert panel, supported by literature evidence, that the specimen preparation type was a much more important performance variable than the source of the tissue or the specific analyte being assessed. 46

47 Validation of a similar scanner A second scanner is purchased by the OZpath group (same model and manufacturer) a) 60 slides for each intended use b) 60 total slides to cover all intended uses c) 20 slides for each intended use d) 20 total slides to cover all intended uses When an additional WSI system of the same make and model as a previously validated scanner is to be used in a laboratory that shares the same network, image management software, and intended clinical use, a separate validation study using a smaller set of cases (eg, 20 cases) may be adequate to detect any significant differences in scanner functionality. 47

48 Validation of an entirely new scanner A third, large scale high-speed scanner is purchased by the OZ pathology group (different manufacturer) a) 60 slides for each intended use b) 60 total slides to cover all intended uses c) 20 slides for each intended use d) 20 total slides to cover all intended uses CAP #5: Revalidation is required whenever a significant change is made to any component of the WSI system. 48

49 Revalidation with system changes Which of the following likely require revalidation: a) Upgrade to a faster computer CPU (same monitor) b) Upgrade to faster network speed c) Change in monitor to larger size, higher resolution, but different color quality, contrast, and brightness d) Upgrade WSI viewing software resulting in faster performance (no change in image quality) CAP #5: Revalidation is required whenever a significant change is made to any component of the WSI system. 49

50 Review #3: revalidation 1. Immunostains: 20 cases for each major methodology For example, one methodology may be visual detection of diaminobenzidine (DAB) to produce a brown color whereas another method one be NBT to produce a purple color. Each methodology would be subject to different potential artifacts. 2. Cytology: 60 cases for each methodology Cytology is particularly challenging, given that most WSI are not acquired with with variable focus planes (Z-stacking) 3. Similar scanner (same manufacturer and model): 20 cases 4. Different scanner (different manufacturer): Full validation 5. System changes: dependent on potential for a change in the image quality Changes to the operating system, improvements in CPU or networking speed are unlikely to affect image quality. Changing a monitor may have a large effect on image quality (likely improved with newer technology) 50

51 Questions? 51

52 Summary Validation should be appropriate for the intended clinical use and closely emulate the real-world clinical environment. The validation study should encompass the entire WSI system, including scanner, network, computer, and monitor. The validation study should establish concordance between digital and glass slides for the same observer. A washout period of 2 weeks may not be sufficient. 52

53 Pearls of Pathology An MD study coordinator with experience in digital pathology is essential. Assemble an appropriate team of IT, scanner operators, administrative support staff, and pathologists. Train your pathologist participants or risk a failed validation. 53

54 References Pantanowitz L, et al. Validating whole slide imaging for diagnostic purposes in pathology: guideline from the college of american pathologists pathology and laboratory quality center. Archives of Pathology and Laboratory Medicine. 2013;137(12): Koch LH, et al. Randomized comparison of virtual microscopy and traditional glass microscopy in diagnostic accuracy among dermatology and pathology residents. Human Pathology. 2009;40(5):

55 Thanks for attending! Complete the online course evaluation! Be an active participant in shaping the CAP s Annual Meeting by sharing your feedback. 55

56