Determining How to Make Data Review More Efficient by Utilizing Visualization Tools. Susan Doleman, Replimune Inc.

Transcription

1 Determining How to Make Data Review More Efficient by Utilizing Visualization Tools Susan Doleman, Replimune Inc.

2 Study Data Review Why? In order to ensure Patient safety Looking for unknown risks Data quality Is there consistency/inconsistency Are there problems with the protocol (adherence, interpretation) Are the outliers routine anomalies or potential signals Does the data look too clean fraud/misconduct? Scientific rigor Analyses are meaningful

3 Traditional Approaches Programmed Edit Checks Fairly straightforward process Logic checks of protocol requirements (visit windows, bp ranges, start dates, etc) All written into Excel Spreadsheet with data domain language to be interpreted by programmers Difficult cross-checks (typically across data domains) are not standard and are often relegated to manual checks, that are less robust After programming often difficult if logic is incorrect and may blow up a live database if not; (Nasty queries getting nasty responses ) Challenges only as good as your specs and programming allows; VERY labor intensive

4 Traditional Approaches Manual Data Review Line listing review: does anything look off or which one does not look like the other ; outliers that aren t programmatically checked. Classic examples: dramatic changes in scores/measurements between visits that don t look real Remember doing it with paper??!! Cross listing review: checking between two datasets for inconsistency checks. Classic examples: AE to Conmeds, PE findings to Medhx or AE, CS lab values to AE Subject profile review: review of entire subject data as a case. Look across individual datasets (e.g., dosing, vitals etc) for inconsistencies/errors Challenges only as good as your reviewers ability to see abnormalities/problems; also ability to document review of data ( what data cut was that again ) responsibilities for follow-up and closeout a very manual and tedious process

5 Traditional Approaches Statistical Programming Checks More sophisticated checks typically run as database is nearing lock Programs run and findings shared with data management/clinical operations Challenges: only as good as the programmer s ability to understand protocol, datasets and correct from incorrect. much time can be spent programming checks that lack meaning have true value to the outcome of the trial tremendous amount of time following errors that have already been reviewed by others and okayed/dismissed. data has been reviewed multiple times and is now re-reviewed for unclear benefit

6 Data Review & Cleaning what do we do Sites enter data and respond to edit checks that are preprogrammed for inconsistent/incorrect data CRAs travel to sites according to the monitoring plan and perform Source Document Verification (SDV) for consistency between medical records and recorded data Data managers review incoming data according to the Data Management Plan and reviews Medical reviewers review incoming data according to the medical data review plan and provide insight into medical anomalies/errors or concerns Clinical reviewers review incoming data according to the data review plan for consistency, difficult to program checks, where clinical/protocol knowledge is needed All of this leads to declaring a trial/subject/page clean and adequate for analysis

7 Data Review & Cleaning what we want to do Minimize time to develop and validate edit checks Reduce on-site time of CRAs doing 100% SDV and allow for more value added activities Reduce need for lengthy manual data review or reliance on statistical programming looking for errors Allow medical and clinical reviewers to see data in an ongoing fashion that is meaningful and current Review data at the source (CRF) and allow for clear trail of what has occurred since first entry while reviewing summary data Have systems for statistical programs that learn as the data is entered and finds data anomalies based upon the data not programmed checks

8 Data Review & Cleaning Visualizations Key Components of Data Visualizations & Use Developing the tools (purchase and develop/maintain internally, use service provider) Which is more suited to your organization For small virtual companies SAAS approach can now be utilized with limited internal staff and infrastructure, and importantly AFFORDABILITY Defining the review process (who, what, where and when) Often different reviewers are reviewing the same data is it necessary? Clearly describe who does what review and when Documenting the review and output (audit trail) Clearly establish what has been reviewed, by whom, and what actions were taken

9 Data Review & Cleaning Visualizations Safety review with drill downs Source:

10 Data Review & Cleaning Visualizations Detecting outliers - fraud Source: Cluepoints Case study 2

11 Data Review & Cleaning Visualizations Outliers

12 Visualizations - pitfalls Example 1: Decision made to implement powerful data visualization tool Internally managed and decision to provide each team with a Superuser Relied heavily on staff members that were volunteered to be these experts (while continuing day job ) Limited engagement of wider team members to develop or understand the visualizations Limited hands-on engagement and training find a superuser to help Refreshing of data not consistent or clear constant struggle with review of old data causing frustration for reviewers and unnecessary re-querying Tracking of issues/queries still relied on excel spreadsheet tracking and closing of issues

13 Visualizations - pitfalls Example 2: Requested use of CRO s data visualization tool Very little known internally at CRO about new tool One person (expert) responsible for implementation No cross-collaboration with Sponsor requested Very unclear how it would be implemented with CRO staff vs Sponsor (who what where when) Sponsor should have been clearer with scope and goals See Example 1 for likely outcome

14 Visualizations Other Data Review Trial conduct data is EQUALLY important to clinical data What is useful trial conduct data Enrollment (actual vs predicted) Adherence to monitoring plan visits on target, reports timely Data collection trends data entry time, query resolution times, CRFs outstanding Safety reporting on time SUSAR reporting Protocol deviation tracking and review TMF status (audit readiness) Issue tracking and resolution Challenges: Access to trial conduct data is most often separated from EDC or loosely integrated and depends on various trackers, various sources of data

15 Visualizations current goals for clinical data Plug and Play solution integration into ANY EDC system as standard part of tool; highly configurable Full and live connection to EDC no need for nightly downloads and uploads for tools to function Data visualizations must allow for drill down directly to current ecrf for understanding of what has occurred prior Data visualizations must allow tagging for actions by staff (e.g. CRA to review, DM to query, CTM to review monitoring plan etc.) All actions must have clear audit trail of reviews conducted, actions taken and resolutions As data allows (e.g. enough datapoints be able to utilize statistical learning to detect anomalies based upon statistical trends (like fraud detection in banking) Dashboards for data health and data quality also allowing drill down into problem areas

16 Visualizations current goals for trial conduct data Single source of truth for all data Provides dashboards and visualizations of study conduct health etmf CTMS/monitoring EDC Safety reporting Drug supply Biospecimens Etc, etc. All readily configurable, agnostic to data source Full drill down capabilities to look at source of any issue Ability to assign tasks for issue resolution with audit trail Validated systems

17 Summary Multiple tools now exist that are available to small emerging biotech Cloud solutions and SAAS allow us to avoid large investments in staff and hardware Data integration and visualizations service and provide information into all aspects of our clinical trial health Over next year: implement and pilot some functionalities that integrate with external vendor systems