Phlexglobal Whitepaper Dashboard Metrics: The Solution to Document Risk-Based Monitoring Presented By: Claire Jarmey-Swan Product Development Manager Email: CJarmey-Swan@phlexglobal.com Phlexglobal Inc. 400 Chesterfield Parkway, Suite 120, Malvern, PA 19355, USA T: +1 484 324 7921 F: +1 610 240 0109 Phlexglobal Ltd. Mandeville House, 62 The Broadway, Amersham, Bucks, HP70HJ, UK T: +44 (0) 1494 720420 F: +44 (0) 1494 720430 xxxxxxxx@phlexglobal.com www.phlexglobal.com Copyright 2015. Phlexglobal Ltd. All rights reserved.
Table of Contents Executive Summary... 2 Why do we need Document Risk-Based Monitoring?... 3 How does the availability of dashboard metrics aid DRBM?... 4 Who should use dashboard metrics?... 4 When should dashboard metrics be used?... 5 Conclusions... 9 References... 10 Private and Confidential Page 1 of 11
Executive Summary Risk-based monitoring of clinical trials is identified as a key Good Clinical Practice (GCP) component. Extending this concept to Document Risk-Based Monitoring (DRBM) of regulatoryrequired clinical trial documents, stored by the sponsor in the Trial Master File (TMF) or by the investigator in the Investigator Site File (ISF), will further enable study managers, clinical research associates (CRAs) and others to drive on time study start and quicker study close-out whilst ensuring inspection readiness at all times. The TMF and the ISF together comprise the entire TMF for the trial and should be established at the beginning of the trial to enable the conduct of the clinical trial, the integrity of the trial data and the compliance of the trial with GCP to be evaluated. Centralized cloud-based electronic TMF (etmf) or ISF (eisf) systems, coupled with business intelligence tools that provide real-time metrics about the etmf/eisf document status can be utilized throughout the course of the trial. These Business Intelligence (BI) tools transform the available etmf/eisf metadata, associated with records contained in the underlying repository, into meaningful and useful information for analytical purposes and dashboard presentation. The display of business and operational-level metrics through easy to interpret visual dashboards combined with risk indicator thresholds that initiate automated alerts are designed to trigger an action such as site follow-ups. Alerts that can be set by the system user would actively bring the value of the DRBM findings to that user. Document risk-based monitoring through dashboard metrics underpins forward planning and decision-making to minimize the key risks associated with ensuring an inspection ready etmf namely document timeliness, completeness and quality. Private and Confidential Page 2 of 11
Why do we need Document Risk-Based Monitoring? In much the same way that risk-based monitoring of clinical trials is identified as a key Good Clinical Practice (GCP) component for identifying risk and allocating resource to mitigate risk for clinical data, the need to extend risk-based monitoring to documents is paramount to ensuring an overarching high quality Trial Master File (TMF) comprising the sponsor-owned TMF and investigator-owned Investigator Site File (ISF). Through the course of a clinical trial numerous essential documents are generated, collected and stored in the TMF, either as paper or electronic. These documents serve to demonstrate compliance of the investigator, sponsor and monitor with the standards of GCP and with other applicable regulatory requirements (1) and enable the inspector to reconstruct the events of the trial. Missing or poor quality documents could jeopardize a successful inspection which may in turn impact a clinical trial phase and ultimately time to market. Analyzing and monitoring paper for trends is difficult at the best of times. It generally results in spreadsheet trackers that are not only time-consuming to maintain but lack uniformity in data collection and are subject to human error. On the other hand, documents stored in an electronic TMF (etmf) or an electronic ISF (eisf) makes it easier to review and record findings and improve the quality of the documentation. The etmf or eisf, typically a cloud-based centralized repository for all trial or site documents respectively, is also subject to rigorous inspection by regulatory authorities. To support the need to demonstrate etmf Document Risk-Based Monitoring (DRBM) practices during inspections, novel applications for business intelligence (BI) tools to the plethora of document metadata captured in the etmf/eisf enables clear, concise and most importantly informative visual dashboard metrics that provide insight into the quality and completeness of the entire TMF as well as oversight of contract research organizations (CROs) and partners using the electronic systems. To ensure the production of meaningful BI dashboards that are responsive to user queries, the etmf/eisf and BI tools must operate hand-in-hand. This successful operation is highly dependent on the range, richness, granularity, timeliness and structure of information held in the etmf/eisf repository. An etmf/eisf solution based on a comprehensive underlying relational database repository can enable users to interrogate and analyze the document metadata in a multitude of ways to establish root causes and measure the impact of Private and Confidential Page 3 of 11
preventative actions. But without this users will be limited in their ability to compare and contrast different sets of metadata in easy-to-understand dashboard metrics. How does the availability of dashboard metrics aid DRBM? As clinical trials continue to expand across the globe so the management of trials may become fragmented across and within numerous organizations and departments. This fragmentation further increases the potential risks envisaged, particularly at the interfaces of quality systems or movements of information/data (2). The centralized etmf/eisf makes remote global document risk-based management of program, trial or site documentation possible through BI dashboards that can be accessed anywhere and in any time zone across the world. Providing the bird s eye view to managing the etmf/eisf and its associated risks, such as quality, timeliness and completeness of the etmf/eisf, by means of dashboard metrics pinpoints where actions need to be taken to ensure a successful inspection. Drilling down from a dashboard to display the underlying validated document metadata can reveal information such as document dates, document versions and content reference. Reacting to certain situations, actions may be taken that reflect the need for additional training or process revision. These actions would be directed to adjust roles or behaviors in order to ensure the inspection readiness of the etmf/eisf. Who should use dashboard metrics? Sponsor/CRO CRAs and study managers need to reduce risk and manage activities associated with ensuring a secure, compliant, and complete TMF which has to be readily available and produced within a reasonable time (often within a few days) for inspectors. Site personnel would have the tools to assess their site documentation status and react accordingly. This can be easily achieved through business intelligence tools that convert the etmf/eisf metadata into meaningful and useful information for analytical purposes. These top-level metrics displayed through visual pie charts, column and bar charts, bubble charts and tables enable quick understanding, interpretation and forward planning by management to minimize the risk associated with ensuring a complete etmf/eisf. In addition the dashboards and underlying metadata can be shared with management, colleagues or business partners for further analysis. Private and Confidential Page 4 of 11
When should dashboard metrics be used? Insight into the etmf/eisf Having invested in an etmf/eisf system, ensuring it is regularly used and actively updated with documents is paramount to reducing the risk of not being able to demonstrate a contemporaneous TMF. The status quo of the entire TMF can be quickly ascertained through etmf/eisf dashboard metrics that provide not only a summary of the document estates but also the trends in document submissions (Figure 1). Management of the etmf content is also made easier through analysis of document submissions by country and site that may be presented in interactive tables or through dynamic filters that enable the user to select and focus on specific details such as compound, protocol, and country or date range. Figure 1: Dashboard metrics for managing etmf document estates Managing Partner Oversight In Europe the sponsor is required by law to maintain oversight of a clinical trial (3) and the FDA believes that risk-based monitoring could improve sponsor oversight of clinical investigations (4). Extending these concepts to DRBM and focusing on the risks associated with the most critical documents and metadata further supports the growing consensus that risk-based approaches to monitoring are more likely to ensure overall study quality than routine visits to all clinical sites (5, 6, 7). Centralized etmf systems together with supporting dashboard metrics provide sponsors with oversight of contract research organizations (CROs) and other third party vendors in terms of their activity within the etmf (Figure 2). Private and Confidential Page 5 of 11
Figure 2: Partner comparison dashboard metrics for oversight of CROs Timeliness of document submissions is a key risk factor, particularly ahead of an inspection, and needs to be actively monitored and managed by the sponsor. Dashboard metrics that display the trends in submission timeliness across partner organizations combined with early warning traffic lighting enables sponsors to pinpoint trends in late submissions and increase their vigilance of potential late-submitter CROs (Figure 3). This may drive the need for setting service level agreements in the form of timeliness targets and key performance indicators (KPIs) with CROs to ensure documents are submitted to the etmf on time and in full. Figure 3: Early warning traffic lighting highlights timeliness of document submissions by organizations Top-level CRO-specific dashboard metrics can be utilized to identify, screen and select potential research partners, manage strategic alliances, evaluate the progress of on-going trials and identify underperforming CROs that may impact the inspection readiness of the TMF. Monitoring etmf Completeness Private and Confidential Page 6 of 11
ICH GCP guidelines state that trial master files should be established at the beginning of the trial, both at the investigator/institution s site and at the sponsor s office and a final close-out of a trial can only be done when the monitor has reviewed both investigator/institution and sponsor files and confirmed that all necessary documents are in the appropriate files (1). Thus prior to achieving site ready or site close-out status, a series of essential documents, defined by the ICH GCP (1), that individually and collectively permit evaluation of the conduct of a trial and the quality of the data produced must be filed in the TMF/eTMF. As any or all of these essential documents may be subject to, and therefore should be available for inspection by regulatory authorities, the risk of having missing or poor quality documents needs to be eliminated. Driven by identification of expected documents, etmf/eisf dashboard metrics configured for site ready or site close-out milestones can swiftly indicate which sites have not submitted required documents and may therefore hinder clinical trial progress (Figure 4). Providing the latest high-level information the etmf/eisf completeness dashboards enable monitors to proactively manage sites, focusing on those that are lagging in submissions and thereby reducing the risk of not meeting a site-ready date milestone or falling short of inspector expectations. Figure 4: etmf completeness dashboard metrics Ensuring a high quality etmf/eisf Dashboard metrics providing a visual overview of etmf/eisf document query trends quickly highlight the commonly occurring document quality concerns which could include metadata discrepancies, incorrect site and document image quality (Figure 5). Private and Confidential Page 7 of 11
Figure 5: etmf document query category metrics Paper documents scanned for submission to the etmf need to be of high quality to ensure accuracy of the information being converted to digital content. Scanning poor quality paper documents increases the risk of capturing a characters incorrectly making searching for the correct document that more difficult - a situation one does not wish to be confronted with during an inspection. The risks associated with ensuring patient confidentiality need to be mitigated. Any occurrences of documents with ipatient identifiers, highlighted during the etmf quality review, may be displayed as a query category segment (Figure 5) thereby enabling corrective and preventative actions to be put in place and reducing the risk of re-occurrence. Figure 6: Queries by organizations Private and Confidential Page 8 of 11
Trends in queries resulting from CROs and partners submitting documents can be monitored over time to highlight the success of interventions such as additional site-visits or training (Figure 6). Conclusions Regulatory bodies such as the FDA encourage greater use of centralized monitoring practices, where appropriate, than has been the case historically, with correspondingly less emphasis on on-site monitoring. Centralized cloud-based etmf/eisf systems, coupled with document riskbased metrics further support regulatory directives enabling the sponsor oversight and management of CRO and investigator sites against agreed key performance indicators. With BI tools, real-time metrics regarding the etmf/eisf can be obtained throughout the course of the trial. This continuous document risk-based monitoring will identify studies and sites that are lagging and enable immediate follow-up and call to action, resulting in on time study start and quicker study close-out. The overall impact: an inspection-ready etmf/eisf through enhanced partner oversight, overall TMF content insight and good document quality management. Private and Confidential Page 9 of 11
References 1. ICH Topic E6, Guideline for Good Clinical Practice: Note for Guidance on Good Clinical Practice, CPMP/ICH/135/95, 2002 2. European Medicines Agency, Reflection paper on risk based quality management in clinical trials, EMA/269011/2013, 2013 3. European Commission s Directive on Good Clinical Practice, Article 7, Directive 2005/28/EC, 2005 4. Food and Drug Administration, Guidance for Industry, Oversight of Clinical Investigations A Risk-Based Approach to Monitoring, FDA, 2013 5. Usher, R, PhRMA BioResearch Monitoring Committee Perspective on Acceptable Approaches for Clinical Trial Monitoring, Drug Inf J. 44: 477-483, 2010 6. Food and Drug Administration, Concept Paper: Quality in FDA-Regulated Clinical Research, FDA, 2007 7. Brosteanu et al., Risk Analysis and Risk Adapted On-Site Monitoring in Noncommercial Clinical Trials, Clin Trials 6: 585-595, 2009 To learn more about Phlexglobal and PhlexEview, please visit our website at www.phlexglobal.com. Private and Confidential Page 10 of 11
Phlexglobal The Complete TMF Solution Private and Confidential Page 11 of 11