Knowledge Representation and Tool Support for Critiquing Clinical Trial Protocols

Size: px

Start display at page:

Download "Knowledge Representation and Tool Support for Critiquing Clinical Trial Protocols"

Maximillian Patrick
6 years ago
Views:

1 Knowledge Representation and Tool Support for Critiquing Clinical Trial Protocols Daniel L. Rubin, 1,2 John Gennari, 3 and Mark A. Musen 2 1 Center for Health Care Evaluation, VA Palo Alto Health Care System, Palo Alto, CA 2 Stanford Medical Informatics, Stanford University School of Medicine 3 Information & Computer Science Department, University of CA, Irvine Abstract The increasing complexities of clinical trials have led to increasing costs for investigators and organizations that author and administer those trials. The process of authoring a clinical trial protocol, the document that specifies the details of the study, is usually a manual task, and thus authors may introduce subtle errors in medical and procedural content. We have created a protocol inspection and critiquing tool (PICASSO) that evaluates the procedural aspects of a clinical trial protocol. To implement this tool, we developed a knowledge base for clinical trials that contains knowledge of the medical domain (diseases, drugs, lab tests, etc.) and of specific requirements for clinical trial protocols (eligibility criteria, patient treatments, and monitoring activities). We also developed a set of constraints, expressed in a formal language, that describe appropriate practices for authoring clinical trials. If a clinical trial designed with PICASSO violates any of these constraints, PICASSO generates a message to the user and a list of inconsistencies for each violated constraint. To test our methodology, we encoded portions of a hypothetical protocol and implemented designs consistent and inconsistent with known clinical trial practice. Our hope is that this methodology will be useful for standardizing new protocols and improving their quality. Background and Rationale The progressive reliance on clinical trials for therapeutic advances and for developing clinical practice guidelines has caused a sharp increase in the number of protocols written each year. Developing new clinical trial protocols is a long and expensive process. Generally, multiple layers of expert committee review are required to finalize a protocol. Despite this oversight, many new protocols have serious problems involving incomplete or inconsistent specifications of treatment procedures and patient monitoring events, as well as errors in study methodology [1]. For instance, a protocol using a cardiotoxic drug may fail to require that patients be screened for heart disease, or may not state the need for a left ventricular function screening test if a patient has a history of cardiac insufficiency. Oncology trials can be particularly problematic, as they are frequently complex and use multiple drug treatments. Chemotherapeutic drugs often have serious toxicities, and multiple monitoring events that vary with disease history must be specified. Drug contraindications must be checked against all eligibility criteria, and potential drug interactions with other drugs or concurrent patient diseases must be considered. Unfortunately, new protocol design usually focuses on the primary efficacy endpoint, with less attention given to side effects relating to the primary treatment [2]. Thus, toxicity-related omissions in protocol design are likely. Automated methods can streamline and improve the entire process of clinical research. The National Cancer Institute is developing a Cancer Informatics Infrastructure (CII) to help automate clinical research [3]. An objective of the CII group is to create computer-based tools that improve the efficacy of protocol authoring and deployment. To support these protocol tools, there is a need for a general-purpose knowledge base for clinical trial protocols. In this paper, we report on initial development of one aspect of protocol authoring: a protocol-critiquing tool. Tools for protocol authoring could increase the efficiency of new trial development and reduce errors in protocol specifications by evaluating their contents. A critiquing tool with a knowledge base of drugs, contraindications, toxicities, recommended patient monitoring activities, and drug interactions could evaluate a new protocol design and alert the author if inconsistencies or errors are detected. If the tool had information about possible corrections in its knowledge base, the tool could even automatically correct problems that it finds. In the long term, the tool could be integrated with other protocol authoring tools, and its knowledge base could be integrated with the knowledge base of other clinical trial applications. Clinical trial protocols can be critiqued on two levels: protocol methodology and protocol structure. Protocol methodology is the statistical and study design: sample size, statistical model, definitions of endpoints, and choice of study arms. Protocol structure is the medical and procedural content: patient treatments, monitoring tests, and drug toxicity. Protocol critiquing systems have been described [4, 5], but these systems have primarily evaluated protocol methodology and statistical design, rather than protocol structure and treatment planning. Errors in protocol structure may be significant because they contribute to misunderstandings and deviations from the protocol [6]. To our knowledge, a sys-

2 Eligibility Criteria Proposed Monitoring Drugs Diseases Drug Toxicity Monitoring Events Drug Treatments PROTOCOL KNOWLEDGE MEDICAL KNOWLEDGE Figure 1. Relationships in the data model for critiquing trial protocols. Circled entities are classes of knowledge in the knowledge base. Arrows depict relationships between classes of knowledge. For example, eligibility criteria are specified using knowledge of diseases and drug treatments. tem that evaluates and critiques clinical trial protocols from a structural perspective has not yet been developed. We hypothesize that authors will produce protocols with have fewer errors if they use a tool that critiques clinical trial protocol structure. We are developing such a tool, and call it the Protocol Inspection and Critiquing Tool of Study Structure, or PICASSO. PICASSO automatically evaluates and critiques the procedural and medical aspects of a clinical trial protocol design. It accomplishes this evaluation by using medical knowledge, clinical trial protocol knowledge, and formal constraints that describe the sorts of errors we wish to find in a protocol. Clinical trial protocols are very detailed, and the requirements to fully model a protocol are extensive. Therefore, we initially chose to restrict our modeling efforts to a subset of the clinical trial protocol. Our initial objective was to develop a knowledge base for the critiquing tool and a prototype engine that implements critiquing with respect to diseases, drugs, and patient monitoring. Because we are using a modular design, our model is extensible, and can be applied to a complete representation of protocols in the future. A Knowledge Base for Critiquing In our knowledge base for clinical trial protocols, we divide knowledge into two classes: medical domain knowledge and clinical trial domain knowledge. The medical domain component of the knowledge base describes information about patient conditions, diagnostic tests, and therapeutic procedures. The clinical trial domain component of the knowledge base represents issues specific to clinical trials: the specification of the protocol, and the notions of toxicity and patient monitoring. Both components are modeled with an ontology: a set of related concepts that defines the relevant domain of discourse. The ontology provides a framework within which to organize knowledge using a hierarchy of taxonomic and other relationships. For example, both daunorubicin and doxorubicin are subclasses of the Antitumor Antibiotic class, which in turn is a subclass of the cytotoxic drug class. This knowledge organization allows us to deduce relationships from the hierarchy, such as the fact that daunorubicin is a cytotoxic drug. Malogolowkin analyzed the clinical trial design process used by experts and showed that protocol content can be classified into information categories for depicting the structure of a clinical trial [2]. We identified several of these attributes in the clinical trial and medical domains. These attributes are fundamental to modeling protocols for critiquing. For the clinical trial domain, they include drugs, eligibility criteria, and patient monitoring activities. For the medical domain, fundamental attributes include diseases, patient treatments, drug contraindications and toxicities (adverse events), and monitoring activities. Figure 1 shows some of these attributes, as well as some of the relationships among them in our model of clinical trial protocols. In our model, a clinical trial protocol consists of eligibility criteria, study treatments, and patient monitoring. Eligibility criteria define the medical conditions of the clinical trial patient population. In our knowledge base, we link drug treatments in a protocol to potential drug toxicities for that agent, and to specific monitoring tests for detecting those toxicities (Figure 1). As an example of some of the information in our knowledge base, there is an instance for the disease breast cancer that contains the knowledge that doxorubicin, cyclophosphamide, and tamoxifen are indicated drugs for this condition, whereas corticosteroids and estrogen are contraindicated. Similarly, there is an instance in the knowledge base describing daunorubi-

Figure 2. Example output produced by PICASSO. PICASSO has evaluated Protocol 3, and critiques this protocol with respect to drug contraindications, drug interactions, and patient monitoring.

3 Figure 2. Example output produced by PICASSO. PICASSO has evaluated Protocol 3, and critiques this protocol with respect to drug contraindications, drug interactions, and patient monitoring. cin, which contains information about disease indications, contraindications, and drug toxicity. Our model of clinical trial protocols is limited, and is not sufficient to represent all aspects of a clinical trial protocol. However, the goal for this initial stage is to demonstrate the feasibility of critiquing protocol structure with a subset of the domain. Because we have used a highly organized ontology to frame our knowledge representation, our model can be easily extended. Future work will expand our knowledge base to model clinical trial protocols more fully. An advantage of using an extensible ontology for capturing protocol knowledge is that the knowledge base can be easily extended or revised to accommodate new information without necessarily requiring changes to the critiquing engine. The knowledge base for the authoring tool was developed with the Protégé suite of tools [7]. Our organization and understanding of the critiquing task evolved as we built our knowledge base. Protégé is designed for rapidly evolving knowledge bases, which made managing changes easier for us. The tool set also made the knowledge base readily available to applications such as our critiquing tool. Protégé is designed as a set of interacting plug-in components. We developed PICASSO as a plug-in, and any Protégé plug-in can access Protégé knowledge bases via a Java applicationprogramming interface. A Protocol Critiquing Tool Figure 2 shows a screen shot of our protocol critiquing tool. The tool can evaluate protocols for several different clinical trials. The figure shows a list of protocols on the left, and a critique of the selected protocol on the right. To use our tool, a protocol author builds a new protocol as a collection of instances describing the eligibility criteria, drug treatments, and patient monitoring events for the trial. Figure 3 shows an example protocol, modeled after a clinical trial for lymphoma. Once the user enters the protocol specification, PICASSO can critique it. In our approach, protocols are critiqued with protocol constraints expressed in a formal axiom language. Thus far, we have captured three constraints about protocols: 1) Do not prescribe contraindicated drugs: The drug treatments listed in the protocol must not be contraindicated by any of the diseases listed in the eligibility criteria. 2) Include all required monitoring actions: The monitoring actions listed in the protocol must include all required actions listed by each of the protocol s drug treatments. 3) Do not prescribe interacting drugs: The drug treatments listed in the protocol must not interact with each other. The specification of these constraints is dependent on our ontologies of medical knowledge and of protocols. As these models become richer, we expect to add new constraints that describe other aspects of protocol authoring. To allow us to add constraints easily, we specify our constraints in the formal axiom language that is part of the Protégé environment (PAL, or protégé axiom language). Figure 4 shows the axiom for drug-drug interactions as written in PAL. By using this type of formal constraint specification, we can be unambiguous, clear, and precise about the types of protocol authoring errors we wish to find. In addition, when we need to add new constraints, we do not need to make any code-level modifications to the system. We can simply add new constraints, and the PAL general-purpose engine will check them

4 Figure 3. A knowledge instance specifying a particular clinical trial protocol. against our knowledge base. Although the current implementation does not include a direct connection between the PAL engine and PICASSO, both are Protégé plug-ins, and thus, could be connected easily. We have not yet conducted formal evaluation studies of our tool. However, we tested the face validity of our approach by entering a few variations on a protocol for treating lymphoma and critiquing them. The initial draft protocol specified lymphoma and heart failure in the eligibility criteria, five drugs (prednisone, cyclophosphamide, vincristine, doxorubicin, and phenytoin), and multiple monitoring events (Figure 3). As shown in Figure 2, PICASSO reported that (1) daunorubicin is contraindicated because heart failure was listed as an eligibility criterion, and that (2) three necessary monitoring events were omitted. When we changed the eligibility criteria from heart failure to rheumatic heart disease, complicated by heart failure, PICASSO still produced the appropriate critique (Figure 2, right top panel). Importantly, although the knowledge base only includes daunorubicin s contraindication in heart failure, PICASSO deduced that it is (forall?protocols (forall?drugtreat1 (forall?drugtreat2 (=> (and (DrugTreatments?protocols?drugTreat1) (DrugTreatments?protocols?drugTreat2) (not (=?drugtreat1?drugtreat2)) (forall?drugknow (forall?druginteract (=> (and (Drug?drugKnow (DrugAdministered?drugTreat1)) (DrugInteractions?drugKnow?drugInteract)) (not (= (DrugAdministered?drugTreat2)?drugInteract)) ))) )))) Figure 4. A simplified axiom in PAL that identifies protocols that have drug-drug interactions. The first portion states that this constraint only applies to protocols with at least two distinct drug treatments. The second implication searches through the drug knowledge base and claims that there should be no drug listed under the interactions for drugtreat1 that matches drugtreat2. also contraindicated in complicated rheumatic heart disease by evaluating the latter s relationship to heart failure in the ontology. Discussion and Conclusions Many clinical trials are developed annually, yet most are written from scratch. Errors in the protocols for new clinical trials occur commonly [1], and include omissions and ambiguities in protocol semantics [6]. We undertook this study to develop a tool that uncovers semantic errors in clinical trial protocols by evaluating their medical and procedural contents. We assume that all necessary information can be modeled in a knowledge base of the medical and clinical trial domains. In developing our critiquing model, we distinguish medical knowledge from critiquing knowledge [8], and model these separately. Medical knowledge includes information about diagnostic and therapeutic procedures, whereas critiquing knowledge includes the information content of clinical trial protocols and how structural design errors may occur. This distinction is useful because medical knowledge is independent of clinical trial protocols, while critiquing knowledge is more specific, and describes how to evaluate protocol structure. Our approach to knowledge modeling for the critiquing task is similar to that used by HyperCritic, an expert system that advises in the management of hypertensive patients [9]. HyperCritic explicitly separates critiquing knowledge from medical knowledge, which is advantageous for knowledge modeling and reuse [9]. A challenge for designers of expert systems like PICASSO is to deal with knowledge that changes over time. In traditional knowledge-based systems, adding new knowledge often breaks the inference engine, especially if that system uses declarative production rules. An advantage of our approach is that knowledge is modeled in an extensible ontology that encodes knowledge in terms of relationships among concepts in a hierarchy.

5 For example, the knowledge base could have an entry stating that all beta blocker drugs are contraindicated in heart failure. As new beta blockers are developed, we could add to them knowledge base and the critiquing tool would correctly recognize that these drugs are also contraindicated in heart failure. Likewise in our test case, we were able to encode high level concepts in our knowledge base (daunorubicin is contraindicated in heart failure), and PICASSO correctly determined that daunorubicin is contraindicated in a protocol that included rheumatic heart disease, complicated by heart failure. For these types of additions to the knowledge base, PICASSO behaves correctly without any need to change the code that implements its critiquing functionality. Our current model and knowledge base for clinical trial protocols are over-simplified. However, an advantage of working within Protégé is that models can be easily extended and integrated with other clinical trial tools. For example, we have developed a tool to assist protocol authors create eligibility criteria for new trials [10]. This eligibility writer tool uses a similar knowledge-based architecture, and could be integrated with PICASSO to create a tool that supports both protocol authoring and critiquing. Our team has also used Protégé for other sorts of clinical trial protocol tools. For example, we have developed tools that screen patients and match them to clinical trials, as well as tools that monitor patients enrolled in clinical trials [11]. All of these tools can utilize an integrated knowledge base for clinical trial protocols. For example, the medical knowledge base for PICASSO was created from a knowledge base developed for our patient monitoring tool. Our approach is to model protocol knowledge from the bottom-up : always with a specific tool or task in mind. Then, the Protégé tool set allows us to easily combine and integrate models and knowledge bases where appropriate. We have demonstrated the feasibility of developing a tool to evaluate the structure of clinical trial protocols and critique them. The expected benefits from our approach include: (1) a flexible critiquing model that can be subsequently modified and extended, (2) systematic use of standardized critiquing strategies, and (3) improved quality and clarity of new protocols. Although our initial prototype does not fully model protocols, our knowledge base can be extended without unduly affecting the critiquing engine. Future work will extend the ontology in order to represent clinical trial protocols more completely. A critiquing tool may improve the quality of protocols in the future, and could support standardization for clinical trials. Tools like PICASSO that support the clinical trial process could lead to faster protocol deployment and patient accrual, and could ultimately lead to a more efficient and successful process for clinical research. Acknowledgments We wish to acknowledge the work and assistance of the Protégé scientific staff, and especially William Grosso for the development of Protege Axiom Language. Thanks to Valerie Natale for comments on this manuscript. References 1. Andersen B. Methodological Errors in Medical Research: An Incomplete Catalogue. Boston: Blackwell Scientific Publications; Malogolowkin MH, Horowitz RS, Ortega JA, Siegel SE, Hammond GD, Weiner JM. Tracing expert thinking in clinical trial design. Comput Biomed Res 1989;22(2): Silva J, Wittes R. Role of clinical trials informatics in the NCI's cancer informatics infrastructure. Proc AMIA Symp 1999: Wyatt JC, Altman DG, Heathfield HA, Pantin CF. Development of Design-a-Trial, a knowledgebased critiquing system for authors of clinical trial protocols. Comput Methods Programs Biomed 1994;43(3-4): Haag U. Knowledge representation for computer-aided planning of controlled clinical trials: the PATriCIa project. Methods Inf Med 1997;36(3): Musen MA, Rohn JA, Fagan LM, Shortliffe EH. Knowledge engineering for a clinical trial advice system: uncovering errors in protocol specification. Bulletin du Cancer (Paris) 1987;74(3): Musen MA, Fergerson RW, Grosso WE, Noy NF, Crubezy M, Gennari JH. Component-Based Support for Building Knowledge-Acquisition Systems. In: Proceedings of the Conference on Intelligent Information Processing (IIP 2000) of the International Federation for Information Processing World Computer Congress (WCC 2000); 2000; Beijing; van der Lei J, Musen MA. The separation of reviewing knowledge from medical knowledge. Methods Inf Med 1995;34(1-2): van der Lei J, Musen MA. A model for critiquing based on automated medical records. Comput Biomed Res 1991;24(4): Rubin DL, Gennari JH, Srinivas S, Yuen A, Kaizer H, Musen MA, et al. Tool support for authoring eligibility criteria for cancer trials. Proc AMIA Symp 1999: Tu SW, Kemper CA, Lane NM, Carlson RW, Musen MA. A methodology for determining patients' eligibility for clinical trials. Methods Inf Med 1993;32(4):

GLEE A Model-Driven Execution System for Computer-Based Implementation of Clinical Practice Guidelines

GLEE A Model-Driven Execution System for Computer-Based Implementation of Clinical Practice Guidelines A Model-Driven Execution System for Computer-Based Implementation of Clinical Practice Guidelines Dongwen Wang, M.Phil., Edward H. Shortliffe, M.D., Ph.D. Department of Medical Informatics, Columbia University