Derivation and Internal Validation of the Ebola Prediction Score for Risk Stratification of Patients With Suspected Ebola Virus Disease

Transcription

1 INFECTIOUS DISEASE/ORIGINAL RESEARCH Derivation and Internal Validation of the Ebola Prediction Score for Risk Stratification of Patients With Suspected Ebola Virus Disease Adam C. Levine, MD, MPH*; Pranav Prathap Shetty, MD, MPH; Ryan Burbach, MPH; Sambhavi Cheemalapati, MSIS, MPA; Justin Glavis-Bloom, BA; Tess Wiskel, MD; J. Kota T. Kesselly, BSN, MPH *Corresponding Author. Study objective: The current outbreak of Ebola virus disease in West Africa is the largest on record and has overwhelmed the capacity of local health systems and the international community to provide sufficient isolation and treatment of all suspected cases. The goal of this study is to develop a clinical prediction model that can help clinicians risk-stratify patients with suspected Ebola virus disease in the context of such an epidemic. Methods: A retrospective analysis was performed of patient data collected during routine clinical care at the Bong County Ebola Treatment Unit in Liberia during its first 16 weeks of operation. The predictive power of 14 clinical and epidemiologic variables was measured against the primary outcome of laboratory-confirmed Ebola virus disease, using logistic regression to develop a final prediction model. Bootstrap sampling was used to assess the internal validity of the model and estimate its performance in a simulated validation cohort. Results: Ebola virus disease testing results were available for 382 (97%) of 395 patients admitted to the Ebola treatment unit during the study period. A total of 160 patients (42%) tested positive for Ebola virus disease. Logistic regression analysis identified 6 variables independently predictive of laboratory-confirmed Ebola virus disease, including sick contact, diarrhea, loss of appetite, muscle pains, difficulty swallowing, and absence of abdominal pain. The Ebola Prediction Score, constructed with these 6 variables, had an area under the receiver operator characteristic curve of 0.75 (95% confidence interval 0.70 to 0.80) for the prediction of laboratory-confirmed Ebola virus disease. Patients with higher Ebola Prediction Scores had higher likelihoods of laboratory-confirmed Ebola virus disease. Conclusion: The Ebola Prediction Score can be used by clinicians as an adjunct to current Ebola virus disease case definitions to risk-stratify patients with suspected Ebola virus disease. Clinicians can use this new tool for the purpose of cohorting patients within the suspected-disease ward of an Ebola treatment unit or community-based isolation center to prevent nosocomial infection or as a triage tool when patient numbers overwhelm available capacity. Given the inherent limitations of clinical prediction models, however, a low-cost, point-of-care test that can rapidly and definitively exclude Ebola virus disease in patients should be a research priority. [Ann Emerg Med. 2015;66: ] Please see page 286 for the Editor s Capsule Summary of this article. A feedback survey is available with each research article published on the Web at A podcast for this article is available at /$-see front matter Copyright 2015 by the American College of Emergency Physicians. SEE EDITORIAL, P INTRODUCTION Background The current outbreak of Ebola virus disease is the largest since the virus was first discovered in 1976 during simultaneous outbreaks in Sudan and the Democratic Republic of the Congo. During those outbreaks, Ebola virus was first differentiated from Marburg virus, both members of Filoviridae but serologically distinct. 1 The current Ebola outbreak in West Africa was declared a public health emergency of international concern by the World Health Organization (WHO) in August 2014, and as of January 2015 there have been more than 21,000 suspected and confirmed cases of Ebola virus disease, resulting in more than 8,600 deaths. 2-4 Early in the course of this epidemic, the increasing number of patients vastly outstripped the available isolation and treatment capacity in the 3 most affected countries, Guinea, Liberia, and Sierra Leone. 5-7 Case incidence is currently declining, and these 3 countries now have the overall capacity to isolate and manage all suspected cases. However, there remains an Volume 66, no. 3 : September 2015 Annals of Emergency Medicine 285

2 Ebola Prediction Score Editor s Capsule Summary What is already known on this topic Early symptoms of Ebola virus disease are similar to those of other illnesses common in Africa. Early diagnosis is useful for grouping patients and for resource prioritization. What question this study addressed Clinical features were recorded on admission and correlated with Ebola test results for 382 patients with suspected Ebola at an Ebola treatment unit in Liberia; 42% were positive. What this study adds to our knowledge An Ebola Prediction Score was developed with 6 variables: sick contact, diarrhea, loss of appetite, muscle pains, difficulty swallowing, and absence of abdominal pain. How this is relevant to clinical practice The Ebola Prediction Score can risk-stratify but should be validated in a separate cohort. Regardless, a rapid, reliable test is needed. uneven distribution of available isolation and treatment beds within each country, particularly in remote areas, leaving some populations without sufficient access to appropriate care. 8 Importance WHO, Médecins Sans Frontières, and other organizations working with viral hemorrhagic fevers have previously developed clinical case definitions for Ebola virus disease infection. These definitions were based on expert consensus about the most useful clinical and epidemiologic predictors of Ebola virus disease in the context of previous epidemics (Appendix E1, available online at com). 9,10 Given the nonspecific nature of initial clinical symptoms, as well as barriers to confirmatory laboratory testing, the diagnosis of Ebola virus disease remains both a significant challenge and an absolute necessity in the context of an epidemic. 11,12 Evaluations conducted after previous Ebola and Marburg virus outbreaks have shown potential for modification of the clinical case definitions to increase their accuracy for detection of cases. 13,14 Novel rapid detection kits for Ebola virus disease are currently undergoing trials, 15 but until they are validated, more research is needed to aid in the early diagnosis and management of patients with suspected Ebola virus disease. Levine et al Goals of This Investigation Although previous research has described the clinical presentation and epidemiologic characteristics of patients with Ebola virus disease, to our knowledge no previous study has empirically derived a clinical prediction model that can be used to objectively risk-stratify patients with suspected Ebola virus disease before confirmatory laboratory testing The goal of this study is to develop a clinical prediction model that can help guide care for patients with suspected Ebola virus disease, provide specific parameters for isolation and admission to treatment centers, and maximize resource use. MATERIALS AND METHODS Study Design This is a retrospective cohort study of deidentified patient data collected at the Bong County Ebola Treatment Unit in Suakoko, Liberia, from September 15, 2014, to January 4, This study received an exemption from ethical review by the Lifespan (Rhode Island Hospital) Institutional Review Board. Setting and Selection of Participants The 52-bed Ebola treatment unit, managed by the humanitarian organization International Medical Corps in cooperation with the local Bong County Ministry of Health and Social Welfare, primarily serves the rural population of Bong County and neighboring counties, which at the time lacked their own Ebola virus disease isolation and treatment facilities. Patients arrived at the Ebola treatment unit in one of 3 ways: by an International Medical Corps ambulance, by a government or private ambulance, or by their own means of transport (private car, taxi, or walking). Patients brought by an International Medical Corps ambulance were screened by trained Ebola treatment unit staff before transport to ensure that they met the organization s suspect case definition for Ebola virus disease, which was created with WHO and Médecins Sans Frontières guidelines, as well as in consultation with local health authorities (Appendix E1, available online at Patients brought by other ambulances may have also been screened with a similar case definition before transport. Methods of Measurement Patient data, including basic demographic information and clinical symptoms, were recorded at admission on standardized paper forms by trained local or international nurses, physician assistants, or physicians, according to the subjective report of patients or their family members. To 286 Annals of Emergency Medicine Volume 66, no. 3 : September 2015

3 Levine et al protect staff from Ebola virus disease transmission, no physical examinations were performed at triage. All clinical staff underwent a comprehensive training covering all aspects of Ebola virus disease diagnosis and management, including practical training in the collection of health data from patients, before working in the Ebola treatment unit. The data collected at admission were transferred to an electronic database (Microsoft Excel, version 14.0) daily and updated as laboratory test results and final outcomes became available. All data were collected as part of routine clinical care and for epidemiologic purposes. After admission, patients were brought to the suspecteddisease ward of the Ebola treatment unit and had a blood sample drawn within 24 hours for initial Ebola virus disease testing. Patients with an initial negative test result who had symptoms for fewer than 3 days were held for repeated testing until 3 days had passed since the onset of their symptoms. Patients with a second negative test result after having symptoms for more than 3 days were discharged home from the suspected-disease ward or transferred to another health facility for further care. Patients with a positive test result were moved to the confirmed ward of the Ebola treatment unit for further management. All patients were treated according to standard treatment protocols based on guidelines developed by WHO and Médecins Sans Frontières. 9,10 This included empiric antimalarial treatment, broad-spectrum antibiotics, and nutritional supplementation for all patients, as well as symptomatic treatment for pain, nausea, and delirium. Patients with anorexia, vomiting, or diarrhea were treated with oral or intravenous fluid rehydration according to the severity of their dehydration. Patients who recovered were discharged once major symptoms resolved and blood testing result for Ebola virus disease was confirmed negative. Patients who died were buried in a designated area near the Ebola treatment unit. Outcome Measures Laboratory diagnosis of Ebola virus disease was performed at the United States Naval Medical Research Center Mobile Laboratory in Bong County, Liberia, with the 1-step quantitative Ebola Zaire real-time reverse transcriptase polymerase chain reaction (TaqMan) assay (Naval Medical Research Center, Frederick, MD). Briefly, Qiagen buffer AVL/ethanol-inactivated blood samples were extracted with QIAamp Viral RNA Mini Kit. Extracted ribonucleic acid was tested for 2 Ebola virus disease gene targets (EBOV Zaire locus and minor groove binding locus), using the Applied Biosystems StepOnePlus instrument. A sample was confirmed to be positive for Ebola virus disease if both targets were detected. Ebola Prediction Score To maximize data quality, a check on the fidelity of data entry was conducted by taking a systematic random sample of 10% of patient files and comparing the data entered in the electronic database with the original paper-based forms for all variables and cases sampled. The initial data quality check found an error rate of 7.4%. In response, the research team conducted a line-by-line review of each patient file, correcting any discrepancies between the paper charts and the electronic record. A second data quality check conducted after this review found an error rate of 2.7%, which was deemed acceptable for analytic purposes. Before our analysis, all data were deidentified by removing any unique patient identifiers, including name, identification, telephone number, and village, as well as the name and village of potential sick contacts. Primary Data Analysis Baseline historical and demographic characteristics were summarized for all admitted patients and compared for patients with and without laboratory-confirmed Ebola virus disease, as were clinical outcomes. All 14 clinical and epidemiologic variables commonly used in the current WHO and Médecins Sans Frontières case definitions for Ebola virus disease were chosen for analysis. 9,10 These included fever, nausea or vomiting, diarrhea, fatigue, abdominal pain, anorexia, muscle pain, joint pain, headache, difficulty breathing, difficulty swallowing, hiccups, unexplained bleeding, and recent sick contact, which was defined as direct or indirect contact with a patient with suspected or confirmed Ebola virus disease in the previous 21 days, including living in the same household or providing direct care for the patient. All variables were coded as present or not present. For each of these 14 variables, a bivariate analysis was performed to test the association of each variable with the final diagnosis of laboratory-confirmed Ebola virus disease. The test characteristics were also calculated for each predictor, including positive predictive value, negative predictive value, sensitivity, specificity, and positive and negative likelihood ratios, for the primary outcome of laboratoryconfirmed Ebola virus disease. Standard guidelines from the literature, including the recently published Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis guidelines, were used to develop a clinical prediction model for Ebola virus disease All 14 potential predictor variables were entered into a multivariable logistic regression model for the outcome of laboratory-confirmed Ebola virus disease. A stepwise backward selection algorithm was used to derive a final clinical prediction model with a stopping rule of P <.05. Interaction terms were not included in the Volume 66, no. 3 : September 2015 Annals of Emergency Medicine 287

4 Ebola Prediction Score model because of a lack of previous knowledge about effect modification among clinical predictors of Ebola virus disease. Because only 2 patients were missing data on potential predictor variables (1 patient missing data on fever and 1 patient missing data on sick contact), we used casewise deletion to handle missing data instead of single or multiple imputation. The final logistic regression model was then converted into a scoring system by ordering the final predictors in tabular format and converting the log odds ratio for each variable into an integer score as described previously. 24 An overall score was assigned to each patient according to the sum of the individual variable scores for each variable in the final model. The discrimination of this scoring system was assessed against the criterion standard of laboratory-confirmed Ebola virus disease with receiver operator characteristic curves. In the absence of an external validation cohort, internal validation was performed with the bootstrap method to assess the optimism of the clinical prediction model as recommended in the literature. 20,21 A bootstrap (with replacement) sample was randomly selected from the study population and used to fit the model, using the same backward selection algorithm with a stopping rule of P <.05. The area under the receiver operator characteristic curve (AUC) for this model was calculated in the bootstrap sample and then again in the full data set. This process was repeated 1,000 times, and the average difference between the AUCs for the bootstrap samples and the full data set was used to calculate an unbiased estimate of the overall optimism of the model. This estimate of optimism was then subtracted from the AUC for the original prediction model in the full data set to calculate a best estimate for the accuracy of the original prediction model in a new population. The test characteristics for the newly developed scoring system were assessed at a range of different cutoff values against the criterion standard of laboratory-confirmed Ebola virus disease. The test characteristics for the WHO suspect case definition were also assessed against the criterion standard of laboratory-confirmed Ebola virus disease. All bivariate analyses and figures were produced with Stata (version 12.0; StataCorp, College Station, TX). Model fitting, performance assessment, and internal validation were conducted with R version (R Development Core Team, Vienna, Austria). Although there is no formal method for calculating the study sample size for the development of a clinical prediction model, a general rule in the literature calls for at least 10 positive events per variable considered for the model. 20,21 Given 14 candidate predictors, this would require a minimum of 140 positive outcomes, or 140 Levine et al patients with laboratory-confirmed Ebola virus disease, to develop a stable prediction model. RESULTS Characteristics of Study Subjects and Main Results A total of 395 patients were admitted to the Bong County Ebola Treatment Unit during its first 16 weeks of operation. Definitive laboratory testing results were available for 382 patients (97%), of whom 160 (42%) tested positive for Ebola virus disease. Of patients with laboratory-confirmed Ebola virus disease, 82 (51%) died during their Ebola treatment unit admission. Among admitted patients, 332 (84%) met the WHO suspect case definitionforebolavirusdisease(appendix E1, available online at Figure 1 summarizes patient presentations by week of Ebola treatment unit operation. Table 1 demonstrates baseline characteristics, including age, sex, occupation, mode of transport, county of symptom onset, and number of days of symptoms for all patients admitted to the Ebola treatment unit, separated by Ebola virus disease status. In addition, the table also demonstrates the differences in outcomes, including both Ebola treatment unit length of stay and overall mortality, based on Ebola virus disease status. Table 2 demonstrates the positive predictive value, negative predictive value, sensitivity, specificity, and positive and negative likelihood ratios for each of the 14 commonly elicited clinical and epidemiologic characteristics in study patients. Characteristics with the highest sensitivity, such as fever and fatigue, tended to have low specificity, whereas those with the highest specificity, such as hiccups and bleeding, tended to have low sensitivity. Figure 1. Total patient admissions by week. EVD, Ebola virus disease. 288 Annals of Emergency Medicine Volume 66, no. 3 : September 2015

5 Levine et al Table 1. Demographic characteristics and outcomes of patients by Ebola virus disease status. Variable EVD Positive EVD Negative Demographic characteristics Male sex, No./total No. (%) 74/160 (46) 128/222 (58) Median age (95% CI), y 33 (5 60) 30 (3 60) Occupation, No./total No. (%) Farmer 65/141 (46) 69/172 (40) Child/student 36/141 (26) 48/172 (28) Business person 13/141 (9) 19/172 (11) Health worker 7/141 (5) 4/172 (2) Other 20/141 (14) 32/172 (19) County of residence, No./total No. (%) Bong 75/160 (47) 104/215 (48) Margibi 52/160 (33) 71/215 (33) Other 33/160 (21) 40/215 (19) Transport to ETU, No./total No. (%) IMC ambulance 107/149 (72) 141/214 (66) Other ambulance 39/149 (26) 57/214 (27) Self 3/149 (2) 16/214 (7) Median interval of symptoms before 3(1 11) 3 (0 18) admission (95% CI), days Outcomes Median ETU length of stay (95% CI), days 8 (2 19) 1 (0 4) Mortality, No./total No. (%) 82/160 (51) 12/222 (5) IMC, International Medical Corps; CI, confidence interval; ETU, Ebola treatment unit. The full logistic regression model with all 14 variables had an AUC of 0.76 (95% confidence interval 0.71 to 0.81) for the prediction of laboratory-confirmed Ebola virus disease. After application of a backward selection algorithm, 6 variables were retained in the final model: sick contact, diarrhea, anorexia, muscle pain, difficulty swallowing, and abdominal pain. Applying a forward selection algorithm and a mixed selection algorithm to the full model produced identical final models. Table 3 Ebola Prediction Score summarizes the regression coefficients, odds ratios, and confidence intervals for the 6 variables retained in the final model, as well as the individual scores assigned to the variables according to their log odds ratios. The Ebola Prediction Score, calculated by summing the individual points for each of these 6 variables, had an AUC of 0.75 (95% confidence interval 0.70 to 0.80) for the prediction of laboratory-confirmed Ebola virus disease. Table 4 demonstrates the test characteristics for the Ebola Prediction Score at several cutoff values, whereas Figure 2 demonstrates the progressive increase in the proportion of patients with laboratory-confirmed Ebola virus disease with each 1-point increase in the Ebola Prediction Score. Table 4 also demonstrates the test characteristics of the WHO algorithm for predicting Ebola virus disease in study patients. The WHO algorithm demonstrated test characteristics similar to those of the Ebola Prediction Score at a score cutoff greater than or equal to 1. Of 160 patients with laboratory-confirmed Ebola virus disease, an Ebola Prediction Score greater than or equal to 1 would have missed 4 Ebola virus disease positive patients captured by the WHO algorithm, whereas the WHO algorithm would have missed 7 disease-positive patients captured by the Ebola Prediction Score. Both the WHO algorithm and the Ebola Prediction Score would have missed an additional 5 patients with laboratory-confirmed Ebola virus disease. Internal validation was performed by randomly drawing 1,000 bootstrap samples from the original data set and fitting a prediction model for Ebola virus disease, using the same 14 variables and backward selection algorithm in each sample. The average AUC for the prediction models developed in the bootstrap samples was 0.76, whereas the average AUC for these same models applied to the full data set was 0.73, yielding an estimate of optimism Table 2. Clinical predictors of Ebola virus disease. Percentage (95% CI) Clinical Predictors Sensitivity Specificity PPV NPV LRD LR Fever 85 (79 91) 21 (16 27) 44 (38 49) 66 (60 73) 1.1 ( ) 0.7 ( ) Nausea/vomiting 56 (48 63) 58 (52 65) 49 (42 56) 65 (57 72) 1.3 ( ) 0.8 ( ) Diarrhea 56 (49 64) 70 (64 76) 58 (50 65) 69 (61 77) 1.9 ( ) 0.6 ( ) Fatigue 83 (77 89) 30 (24 36) 46 (40 52) 71 (65 78) 1.2 ( ) 0.6 ( ) Abdominal pain 56 (49 64) 45 (39 52) 42 (36 49) 59 (52 66) 1.0 ( ) 1.0 ( ) Anorexia 76 (70 83) 45 (39 52) 50 (44 56) 72 (66 79) 1.4 ( ) 0.5 ( ) Muscle pain 56 (49 64) 57 (50 63) 48 (41 56) 64 (57 72) 1.3 ( ) 0.8 ( ) Joint pain 53 (45 61) 50 (43 57) 43 (36 50) 60 (53 67) 1.1 ( ) 0.9 ( ) Headache 64 (56 71) 39 (32 45) 43 (37 49) 60 (53 67) 1.0 ( ) 0.9 ( ) Difficulty breathing 26 (19 32) 77 (71 82) 44 (34 54) 59 (51 66) 1.1 ( ) 1.0 ( ) Difficulty swallowing 35 (28 42) 81 (75 86) 57 (47 66) 63 (55 71) 1.8 ( ) 0.8 ( ) Hiccups 9 (5 14) 91 (87 95) 43 (26 59) 58 (51 66) 1.0 ( ) 1.0 ( ) Bleeding 8 (4 12) 94 (90 97) 48 (29 67) 59 (51 66) 1.3 ( ) 1.0 ( ) Sick contact 65 (58 73) 61 (54 67) 54 (47 62) 71 (64 79) 1.7 ( ) 0.6 ( ) PPV, Positive predictive value; NPV, negative predictive value; LRþ, positive likelihood ratio; LR, negative likelihood ratio. Volume 66, no. 3 : September 2015 Annals of Emergency Medicine 289

6 Ebola Prediction Score Levine et al Table 3. Ebola Prediction Score. Variable Regression Coefficient (95% CI) Odds Ratio (95% CI) Assigned Score Sick contact 1.23 (0.76 to 1.69) 3.41 (2.14 to 5.42) 2 Diarrhea 0.96 (0.47 to 1.45) 2.62 (1.60 to 4.30) 1.5 Anorexia 0.74 (0.22 to 1.27) 2.12 (1.25 to 3.60) 1 Muscle pain 0.62 (0.15 to 1.09) 1.87 (1.16 to 3.00) 1 Difficulty 0.62 (0.08 to 1.15) 1.86 (1.09 to 3.18) 1 swallowing Abdominal pain 0.64 ( 1.15 to 0.13) 0.52 (0.31 to 0.87) 1 of Subtracting this estimate of optimism from the performance of the Ebola Prediction Score in the full data set yielded a validated estimate of 0.72 for the AUC of the Ebola Prediction Score compared with the criterion standard of laboratory-confirmed Ebola virus disease. LIMITATIONS This study includes data on patients admitted to a single Ebola treatment unit in rural Liberia. As such, the demographic and epidemiologic characteristics of our study population may not be representative of all patients with Ebola virus disease in this current epidemic or future ones, although they are generally in line with those reported in other recent studies. 16,17,25 Although prospective validation in a new study cohort would be ideal, that may not be possible in the context of the current epidemic. Moreover, internal validation using bootstrap sampling with 1,000 repetitions suggests that the Ebola Prediction Score would perform similarly in a new study population. Because of the limited amount of evidence in the literature on known predictors and terms to assess for effect modification, we did not assess interaction terms and had to use a model selection method based on statistical significance. Thus, our results and conclusions should be viewed as exploratory due to the nature of our multivariable modeling, as well as the lack of an external validation cohort. Figure 2. Patients categorized by Ebola Prediction Score. Data quality remains another important limitation. The data for this study were collected for clinical and epidemiologic purposes under extraordinarily difficult circumstances. Although our final data quality check demonstrated a secondary data entry error rate of 2.7% (from the paper chart to the database), there remains an unquantifiable data entry error rate in the primary collection of patient data (from the patient to the paper chart). In addition, laboratory data were missing for 13 study patients (3%) and data on potential predictor variables were missing for 2 study patients (<1%). Because of the rural nature of Bong County, which makes independent travel difficult, two thirds of patients in this study were brought to the Ebola treatment unit by an International Medical Corps ambulance and were therefore screened before arrival by an International Medical Corps algorithm similar to the WHO suspect case definition (Appendix E1, available online at com). As such, it is difficult to directly compare the performance of the WHO algorithm to that of the Ebola Prediction Score in this patient population. An independent evaluation in a new population of unscreened patients would be required to make a fair comparison. However, the goal Table 4. Performance of Ebola Prediction Score at each cut point and the WHO suspect algorithm. Percentage (95% CI) Total Score PPV NPV Sensitivity Specificity LRD LR EPS 1 46 (40 51) 83 (77 88) 94 (91 98) 19 (14 25) 1.2 ( ) 0.3 ( ) EPS 2 50 (44 56) 76 (70 83) 82 (76 88) 42 (35 48) 1.4 ( ) 0.4 ( ) EPS 3 64 (57 72) 75 (67 83) 65 (58 73) 74 (68 80) 2.5 ( ) 0.5 ( ) EPS 4 82 (73 90) 69 (60 78) 42 (34 50) 93 (90 97) 6.2 ( ) 0.6 ( ) WHO algorithm 46 (41 52) 81 (75 86) 93 (88 97) 23 (17 28) 1.2 ( ) 0.3 ( ) EPS, Ebola Prediction Score. 290 Annals of Emergency Medicine Volume 66, no. 3 : September 2015

7 Levine et al of this study was not to demonstrate the superiority of the Ebola Prediction Score to current case definitions in use, but rather to demonstrate its utility as an additional tool to help clinicians risk-stratify patients already meeting 1 or more suspect case definitions for Ebola virus disease. Finally, the median time from symptom onset to Ebola treatment unit arrival in this study population was 3 days, with an interquartile range of 2 to 6 days. As such, the Ebola Prediction Score may perform differently for patients who present for clinical care later in their disease process, when the clinical characteristics of Ebola virus disease begin to change. According to these data, use of the Ebola Prediction Score should be limited to patients presenting within the first week of symptoms. DISCUSSION Management of an Ebola treatment unit in the context of an Ebola virus disease epidemic requires balancing several conflicting imperatives. From the public health perspective, ending the overall epidemic requires identifying and isolating all patients with Ebola virus disease, requiring a highly sensitive algorithm for identifying patients with suspected disease. From a clinical perspective, however, admitting a patient to an Ebola treatment unit or community-based isolation center who is unlikely to have Ebola virus disease puts that patient at risk for nosocomial infection and makes it far less likely that his or her true illness will be identified and appropriately treated in a timely manner. 26 Finally, from a logistical perspective, patient numbers may often overwhelm available resources, especially during the early phase of an epidemic when both Ebola treatment unit beds and trained health workers to staff them are limited. 5-7 Determining which patients to admit to an Ebola treatment unit for definitive testing and treatment requires balancing the epidemiologic imperative to break the train of transmission in the community against the ethical imperative to do no harm to each patient, all within the context of severe resource constraints. Our data show that the current WHO algorithm does an excellent job of meeting the first imperative described above. With a negative predictive value of 83%, patients who do not meet the criteria for being a suspected case based on this algorithm are unlikely to test positive for Ebola virus disease. However, with a positive predictive value of only 46%, most patients who do meet the suspected-case criteria for Ebola virus disease will still eventually test negative for the virus. Often, definitive laboratory testing may not be available on site at the Ebola treatment unit, and patients with symptoms for fewer than 3 days will generally still have to wait for a second negative test result before discharge. 27 This means that patients without Ebola virus disease will likely spend at Ebola Prediction Score least 1 and potentially several days in the Ebola treatment unit, during which time they are at risk for nosocomial transmission of the disease. To our knowledge, the Ebola Prediction Score is the first clinical tool for risk stratification of patients with suspected Ebola virus disease developed according to empirical data collected during an Ebola virus disease outbreak. Several previous studies, including ones conducted during this current epidemic, have examined the prevalence of various clinical and epidemiologic variables in patients with confirmed Ebola virus disease, but none have evaluated their test characteristics or combined them into a multivariable regression model. 16,17,25 A single previous study by Roddy et al 14 developed a clinical prediction model for Marburg virus based on data collected from 102 patients presenting to a single hospital in Uige, Angola, during a Marburg virus outbreak in Their final prediction model included the variables of sick contact, myalgia or arthralgia, and hemorrhage. The WHO algorithm was also found to have good sensitivity (79%) but poor specificity (39%) for the prediction of laboratoryconfirmed Marburg virus in their study population. Sick contact with a suspected or confirmed Ebola virus disease patient was the strongest independent predictor of Ebola virus disease in our study, as in the study by Roddy et al. 14 Diarrhea, loss of appetite, muscle pains, and difficulty swallowing were also statistically significant positive predictors of Ebola virus disease in our study population. Although not associated with Ebola virus disease in bivariate analysis, abdominal pain was found to be a significant negative predictor of Ebola virus disease when taken in the context of all other signs, perhaps because many patients who tested negative for Ebola virus disease actually had another disease such as typhoid fever, which is more likely to cause severe abdominal pain than Ebola virus disease. Other commonly held predictors of Ebola virus disease, such as fever and unexplained bleeding, were not found to be significant, either alone or in combination with other signs, in our study population. Although fever was a highly sensitive sign of Ebola virus disease, it was not useful in discriminating it from all the many other infectious diseases common in the region. In the case of unexplained bleeding, it is likely that the finding is simply too rare early in the course of the disease to be helpful as part of a screening tool. In accordance with our data, we cannot recommend that the Ebola Prediction Score supplant any of the Ebola virus disease case definitions currently in use. Instead, we see the utility of the score in helping clinicians to further riskstratify patients who have already been identified as having suspected disease. This risk stratification can help clinicians to form cohorts of patients with similar levels of risk, ideally Volume 66, no. 3 : September 2015 Annals of Emergency Medicine 291

8 Ebola Prediction Score in separate wards with separate latrines or toilets, but at the very least in separate rooms or areas within an Ebola treatment unit or community-based isolation center. Additionally, when the numbers of suspect cases overwhelm available bed capacity, the Ebola Prediction Score can be used as a triage tool to determine which patients with suspected disease should be admitted and which should be referred elsewhere. To our knowledge, this is the first study to empirically derive and internally validate a clinical prediction model for laboratory-confirmed Ebola virus disease. The Ebola Prediction Score can be used by clinicians in the context of an active Ebola virus disease epidemic for the purpose of cohorting, or separating, patients within an Ebola treatment unit or community-based isolation center or as an additional triage aid when available resources are overwhelmed. This study also highlights the importance of developing low-cost, point-of-care tests that can be used to rapidly and definitively exclude or confirm Ebola virus disease in patients because neither the WHO algorithm nor the Ebola Prediction Score was perfectly sensitive or specific for the disease. The authors acknowledge the International Medical Corps, the Bong County Ministry of Health and Social Welfare, the United States Naval Medical Research Center Mobile Laboratory, and Cuttington University for their support of this research project; the Liberian and international health workers providing patient care in the Bong County Ebola Treatment Unit; Jennifer Leaning, MD, MPH, Hilarie Cranmer, MD, MPH, and Patricia Henwood, MD, for their editorial assistance; and all those affected by Ebola virus disease during this current epidemic and the contribution they have made to helping us improve future care for patients with this disease. Supervising editor: Gregory J. Moran, MD Author affiliations: From the Warren Alpert School of Medicine, Brown University, Providence, RI (Levine, Glavis-Bloom, Wiskel); the College of Allied Health Sciences, Cuttington University, Suakoko, Liberia (Kesselly); and the International Medical Corps, Los Angeles, CA (Shetty, Burbach, Cheemalapati). Author contributions: ACL and PPS conceived the study and supervised data collection. RB, SC, and JKTK managed the data, including quality control. ACL and JG-B analyzed the data. ACL, JG-B, and TW drafted the article, and all authors contributed substantially to its revision. ACL takes responsibility for the paper as a whole. Funding and support: By Annals policy, all authors are required to disclose any and all commercial, financial, and other relationships in any way related to the subject of this article as per ICMJE conflict of interest guidelines (see The authors have stated that no such relationships exist and provided the following details: Levine et al Although funding from the United States Agency for International Development Office of Foreign Disaster Assistance (OFDA) supported clinical operations at the Bong County Ebola Treatment Unit, no direct financial support was provided for this research study. Publication dates: Received for publication February 3, Revision received March 1, Accepted for publication March 12, Available online April 3, REFERENCES 1. Emond RT, Evans B, Bowen ET, et al. A case of Ebola virus infection. Br Med J. 1977;2: World Health Organization. Statement on the 1st meeting of the IHR Emergency Committee on the 2014 Ebola outbreak in West Africa. 2014:1-5. Available at: statements/2014/ebola /en/. Accessed February 1, World Health Organization. Statement on the 3rd meeting of the IHR Emergency Committee regarding the 2014 Ebola outbreak in West Africa. 2014:1-4. Available at: statements/2014/ebola-3rd-ihr-meeting/en/. Accessed February 1, World Health Organization. Ebola virus disease. Available at: Published September 1, Accessed February 1, Martin-Moreno JM, Ricciardi W, Bjegovic-Mikanovic V, et al. Ebola: an open letter to European governments. Lancet. 2014;384: Rebaudet S, Moore S, Piarroux R. Ebola virus disease in West Africa the first 9 months. N Engl J Med. 2015;372: Farrar JJ, Piot P. The Ebola emergency immediate action, ongoing strategy. N Engl J Med. 2014;371: World Health Organization. Ebola Situation Report: 21 January :1-13. Available at: /1/roadmapsitrep_21Jan2015_eng.pdf. Accessed February 1, Médecins Sans Frontières. Filovirus Haemorrhagic Fever Guideline. MSF, Barcelona, Spain: 2008: World Health Organization. Clinical management of patients with viral haemorrhagic fever: a pocket guide for the front-line health worker. 2014: Available at: /2/WHO_HSE_PED_AIP_14.05.pdf. Accessed February 1, Kelly JD. Make diagnostic centres a priority for Ebola crisis. Nature. 2014;513: Dhillon RS, Srikrishna D, Sachs J. Controlling Ebola: next steps. Lancet. 2014;384: Pittalis S, Fusco FM, Lanini S, et al. Case definition for Ebola and Marburg haemorrhagic fevers: a complex challenge for epidemiologists and clinicians. New Microbiol. 2009;32: Roddy P, Thomas SL, Jeffs B, et al. Factors associated with Marburg hemorrhagic fever: analysis of patient data from Uige, Angola. J Infect Dis. 2010;201: Vogel G. Infectious diseases. Testing new Ebola tests. Science. 2014: Bah EI, Lamah M-C, Fletcher T, et al. Clinical presentation of patients with Ebola virus disease in Conakry, Guinea. N Engl J Med. 2015;372: Schieffelin JS, Shaffer JG, Goba A, et al. Clinical illness and outcomes in patients with Ebola in Sierra Leone. N Engl J Med. 2014;371: Ansumana R, Jacobsen KH, Idris M, et al. Ebola in Freetown Area, Sierra Leone a case study of 581 patients. N Engl J Med. 2015;372: Annals of Emergency Medicine Volume 66, no. 3 : September 2015

9 Levine et al Ebola Prediction Score 19. Dallatomasinas S, Crestani R, Squire JS, et al. Ebola outbreak in rural West Africa: epidemiology, clinical features and outcomes. Trop Med Int Health. 2015;20: Harrell FE, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15: Stiell IG, Wells GA. Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med. 1999;33: Royston P, Moons KGM, Altman DG, et al. Prognosis and prognostic research: developing a prognostic model. BMJ. 2009;338:b Collins GS, Reitsma JB, Altman DG, et al. Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD Statement. Ann Intern Med. 2015;162: Sullivan LM, Massaro JM, D Agostino RB. Presentation of multivariate data for clinical use: the Framingham Study risk score functions. Stat Med. 2004;23: WHO Ebola Response Team. Ebola virus disease in West Africa the first 9 months of the epidemic and forward projections. N Engl J Med. 2014;371: Shears P, O Dempsey TJD. Ebola virus disease in Africa: epidemiology and nosocomial transmission. J Hosp Infect. 2015: O Dempsey T, Khan SH, Bausch DG. Rethinking the discharge policy for Ebola convalescents in an accelerating epidemic. Am J Trop Med Hyg. 2015;92: Links to Additional Resources Does the treatment or medicine you offer your patients have real impact? How do you know? The Annals Web site provides a link to The NNT, or Number Needed to Treat, Web site ( The NNT offers a measurement of the impact of a medicine or therapy by estimating the number of patients that need to be treated in order to have an impact on one person. Go to the Resources pull-down menu on the Annals Web site and click on The NNT link. Volume 66, no. 3 : September 2015 Annals of Emergency Medicine 293

10 Ebola Prediction Score Levine et al APPENDIX E1 Ebola virus disease suspected-case definitions Figure E3. International Medical Corps (IMC) algorithm. Figure E1. WHO algorithm.* *World Health Organization. Clinical management of patients with viral haemorrhagic fever: a pocket guide for the front-line health worker. 2014: Available at: /2/WHO_HSE_PED_AIP_14.05.pdf. Figure E2. Médecins Sans Frontières algorithm.* *Médecins Sans Frontières. Filovirus Haemorrhagic Fever Guideline. Barcelona, Spain; 2008: e1 Annals of Emergency Medicine Volume 66, no. 3 : September 2015