The quality of reports of randomized trials in multiple sclerosis: a review

Transcription

1 444327MSJ / Signori et al.multiple Sclerosis Journal 2012 Topical Review MULTIPLE SCLEROSIS JOURNAL MSJ The quality of reports of randomized trials in multiple sclerosis: a review Multiple Sclerosis Journal 18(6) The Author(s) 2012 Reprints and permissions: sagepub.co.uk/journalspermissions.nav DOI: / msj.sagepub.com Alessio Signori, Alice Baccino and Maria Pia Sormani Abstract Randomized clinical trials (RCT) in multiple sclerosis (MS) have a recent tradition, but their number has been exponentially increasing since the first study detecting the efficacy of a disease modifying drug in MS. To examine the methodological details of reports of RCT in MS, we extracted from five leading journals all the reports of RCT published between 1993 and Trial reports were compared for different periods ( , , ) for a set of indicator variables reflecting methodological quality (including details about randomization and blinding, statistical methods, results reporting, subgroup analyses). Fifty-three reports were included in the analysis. All of the methodological items indicated an improvement over time in the quality of reporting, the main weaknesses being frequent and inappropriate use of significance testing for assessing baseline imbalances and the statistical approach to subgroup analysis. A complete and transparent reporting of trial methodology is becoming even more important in an era when new design strategies are required for the feasibility of future trials in MS. Keywords Randomized clinical trials, CONSORT statement, clinical trials methodology, subgroup analysis Date received: 9th February 2012; revised: 29th February 2012; 11th March 2012; accepted: 13th March 2012 Introduction The first positive trial assessing the efficacy of a disease modifying drug (DMD) in multiple sclerosis (MS) is dated in fixing at that year the start of the era of randomized clinical trials (RCT) in MS, much later than, for example, oncology which boasts a long tradition in RCT since the late 1950s. Since that year, many RCT have been conducted in MS and a standard methodology has been consolidated ranging from phase II placebo-controlled trials, typically lasting 6 months with Magnetic Resonance Imaging (MRI) endpoints, to phase III placebo-controlled trials lasting 1 2 years with relapse or disability progression endpoints. This process has brought to the market new drugs as alternatives to Interferon, and other therapies will become available in the near future. So we have entered a new era of RCT in MS: since patients are enrolled in trials which are more and more benign and the use of placebo has become ethically questionable 2 new design strategies are required to handle this evolution, and a large effort is devoted to study, among other strategies, the validation of surrogate outcomes 3,4 and the application of adaptive designs. 5,6 Therefore, after about 20 years of RCT history in MS, the aim of this paper is to review the quality of RCT in MS reports from 1993 to In this last decade, the quality of RCT reporting has in general constantly improved, with many journals having statistical refereeing and clearer guidelines to authors. The Consolidated Standards of Reporting Trials (CONSORT) Statement, first published in and revised in and provides recommendations for authors regarding how to prepare reports of trial findings. In this paper we assessed the methodology of RCT reports in MS using indicators of quality based mainly on the explicit reporting of details of trial design and conduction, statistical analysis and the display of results. Material and methods We searched reports of RCT in MS published between 1993 and July 2010 in The Lancet, Lancet Neurology, Annals of Neurology, Neurology and The New England Department of Health Sciences, University of Genoa, Italy Corresponding author: Maria Pia Sormani, Department of Health Sciences (DISSAL), University of Genoa, Via Pastore 1, Genoa, 16132, Italy. mariapia.sormani@unige.it

2 Signori et al. 777 Journal of Medicine. We screened electronic databases (Ovid MEDLINE (1993 July 2010), PubMed (1993 July 2010), the Cochrane Central Register of Controlled Trials (1993 July 2010) to identify trials fulfilling the following inclusion criteria: randomized, phase II or phase III, placebo or active-controlled trials in MS, assessing the efficacy of DMD of any class. Reviews, observational studies and secondary publications were excluded. Two researchers (AB and AS) independently extracted data relevant for the analysis and any discrepancies were checked by a third assessor (MPS) and resolved by discussion. MS type, trial phase, number of centers, treatment arms, sample size and follow-up length were extracted as general items. The presence of the term randomized in the title, systems to implement randomization and method to generate randomization lists, blinding methodology, explicit definition of the primary outcome, intention to treat (ITT) analysis and details about calculation of sample size were extracted as methodological items. The presence of a flow diagram, details on patients lost to follow-up, reporting and comparison of baseline characteristics between randomized group and possible imbalances noticed, covariate adjustment for primary outcome (pre-planned or not) and which analysis (unadjusted or adjusted) received more emphasis were collected as items related to presentation of results. The number and proportion (%) of trials reporting different items of the CONSORT statement were reported. Years of publication were categorized in three classes, as , , SPSS (v.18; IBM Corporation) was used for computation. Results A total of 288 studies were screened. Of these, 226 were excluded mainly because they were solely observational studies or they were focused on symptomatic drugs. Among the remaining 62 papers, nine were excluded after reading the full articles (Figure 1) and 53 trials were finally included: 23 published in Neurology, 10 in The Lancet, 9 in The New England Journal of Medicine, 7 in Lancet Neurology and 4 in Annals of Neurology. Seventeen studies were published in , 17 in and the remaining 19 studies in A table listing all of the studies included in this review is reported in the Supplementary Material. General characteristics Table 1 reports the general characteristics of the included studies: 29 (54.7%) were on relapsing remitting (RR) MS patients, 8 (15.1%) on secondary progressive (SP) MS patients, 4 (7.5%) on clinically isolated syndrome (CIS) subjects and the remaining 12 (22.6%) on a mixture of RR, SP and primary progressive (PP) MS patients; 23 studies (43.4%) were phase II, 30 (56.6%) were phase III studies. The vast majority of RCT in MS was sponsored by Figure 1. Flowchart of the procedure for the selection of reports of clinical. industry: in our sample 42 (79%) trials were sponsored by a single company, 7 (13%) had a mixed funding (industry and academia or MS societies) and only 3 (6%) were sponsored by non-profit institutions. The median number of patients enrolled was 297 (range ) with a median follow-up of two years (range 6 36 months). Just four trials were monocentric, and the median number of centers involved in multicenter studies was 32 (range 1 198). The number of centers involved was significantly higher in (median=50) as compared to (median=37) and (median=11, p=0.001, test for trend). Twenty-eight trials (52.8%) had two arms, 22 trials (41.5%) had three arms, two trials (3.8%) had four arms and one trial (1.9%) had five arms. Overall 44 trials (83%) were placebo-controlled. Primary outcomes were MRI outcomes (41.5%), relapses (26.4 %) or disability (22.6%); other outcomes were both relapses and disability (7.5%) or relapses and MRI (1.9%). Methodological characteristics Table 2 shows information collected on methodological items. The term randomized is reported after 2001 (55.6% of trials) slightly more frequently than in the period (35.3%, p= 0.28). Most of the trials (about 83%) did not report the systems of allocation concealment and, if quoted, the interactive voice responder resulted the most adopted method (13.2%). The statistical method used to generate the randomization list was specified in 43 trials (81.1%), with block randomization as the most used method (81.4%). In 25 out of the 49 multicenter studies (51%), randomization was balanced for center. Among the 17 trials for which a randomization balanced for baseline covariates was done, seven studies involved

3 778 Multiple Sclerosis Journal 18(6) Table 1. General characteristics of randomized trials. Total (n=53) (n=19) Type of multiple sclerosis (n, %) RR 29 (54.7%) 8 (47.1%) 8 (47.1%) 13 (68.4%) SP 8 (15.1%) 4 (23.5%) 3 (17.6%) 1 (5.3%) CIS 4 (7.5%) 2 (11.8%) 1 (5.9%) 1 (5.3%) RR-SP-PP 12 (22.6%) 3 (17.6%) 5 (29.4%) 4 (21.1%) Phase of trial (n, %) II 23 (43.4%) 7 (41.2%) 5 (29.4%) 11 (57.9%) III 30 (56.6%) 10 (58.8%) 12 (70.6%) 8 (42.1%) Sample size (median, IQR) 297 ( ) 251 (97 372) 318 ( ) 306 ( ) Centres (median, IQR) 32 (11 57) 11 (7 22) 37 (19 92) 50 (32 80) Follow-up length (median, IQR) 24 (12 24) 24 (9 26) 24 (12 24) 18 (12 24) Number of treatments (n, %) Two 28 (52.8%) 11 (64.7%) 9 (53%) 8 (42.1%) Three 22 (41.5%) 6 (35.3%) 8 (47%) 8 (42.1%) More than three 3 (5.7%) (15.8%) Placebo (n, %) 44 (83%) 17 (100%) 13 (76.5%) 14 (73.7%) CIS: clinically isolated syndrome; IQR: interquartile range; PP: primary progressive; RR: relapsing remitting; SP: secondary progressive only one baseline characteristic and a maximum of four features was balanced for. The baseline characteristics most used for adjustment were age, sex and Expanded Disability Status Scale (EDSS). In the majority of trials (90.6%) a double-blind procedure was implemented, with only one study conducted without any method of blinding. Among blinded trials, 49 (94.2%) reported details on who was blinded and in 31 (59.6%) details on how blinding was achieved were shown. Details about calculation of sample size were reported in 44 trials (83%). Primary outcome was predefined in all but two trials, both run in the period Thirty-four trials (64.2%) were positive trials, with a significant treatment effect. In 42 trials (79.2%) the ITT principle was adopted for data analysis, with an increase in more recent trials ( : 88.9%) as compared to older studies ( %, p=0.02). Trial reporting Table 3 gives information about the reports results sections. Eleven trials (20.8%) did not report any CONSORT diagram, eight of which were between 1993 and 2001 (p=0.002 for the comparison vs ). The number of levels of the diagrams increased from a median of four (range 0 7) in the period to a median of six (range 0 14, p=0.009). Almost all trials (52/53) reported information on the number of patients lost to follow-up, with 50 trials (94.3%) also reporting details about reasons for the loss to follow-up. All trials showed a table of baseline characteristics by treatment allocation. The number of baseline features varied widely with a median of 10 features and a maximum of 22 features. About half of the trials (28/53, 52.8%) assessed imbalances between treatment groups by significance tests. Imbalance, based on p<0 05, was claimed in six trials, for a total of eight significant baseline differences, which is 3% of 274 identifiable significance tests. The use of statistical tests to evaluate imbalances of baseline characteristics has not decreased in more recent years (p for heterogeneity=0.22, Table 3). Thirty-nine trials (73.6 %) analyzed primary outcomes with adjustment for some covariates and in 35.9% of trials adjustment was pre-planned. The number of covariates used to adjust for was never higher than 6 and adjustment for 2 4 covariates was the most frequent case. The majority of studies (76.9%) using a covariate adjustment for the primary outcome reported just the adjusted results and only in 15.4% of trials both unadjusted and adjusted results were shown; none of these trials found any difference between adjusted and unadjusted results. Correction for the center effect was applied to 20 (52.6%) out of the 37 multicenter trials including a covariate adjustment. Covariates frequently included into the main analysis were baseline EDSS score, previous years relapses or baseline MRI activity. The statistical methods most widely used were the Cox proportional hazard model or the log-rank test for time to progression of disability; the analysis of covariance (ANCOVA) was used for the number of MRI lesions or the rate of clinical relapses. In more recent years this model was replaced by generalized linear models, as Poisson regression (for relapse rate) and Negative Binomial regression (for MRI lesions). Fifteen trials (28.3%) included subgroup analysis in the principal report while other three trials had a subgroup analysis performed and published in separate papers. Among these 15 trials, four (26.7%) presented subgroup results without statistical tests, eight (53.3%) reported the p-values

4 Signori et al. 779 Table 2. Reporting of methodological characteristics. Total (n=53) (n=19) Randomized stated in title Yes 26 (49.1%) 6 (35.3%) 9 (53%) 11 (57.9%) No 27 (50.9%) 11 (64.7%) 8 (47%) 8 (42.1%) Method of allocation concealment Defined 9 (17%) 0 6 (35.3%) 3 (15.8%) Undefined 44 (83%) 17 (100%) 11 (64.7%) 16 (84.2%) Method to generate random sequence Specified 43 (81.1%) 10 (58.8%) 17 (100%) 16 (84.2%) Block randomization 35 (66%) 8 (47%) 13 (76.5%) 14 (73.7%) Minimization 6 (11.3%) 1 (5.9%) 3 (17.6%) 2 (10.5%) Other 2 (3.8%) 1 (5.9%) 1 (5.9%) 0 Unspecified 9 (16.9%) 7 (41.2%) 0 3 (15.8%) Randomization balanced for center 1 Yes 25 (51%) 6 (42.9%) 8 (50%) 11 (57.9%) No 14 (28.6%) 1 (7.1%) 8 (50%) 5 (26.3%) Unspecified 10 (20.4%) 7 (50%) 0 3 (15.8%) Randomization balanced for other baseline factors Yes 17 (32.1%) 3 (17.7%) 7 (41.2%) 7 (36.8%) No 24 (45.3%) 5 (29.4%) 10 (58.8%) 9 (47.4%) Unspecified 12 (22.6%) 9 (52.9%) 0 3 (15.8%) Blinding Double-blind 48 (90.6%) 17 (100%) 15 (88.2%) 16 (84.2%) Single blind 4 (7.5%) 0 2 (11.8%) 2 (10.5%) Details reported 49 (94.2%) 14 (82.4%) 17 (100%) 18 (100%) Details not reported 3 (5.8%) 3 (17.6%) 0 0 No blinding 1 (1.9%) (5.3%) Blinding - how achieved Reported 31 (59.6%) 8 (47.1%) 11 (64.7%) 12 (66.7%) Not reported 21 (40.4%) 9 (52.9%) 6 (35.3%) 6 (33.3%) Sample size calculation Yes 44 (83%) 12 (70.6%) 15 (88.2%) 17 (89.5%) No 9 (17%) 5 (29.4%) 2 (11.8%) 2 (10.5%) Primary outcome definition Yes 51 (96.2%) 15 (88.2%) 17 (100%) 19 (100%) No 2 (3.8%) 2 (11.8%) 0 0 Intention to treat Yes 38 (71.7%) 10 (58.8%) 14 (82.4%) 14 (73.7%) No 15 (28.3%) 7 (41.2%) 3 (17.6%) 5 (26.3%) 1 Excluded four studies run in a single center. of comparison between drugs in each subgroup while only three (20%) performed an interaction analysis to test for heterogeneity of treatment effect between the subgroups. All reports did subgroup analysis on the primary outcome of the study while in seven studies (46.7%) this was done on more than one outcome. Six reports (40%) claimed features from subgroup analysis on abstract or conclusions. Discussion The clinical trial scenario in MS has been rapidly changing in these last years. The advent of new effective therapies has made the use of placebo unethical, 2 at least in trials lasting more than six months. Methodological research in MS is focusing on the search for better clinical outcomes and valid surrogate markers to investigate the efficacy of new drugs with enough power. With this background, in this review we give a picture of the state of the art of methodology of RCT in MS, focusing on the quality of reporting in five leading journals. A detailed documentation of randomization methods was lacking in the first MS trial reports, but improved after the publication of the CONSORT statement in Methods used to generate the random sequence, details

5 780 Multiple Sclerosis Journal 18(6) Table 3. Reporting of results. Total (n=53) (n=19) Consort flow diagram Included 42 (79.2%) 9 (52.9%) 16 (94.1%) 17 (89.5%) Not included 11 (20.8%) 8 (47.1%) 1 (5.9%) 2 (10.5%) Loss to follow-up Reported 52 (98.1%) 16 (94.1%) 17 (100%) 19 (100%) Not reported 1 (1.9%) 1 (5.9%) 0 0 Statistical test for difference at baseline Yes 26 (49.1%) 8 (47.1%) 11 (64.7%) 7 (36.8%) Imbalance noted 6 (23.0%) 0 3 (50.0%) 3 (50.0%) No 27 (50.9%) 9 (52.9%) 6 (35.3%) 12 (63.2%) Covariate adjustment for primary outcome analysis Yes 39 (73.6%) 11 (64.7%) 12 (70.6%) 16 (84.2%) No 14 (26.4%) 6 (35.3%) 5 (29.4%) 3 (15.8%) Pre-planned adjustment 1 Yes 14 (35.9%) 2 (18.2%) 6 (50%) 6 (37.5%) No 25 (64.1%) 9 (81.8%) 6 (50%) 10 (62.5%) Analysis more emphasized 1 Unadjusted 3 (7.7%) 1 (9.1%) 2 (16.7%) 0 Adjusted 30 (76.9%) 9 (81.8%) 6 (50%) 15 (93.8%) Either 6 (15.4%) 1 (9.1%) 4 (33.3%) 1 (6.3%) 1 Considering only trials with covariate adjustment. about blinding, details about the balancing of randomization, assumptions for sample size calculation and application of the ITT analysis have all featured in improved documentation after This improvement can be imputed to the editorial policy of medical journals that, with the endorsement of the CONSORT statement, both helps authors in providing the clarity needed by readers wishing to assess a trial s validity, and also forces researchers and sponsors to routinely implement the basic procedures of good methodology of RCT. A questionable practice largely adopted is the use of significance tests for detecting baseline differences. The null hypothesis of no differences between the treatment arms is true by definition, since the randomization procedure guarantees that any baseline differences are due to chance alone. About half of the trials do such significance tests and this practice does not decrease in recent years; on the other hand, usually no information is given on whether baseline factors are associated with outcome, a more interesting insight than the focus on significant differences that are by chance. Our 3% rate of significant baseline comparisons is slightly lower than in earlier surveys with a 4 6% rate and is consistent with the rate of 5% of such tests expected to reach p<0 05. The statistical issues related to subgroup analysis in RCT are well known including the concept that the reliance on subgroup p values is misleading. Nevertheless the use of interaction tests to assess the heterogeneity of treatment effect across different subgroups, the correct approach, is still underused. Subgroup analysis reflects the scientifically important issue that the treatment may have really different effects according to certain baseline characteristics. However, if the overall treatment effect is significant, some subgroups will and some will not show significant differences depending on chance and the smallness of subgroups. An example is the subgroup analysis of the CHAMPS study, 15 published in a companion paper. In the abstract the authors stated, Intramuscular Interferonβ-1a initiated at a first demyelinating attack delayed clinically definite MS in monofocal patients (P=0.0013), patients with or without gadolinium enhancing lesions (P=0.0007, P=0.0405) and patients with at least nine T2 lesions at baseline (P=0.0044). This sentence suggests that the drug delayed clinically definite MS just in the indicated subgroups. As for patients classified according to their number of baseline T2 lesions, in the results we find that over 2 years, the unadjusted Hazard Ratio (HR) was 0.50 (95% Confidence Interval (CI) ; P=0.1700) for patients with fewer than nine T2 lesions and 0.53 (95%CI ; P=0.0044) for patients with nine or more T2 lesions. So the treatment effect was pretty much the same in the two subgroups, but it was non-significant in patients with fewer than nine T2 lesions because their group was small (90 subjects), while it was significant for patients with nine or more T2 lesions who were more numerous (286 subjects). The correct approach, based on an interaction test, would have compared the HRs (0.50 vs. 0.53), indicating a lack of evidence that the intervention s effect (significant on the whole

6 Signori et al. 781 group) depended on the number of baseline T2 lesions. Subgroup analysis constitutes additional data usually not appearing in the first report that would be of interest for MS specialized journals. It must be noted that, since interaction tests commonly lack statistical power in single trials, the combination of individual patients data in meta-analyses would be crucial to detect subgroup effects. Meta-analyses pooling individual-patient data from different RCT have never been performed in MS research, due to the companies reluctance to combine their data. MS specialized journals should promote the publication of such kind of pooled analyses. Since most of the RCT in MS are fully or partially sponsored by industry (92% of those included in our analysis) it is not possible to assess the impact of the funding source on the quality of reporting. Otherwise, while industry-funded research largely resulted associated with increased likelihood of pro-industry results, there are conflicting results on the limited data about the association of industry funding with methodological quality, with some studies indicating even a higher quality (probably associated with the greater financial resources) compared to non-industry funded trials In conclusion, reporting of several important aspects of trial methods has improved since The editorial policy of medical journals has driven this improvement and should continue to enrich the guidelines for authors, covering also technical aspects (e.g. the use of interaction test is required for subgroup analysis ) whose application will end up to educating authors to the adoption and reporting of correct procedures. Funding This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. Conflict of interest MPS has received consulting fees or honoraria from Biogen IDEC, Merck Serono, Actelion, Synthon, Allozyne. AS and AB have no conflicts of interest. References 1. The IFNB Multiple Sclerosis Study Group. Interferon betalb is effective in relapsing-remitting multiple sclerosis. I. Clinical results of a multicenter, randomized, double-blind, placebo-controlled trial. Neurology 1993; 43(4): Polman CH, Reingold SC, Barkhoff, et al. Ethics of placebocontrolled clinical trials in multiple sclerosis: a reassessment. Neurology 2008; 70: Sormani MP, Bonzano L, Roccatagliata L et al. Magnetic resonance imaging as a potential surrogate for relapses in multiple sclerosis: a meta-analytic approach. Ann Neurol 2009; 65: Sormani MP, Bonzano L, Roccatagliata L et al. Surrogate endpoints for EDSS worsening in multiple sclerosis. A metaanalytic approach. Neurology 2010; 75: Friede T, Parsons N, Stallard N, et al. Designing a seamless phase II/III clinical trial using early outcomes for treatment selection: an application in multiple sclerosis. Stat Med 2011; 30 (13): Chataway J, Nicholas R, Todd S, et al. A novel adaptive design strategy increases the efficiency of clinical trials in secondary progressive multiple sclerosis. Mult Scler 2011; 17: Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996; 276: Moher D, Schulz KF and Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001; 357: Schulz KF, Altman DG and Moher D, for the CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. Ann Int Med 2010; 152: Assmann SF, Pocock SJ, Enos LE et al. Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet 2000; 355: Senn S. Testing for baseline balance in clinical trials. Stat Med 1994; 13: Altman DG. Comparability of randomised groups. Statistician 1985; 34: Yusuf S, Wittes J, Probstfield J, et al. Analysis and interpretation of treatment effects in subgroups of patients in randomised clinical trials. JAMA 1991; 266: Pocock SJ, Hughes MD and Lee RJ. Statistical problems in the reporting of clinical trials: a survey of three medical journals. N Engl J Med 1987; 317: O Connor P, Kinkel RP and Kremenchutzky M. Efficacy of intramuscular interferon beta-1a in patients with clinically isolated syndrome: analysis of subgroups based on new risk criteria Mult Scler 2009; 15: Bhandari M, Busse JW, Jackowski D, et al. Association between industry funding and statistically significant proindustry findings in medical and surgical randomized trials. CMAJ 2004; 170: Montgomery JH, Byerly M, Carmody T, et al. An analysis of the effect of funding source in randomized clinical trials of second generation antipsychotics for the treatment of schizophrenia. Control Clin Trials 2004; 25: Yaphe J, Edman R, Knishkowy B, et al. The association between funding by commercial interests and study outcome in randomized controlled drug trials. Fam Pract 2001; 18: Thomas O, Thabane L, Douketis J, et al. Industry funding and the reporting quality of large long-term weight loss trials. Int J Obes (Lond) 2008; 32: Rios LP, Odueyungbo A, Moitri MO, et al. Quality of reporting of randomized controlled trials in general endocrinology literature. J Clin Endocrinol Metab 2008; 93: Khan NA, Lombeida JI, Singh M, et al. Association of industry funding with the outcome and quality of randomized controlled trials of drug therapy for rheumatoid arthritis. Arthritis Rheum doi: /art [In press].