Cchrane Statistical Methds Training Curse 4 5 th March 2010, Cardiff, UK Minutes and actins frm the sessin: Statistical cntributin t CRGs Facilitatr: Steff Lewis Minutes: J McKenzie This sessin began with a brief verview by Gergia Salanti f the survey f CRG statisticians which tk place in June 2009. This was fllwed by discussin n each f the fllwing tpics: Hw far shuld we g with refereeing?, Cmmn statistical issues in refereeing, Help/advice/training needs f CRG statisticians, and Hw t give feedback t review authrs. This sessin was attended by apprximately 37 attendees. 1. Results f survey t CRG statisticians Gergia Salanti presented results frm the survey f CRG statisticians which aimed t assess the statistical issues cmmnly encuntered and t identify areas where the statisticians wuld value mre discussin, training, and research. The survey was carried ut in June 2009. Frm data in Archie and persnal cmmunicatin it appears that 75 statisticians prvide supprt t 43 f the 52 CRGs. All 75 statisticians were apprached t take part in the survey; nly 19 replied (25%). The survey asked abut statistical appraches used as well issues related t the reviewing prcess. Key pints frm the latter fllw: Prcess f reviewing: The median time spent per week reviewing was 6.5 hurs (range 1 20). 4 check the data either in a sample r smetimes. 9 dn t check the data unless they are a c-authr. Cchrane Statistical Methds Training Curse: Statistical cntributin t CRGs - 1 -
Recmpensing ccurs thrugh different means: 4 are paid, 7 reprted being c-authrs/cllabratins, 6 receive funding t attend events (such as the Cllquia). Strategies suggested t attract statisticians: Payment f their time. Payment fr Cllquia and curses. C-authring. Editrial bard membership. Cnvince them that it is imprtant. Capture them yung. Being valued by their CRGs. 2. Hw far shuld we g with refereeing? Key pints frm the discussin n hw far we shuld g with refereeing were: 1. There is variatin between CRGs regarding requirements f statistical supprt fr reviews. Sme CRGs require the review team t include a statistician. Fr ne f these CRGs, at the title registratin stage, they will request t see the included statistician s CV. If the review team is nt able t invlve a statistician, the title will nt be registered. Other CRGs require that the review team has access t statistical supprt (4 CRGs). 2. Discussin tk place n the issue f whether there shuld be mre standard text in prtcls. There was cncern frm a cuple f statisticians regarding this suggestin. They felt that it was better fr the review authrs t attempt t write the text since when this ccurs, it is likely t be mre evident if the authrs d nt understand the methds. If prblems in understanding are nt detected at the prtcl stage, then this may result in much greater prblems at the review stage. 3. There was sme disagreement abut statisticians rle in the peer review prcess. One felt that statisticians need t negtiate with Cchrane Statistical Methds Training Curse: Statistical cntributin t CRGs - 2 -
CRGs abut their peer review rle, t ensure that it is limited t peer review. We dn t currently have clarity f the rle f statistical peer reviewers. Anther felt that it is the respnsibility f the CRG t train reviews. One did nt agree with this view and felt that this type f view needed t change. It was stated that statisticians d nt have time t g back t the trial papers, except in the case f CRGs which receive specific funding fr this task. 4. Sme f the CRGs represented at the meeting had multiple statistical editrs, and sme received funding fr statistical supprt. 5. The questin was psed as t whether it shuld be a requirement that review teams have a methdlgist/statistician. Title registratin frms which d nt have a methdlgist/statistician culd be turned dwn. Althugh it is difficult t assess whether smene is a real methdlgist/statistician withut a CV. 3. Cmmn statistical issues in refereeing Key pints frm the discussin n cmmn statistical issues in refereeing were: 1. Missing papers: it was felt that papers missed in the search strategy were the respnsibility f the trial search c-rdinatr, nt the statistician. 2. Cncern was raised ver the results f recent empirical research which has estimated that 7% f trials included in Cchrane reviews had utcme data fully reprted in the publicatin, but this was nt included in the Cchrane reviews (Kirkham JJ, Dwan KM, Altman DG, Gamble C, Ddd S, Smyth R, Williamsn PR: The impact f utcme reprting bias in randmised cntrlled trials n a chrt f systematic reviews. BMJ 2010, 340:c365). It was suggested that this may be a prblem f training where reviewers are ften taught that sample sizes, means, and SDs are required fr entering data int RevMan. It was suggested that it might be useful fr statistical reviewers at the prtcl stage t ask review authrs t state in the prtcl that they will extract data frm CIs, p-values, etc. Cchrane Statistical Methds Training Curse: Statistical cntributin t CRGs - 3 -
3. There was discussin abut whether it wuld be useful fr the SMG t cllect checklists used by statisticians in CRGs t assess prtcls and reviews. There was general agreement that this wuld be useful with the aim f creating a general checklist fr statisticians. This culd frm part f the training materials fr statisticians, which wuld be made available n the SMG website. 4. Sme ther issues raised included: (i) when a meta-analysis has nly tw trials, fr example, it is cmmn fr review authrs t state results frm the meta-analysis indicated, when it wuld be better t state results frm tw studies indicated, (ii) review authrs cmmnly vte cunt in abstracts, (iii) reviews shuld nt use quality scres, and (iv) because f cncerns f multiple testing, it was suggested that the number f utcmes per review shuld be reduced. 5. Anther issue which was raised was the prblem f the time between when the prtcl is published and the review is cmpleted. During this time, the Cllabratin may have adpted majr methdlgy changes. Sme review authrs are nt prepared t adpt these changes; they will nly d what they have specified in their prtcl, even althugh this may be utdated. Actin: Send an email t the SMG list asking fr checklists used t assess prtcls and reviews. Ask fr a grup f vlunteers t review the checklists and create cmbined checklist. Once this is finalised it shuld be psted n the SMG and shared with training teams. 4. Help/advice/training needs f CRG statisticians Key pints frm the discussin n help/advice/training needs were: 1. There was sme discussin ver the netwrk f CRG statisticians which will be set up by the SMG. This netwrk will include ne r mre persns with an interest in statistics within each f the CRGs. The purpse f which is t allw easy cmmunicatin between the CRG statisticians and grups such as HAG and MARS. It was suggested that the SMG list may currently fulfil the rle f the CRG netwrk if the Cchrane Statistical Methds Training Curse: Statistical cntributin t CRGs - 4 -
membership cvers thse in CRGs with an interest in statistics. Hwever, it was suggested that it may be better t have a separate netwrk since the purpse f the CRG netwrk and the SMG list are different. Fr example, there are many n the SMG list wh d nt have a rle f prviding direct editrial supprt t CRGs. 2. The questin was asked as t whether it was mandatry fr each CRG t have a statistician. Currently there is n frmal agreement, but there are current discussins taking place regarding this in the Cllabratin. 3. Sme discussin tk place abut whether we need a frmal mentring prcess. One statistician respnded that a successful apprach fr him has been t bring alng new clleagues in his educatinal institutin; nt necessarily thrugh the Cllabratin. 4. A set f exemplar reviews and prtcls are being created/cllected as an additinal resurce t the Handbk fr review authrs. 5. It was als suggested that it wuld be helpful t have a webpage fr statisticians which cntained details f majr changes t statistical advice in revised versins f the Handbk. 5. Hw t give feedback t review authrs Key pints frm the discussin n prviding feedback t review authrs were: 1. The questin was psed as t whether we shuld be able t reject prtcls/reviews. One stated that we shuld be able t state that the prtcl/review is nt publishable. Anther stated that it is difficult t prvide cnstructive feedback when the review is s pr. One review grup has a screening prcess where the managing editr cmpletes a checklist fr the review, and if it passes the checklist, the review is then peer reviewed by the grup s statistician. 2. Sme discussin tk place abut re-review. Apprximately 6 f the statisticians indicated that they re-reviewed reviews. Sme felt that all thse invlved in the editrial prcess f a review needed t be satisfied that apprpriate changes were made. It was suggested that Cchrane Statistical Methds Training Curse: Statistical cntributin t CRGs - 5 -
when there are disagreements abut changes t be made, it is the editr s rle t make final decisins. Fr diagnstic test accuracy reviews, the reviews are signed ff by the CRG and the DTA editrial grup. It was suggested that it wuld be gd if this type f prcess als ccurred fr Cchrane reviews f interventins, where the review shuld be signed ff by a statistician. 6. AOB It was generally felt that a meeting such as this prvided a gd supprt frum fr statisticians within the Cllabratin. Cchrane Statistical Methds Training Curse: Statistical cntributin t CRGs - 6 -