Evaluation Tools for COMPUS

Transcription

1 Evaluation Tools for COMPUS Background The Canadian Optimal Medication Prescribing and Utilization Service (COMPUS) identifies, evaluates, promotes and facilitates best practices in drug prescribing and use among health care providers and consumers. The objectives of the program are to optimize drug-related health outcomes and costeffective use of drugs. COMPUS is a nationally coordinated program, funded by Health Canada and delivered by the Canadian Coordinating Office for Health Technology Assessment (CCOHTA), as a service to federal/provincial/territorial jurisdictions and other stakeholders. Role of COMPUS To identify, evaluate, promote and facilitate the implementation of best practices in drug prescribing and use To consolidate information resources on best practices To develop and support networks for best practices in drug prescribing and use To encourage health care provider and customer behaviours that will result in improved drugrelated health outcomes To identify best practice information gaps Evaluating Best Practices the Need for Tools CCOHTA s mission is to provide timely, relevant and rigorously derived evidence-based information to decision makers, and to support the decision making processes. Building on this foundation, the focus of COMPUS is best practices that are supported by evidence in the form of scientific studies. The quality of the scientific evidence varies, however, depending on the study design, conduct and analysis, necessitating a method to evaluate the evidence behind best practices. A number of tools have been developed around the world for evaluating and grading evidence: quality assessment instruments (QAIs) and evidence grading systems (EGSs). Considering that no gold standard exists, COMPUS applied a systematic process to select the most appropriate evaluation tools for the program. These tools assess the quality of evidence, grade the strength of the body of evidence and grade the strength of clinical recommendations based on evidence. Process Used to Select Evaluation Tools COMPUS used the following process to select QAIs for assessing the quality of systematic reviews, randomized controlled trials and observational studies (cohort and case-control studies), as well as to select an evidence grading system. December 20, 2005 Page 1 of 6

2 Working group assembled The working group was comprised of internal researchers, information specialists, methodology experts, and external researchers. The group was mandated to direct the selection process. Review articles searched and selected The Agency for Healthcare Research and Quality (AHRQ) Evidence report/technology assessment number 47, entitled Systems to rate the strength of scientific evidence, 1 provided an evaluation of quality assessment instruments and evidence grading systems found in the literature published between 1995 and mid Accepting the findings for the period between 1995 and 2000 from the AHRQ report number 47, a comprehensive search of review articles of QAIs and EGSs, published between 2000 and 2005, was designed and carried out. This resulted in more than 3,000 citations for selection. Eleven review articles were selected for further analysis based on the selection criteria specified by the working group. QAIs and EGSs identified from review articles Approximately 230 QAIs for systematic reviews, randomized controlled trials and observational studies as well as nearly 50 EGSs were identified from the 11 review articles, after eliminating duplicates. Initial expert consultation conducted Canadian and international experts in evidence evaluation methodology helped identify instruments or systems not included in the list of identified QAIs and EGSs. Ten additional instruments and systems were identified as a result of this expert input. Individual QAIs and EGSs searched and selected A second literature search for articles reporting on QAIs or EGSs, published after 2000, was conducted. This resulted in the identification of several more instruments based on the selection criteria specified by the working group. QAIs and EGSs identified for evaluation Instruments and systems recommended in review articles, with and without recommendations, as well as those identified by expert consultation and the second search, were tagged for further evaluation. Identified QAIs and EGSs evaluated The instruments and systems identified were evaluated by type of study using AHRQ evaluation grids, and descriptive items. 1 The highest scoring instruments and systems from each grid represented the proposed selections. Second expert consultation conducted The list of selected QAIs and EGSs proposed for COMPUS use were sent to experts (those who participated in the initial expert consultation) for review and comment. QAIs and EGSs identified for COMPUS Based on the second expert consultation, a list of four QAIs and two EGSs were chosen as potential tools for COMPUS. Stakeholder consultation conducted The proposed QAIs and EGSs were posted on the CCOHTA web site, along with an online feedback form, to encourage stakeholder input from all interested parties. Feedback was reviewed and considered. Based on this feedback, the evaluation tools for COMPUS were selected (Table 1). December 20, 2005 Page 2 of 6

3 Table 1: Selected QAIs and EGSs for COMPUS Type of Study Systematic reviews Tools QAI AMSTAR 2005 (Beverley Shea, CIET, Institute of Population Health, Ottawa: personal communication, 2005 Oct) (Appendix 1) Randomized controlled trials SIGN : methodological checklist for RCTs 2 Cohort studies SIGN : methodological checklist for cohort studies 3 Case-control studies SIGN : methodological checklist for case-control studies 4 EGS GRADE ,6 and SIGN December 20, 2005 Page 3 of 6

4 References 1. West S, King V, Carey TS, Lohr KN, McKoy N, Sutton SF, et al. Systems to rate the strength of scientific evidence [Evidence report/technology assessment no 47]. Rockville (MD): Agency for Healthcare Research and Quality; 2002 Mar. AHRQ Publication No 02-E016. Available: (accessed 2005 Mar 10). 2. Scottish Intercollegiate Guidelines Network. Methodology checklist 2: randomised controlled trials. In: SIGN 50: a guideline developers' handbook. Edinburgh: The Network; Available: (accessed 2005 Nov 17). 3. Scottish Intercollegiate Guidelines Network. Methodology checklist 3: cohort studies. In: SIGN 50: a guideline developers' handbook. Edinburgh: The Network; Available: (accessed 2005 Nov 17). 4. Scottish Intercollegiate Guidelines Network. Methodology checklist 4: case-control studies. In: SIGN 50: a guideline developers' handbook. Edinburgh: The Network; Available: (accessed 2005 Nov 17). 5. Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) Working Group. Grading quality of evidence and strength of recommendations: abridged version. BMJ 2004;328(7454): Available: (accessed 2005 Nov 17). 6. Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) Working Group. Grading quality of evidence and strength of recommendations: full version. BMJ 2004;328(7454). Available: (accessed 2005 Nov 17). 7. Scottish Intercollegiate Guidelines Network. Forming guideline recommendations. In: SIGN 50: a guideline developers' handbook. Edinburgh: The Network; Available: (accessed 2005 Nov 17). December 20, 2005 Page 4 of 6

5 Appendix 1: A MeaSurement Tool to Assess Reviews (AMSTAR), Was an a priori design provided? The research question and inclusion criteria should be established before conducting the review. 2. Was there duplicate study selection and data extraction? There should be at least two independent data extractors and the consensus procedure for disagreements should be reported. 3. Was a comprehensive literature search performed? At least two electronic sources should be searched. The report must include years and databases (e.g. Central, EPOC, and MEDLINE ). Key words and/or MESH terms must be stated, and where feasible the search strategy should be provided. All searches should be supplemented by consulting current contents, reviews, textbooks, specialized registers, or experts in the particular field of study, and by reviewing the references in the studies found. 4. Was the status of publication (i.e. grey literature) used as an exclusion criterion? The authors should state that they searched for reports regardless of their publication type. The authors should state whether they excluded any reports (from the systematic review), based on their publication status. 5. Was a list of studies (included and excluded) provided? A list of included and excluded studies should be provided. 6. Were the characteristics of the included studies provided? In an aggregated form, such as a table, data from the original studies should be provided on the participants, interventions and outcomes. The ranges of characteristics in all the studies analyzed (e.g., age, race, sex, relevant socioeconomic data, disease status, duration, severity, or other diseases) should be reported. 7. Was the scientific quality of the included studies assessed and reported? A priori methods of assessment should be reported (e.g., for effectiveness studies if the author(s) chose to include only randomized, double-blind, placebo controlled studies, or allocation concealment as inclusion criteria); for other types of studies alternative items will be relevant. t applicable t applicable t applicable t applicable t applicable t applicable t applicable December 20, 2005 Page 5 of 6

6 8. Was the scientific quality of the included studies used appropriately in formulating conclusions? The results of the methodological rigour and scientific quality should be considered in the analysis and the conclusions of the review, and explicitly stated in formulating recommendations. 9. Were the methods used to combine the findings of studies appropriate? For the pooled results, a test should be done to ensure the studies were combinable, to assess the homogeneity (i.e., chi-squared test for homogeneity, I²). If heterogeneity exists, a random effects model should be used and/or the clinical appropriateness of combining should be taken into consideration (i.e., is it sensible to combine?). 10. Was the likelihood of publication bias assessed? An assessment of publication bias should include a combination of graphical aids (e.g., funnel plot) and statistical tests (e.g., Egger regression test). 11. Was the conflict of interest stated? Potential sources of support should be clearly acknowledged in both the systematic review and the included studies. t applicable t applicable t applicable t applicable Source: AMSTAR 2005 (Beverley Shea, CIET, Institute of Population Health, Ottawa: personal communication, 2005 Oct) December 20, 2005 Page 6 of 6