ACC: Review of the Weekly Compensation Duration Model

Transcription

1 ACC: Review of the Weekly Compensation Duration Model Prepared By: Knoware Creation Date July 2013 Version: 1.0 Author: M. Kelly Mara Knoware

2 Document Control Change Record Date Author Version Change Reference July 2013 Dr Kelly Mara V0.1 Initial draft Aug 2013 Dr Kelly Mara V1 Final following feedback and review. Reviewers Name Robert Noe Colin Harris Position Analytics Practice Lead Technical Director Distribution Copy No Name Location 1 Library Master Knoware Page 2 of 10

3 Table of Contents 1. Introduction Review of the Weekly Compensation Duration Model Scope of the Review What is not Covered Findings Methodology General Approach Methodology Documentation and Specific Sections in the Document Validation of the model the levels of accuracy - extent to which the model predicts WC outcomes Conclusions Recommendations Page 3 of 10

4 1. Introduction Review of the Weekly Compensation Duration Model This document covers a review of Weekly Compensation Duration Model - Modelling Approach which deals with the modelling approach taken to predicting the duration of weekly compensation claims at ACC. Scope of the Review Statistical appropriateness of the modelling approach The details of the presented model, including method of selection of predictors Documentation on modelling approach Validation of the model the levels of accuracy - extent to which the model predicts WC outcomes Recommendations. What is not Covered The data detail neither the data used to generate the model nor the data used for validation of prediction, except in relation to general documentation Any programming code used to construct or validate the model Operational implementation of the model. 2. Findings Methodology General Approach The purpose of the modelling is to predict the lifetime duration of Weekly Compensation claims registered with ACC. The modelling approach adopted is very appropriate for several reasons: Survival Analysis is a well-developed statistical field that has been applied to lifetime data in a wide variety of medical and engineering fields. Regression analysis may, initially, seem suitable. It is intuitively straightforward. Selection of a parametric model for WC lifetimes (for example using Exponential or Weibull distributions) requires considerable assumptions that would require checking and, given the large amount of censoring in WC durations that is involved in this data, it would be likely to lead to biased predictions. The Cox Proportional Hazard is the most commonly used regression model for survival data. This model makes only the assumption that different individuals (WC events) have hazard functions that are proportional to one another, dependent upon input factors (diagnosis, age, etc. for the WC situation). Page 4 of 10

5 Given the large amount of input variables (and, indeed, the very large amount of data that ACC has), the Proportional Hazards approach is the most understandable in any operational sense. Methodology Documentation and Specific Sections in the Document Section 3.1 Intro, p2 - Agreed standard regression techniques are likely to lead to biased results given the level of censoring for more recent claims. This is more relevant for 2013 claims. (Table 1 gives claims still open it is not surprising that 2013 is high.) Section 3.1 Items 1) and 2) - This is an operational argument, not a methodological one. Section 3.1 Item 3 - The assumptions have been shown to be broadly satisfied but the documentation shows a broad graphical compatibility. The proportional hazard assumption can be formally tested. This should be included in an appendix, not in the main body of the report. Section 3.1 Item 4 - Yes, the survival curve is, of itself, not of primary interest. Section 3.1 Item 5 - Using time variable covariates in the model is, in the longer term, possible, but requires testing. The idea has intuitive appeal, but it is not totally clear that the incapacity duration will have an effect how often does the incapacity duration change during the course of the claim? In Table 1 (p3), the historical still open claims are tabulated. Table 1 is relevant only for 2012 and 2013 since the model focusses on short term claims (less than 182 days). Section 5.1 (p6), the use of expert experience is a very good starting point. The model should start with the consensus primary factors. Which factors proved useful and which did not? A few are listed. Further detail on some aspects of the nature of the variables is given in the Appendix. The overall summary of results including a description of the whole model, with the listed factors, should be given in a table (in the Appendix). There is a statement that clients select provider on the basis of seriousness of injury. This seems plausible, but what is the evidence for this? Last paragraph there should be their. Section 6 immediately becomes 6.2?? Section 6.2 General Comments. The stated aim is to produce accuracy of prediction, rather than a model that is easy to interpret. This is fine, but for the purposes of further, future justification, the model should be amenable to some sort of conceptual interpretation. The influence of each variable explained to case management experts is an excellent start. Some interpretation should be possible for case managers so that they can have confidence in the output from the model. Removal of the 5% threshold for significance down to 0.01% - completely agreed. With so much data, even 0.01% may not be stringent enough. With data points, almost any predictor will be determined as significant at almost any level. The Forward Selection regression method is a very good start for adding/not adding variables but the capability of the overall model or the overall value of adding a Page 5 of 10

6 new variable to the model should be tested. [Likelihood Ratio testing??] (SAS may generate comparisons progressively). Conversion of categorical to numeric this is sensible maybe a couple of examples?? Section 6.3 The near completeness of the data for all variables is fortunate for the purposes of model fitting. Section 6.4 Item 1: What was the base data and were there any anomalies? Section 6.4 Item 2: Forward selection for addition of variables (or transformed variables). This is very appropriate. Section 6.4 Item 3: Use of proportional Hazards for other variables is appropriate. Section 6.4 Item 4b: Smaller providers are excluded from the generation of the model OK, but how well did the model predictions perform for these smaller providers? Were these providers used for the validation exercise? Section 6.4 Item 4c: Same as above- did the model perform well for the small occupation categories? The use of data binning is entirely appropriate given the very large number of diagnoses, occupations, providers it means that any interpretation of the prediction results is limited to the level of aggregation, but this may be all that is required. However, the maximum accuracy requirement may be compromised at the fine level. This also applies where the input variables are continuous (as per 6.6) Section 6.6 The steps taken here ensure that the parameter estimates are reasonably robust Figure 1 gives evidence for prediction accuracy (How is R2 computed?). Section 6.7 Determination of Optimum Complexity of Model Technically, assessment of model in full would require assessment of performance of the model for all survival times t. This relies upon evaluating the Relative Operating Characteristic (ROC) which, in its turn, requires a dichotomous output. Comparisons at the ACC critical, points (t = 28, 49, 70 and 183 days) is very sensible. Use of a 80/20% split of training vs validation data is appropriate. Some description of how the split was determined would be helpful, to exclude the possibility of any structural differences between the training/modelling data and the validation data. Immediate Issues not affecting the appropriateness of the model or its application, but which may require explanation. While the overall purpose of the modelling is to generate useable predictions, it is important to ensure that the predictions are robust to deviations from the core assumptions. The proportionality of hazards, for the input variables, is a requirement of the model. This is initially assessed visually in the Appendix. In most cases it is quite clear that there is no substantial deviation from the assumption of proportionality. However, it Page 6 of 10

7 is noted that, occasionally, proportionality is possibly violated on visual evidence. For example, Diagnosis Group, pp 20,21. The proportionality assumption should be tested (I assume that SAS does this either automatically or as a result of an option?? Schoenfeld residuals??) and any effect of this deviation on the model outcome should be noted. Although maximum accuracy of prediction is desired, part of analysis should cover residual analysis not as part of the core report, but as an appendix. Again, I would suppose that SAS can produce residual plots. The final model includes many variables it is not clear that some variables add much method of FS addition of variables adds when the significance is 0.01% seems conservative, but when the number of data points is so large, almost anything will come in as significant. I am generally a supporter of the principle of parsimony for modelling using as few predictors as necessary. There is always some trade-off between maximising predictive accuracy and minimising model complexity. Figure 2, p 16 chooses 121 degrees of freedom for maximum AUR but it is not clear that fewer variables would not give an approximate equivalence (it is not clear what the real difference between a model with 80df and one with 120df, at least in terms of predictive capability. I realise that the intent is to obtain maximum accuracy - so the prediction process becomes a black box but fewer predictors may give almost the same accuracy and still be open to more operational explanation. The computing capacity at ACC may allow for a much greater number of predictors to generate predictions but does the addition of the more marginal variables improve the overall fit? [Use Likelihood Ratio test for comparison of models??]. Granularity of Data The model uses the dependent variable as days on WC. When a claim is made, does the time on WC determined from days or weeks (days*7) since the critical points for measurement at ACC are 28, 49, 70 and 183 (4, 7, 10 and 26 weeks), does this affect the continuity of the data? The analysis approach is still highly appropriate, but the predictions have a higher level of variability than might be initially evident. [For example, is prediction of 13 days vs 14 days really a prediction of 1 week vs 2 weeks?] Validation of the model the levels of accuracy - extent to which the model predicts WC outcomes. Christchurch Pilot The second document tabulates the results of measuring the model predictions against actual outcomes for Christchurch WC claims since 27 August, There are 10 groups of claims, ordered by severity. It is not clear how the groups were determined. [By simple percentile rankings of actual durations? By some other index not declared here??] For claims in each group, the predictions from the model are compared against the actual outcomes. [At least on Lower Quartile; Median and Upper Quartile of the distribution of claim durations.] Page 7 of 10

8 The comparison is, generally very favourable. The model does function well on these measures. However, I would like to see actual distributions of predictions and actuals to determine any goodness of fit. The model does perform well for most claims (possibly excepting the longer, severe claims) but the three statistics used are fairly robust measures of a distribution. Is there any structural mismatch? [ie, an examination of residuals is important] Conclusions For predicting the WC outcome for individual cases, the modelling approach adopted is very appropriate The model uses well accepted statistical analysis methods The approach is to develop a black-box prediction process, this results in a large number of predictors in the model Reducing the total number of predictors used for the model may sacrifice little in terms of precision but may be very useful for future useability and acceptance Page 8 of 10

9 3. Recommendations 1. As part of any future examination conduct/present a summary Residual Analysis. There are several purposes behind a closer examination of errors or residuals, some of which have been commented on earlier. While these items do not affect the overall goal of maximum accuracy of prediction, analysis of residuals is important to offer insight into the functioning of the model. Discovering Best Functional Form the method used adopts a high accuracy approach (since computing ability is not a limitation) so every variable is in the predictive model -that is OK, but further analysis could reveal the best set of predictors. Identifying subjects not well predicted by the model the construction of the model excludes small categories (quite correctly) but how well does the prediction model perform for the small cases? This could well be part of the validation phase. Identifying Influential Points are there any points that have a strong effect on model parameters/have high influence on parameter estimates? Assessment of Proportional Hazards Assumption noted above. If the proportional hazard assumption is valid, the Schoenfeld residuals should be a random walk. Conversely, if some variable has a large positive effect early on but trails away ( beyond, say, some point once the claimant is cured, then the assumption of proportionality of hazard for that variable is not appropriate. 2. As part of any future work, examination of the level of contribution of predictor variables could be covered. This is with a view toward seeking an explanation and minimisation of predictors. Without exploring a considerable amount of formal statistical and analytical detail, some commentary is appropriate as an attached part of an appendix. While this will certainly be of little use to ACC Case Managers in the short term, it can provide a background for model refinement. I realise that the initial intent is to obtain maximum accuracy - so the prediction process becomes a black box but fewer predictors may give almost the same accuracy and still be open to more operational explanation. The computing capacity at ACC may allow for a much greater number of predictors to generate predictions, but does the addition of the more marginal variables improve the overall fit? The treatment may influence the outcome substantially early on, but after a while it has little or no effect on survival. Seeking a model that is amenable to explanation may more readily allow for interpretation and evaluation of the effect policy or operational changes on duration outcomes. It is important to remember that the present model provides a system- Page 9 of 10

10 wide prediction, not necessarily the outcome for a specific case. [The use of ranges Lower Quartile, Median, Upper Quartile is very beneficial, but understanding the drivers of the outcome is important to determine if, in future, which or any policy change are likely to achieve a change in outcome. If it does, then the model can be re-calibrated to accommodate the new policy. However, a great advantage of the present model, in the case of ACC with little limitation to computing ability, using more variables to maximise accuracy is appropriate as an initial project. Page 10 of 10