Can Predictive Coding Add Value to Your Case? A Quick Assessment Tool

Transcription

1 WHITE PAPER Can Predictive Coding Add Value to Your Case? A Quick Assessment Tool Developed by Sophie Ross, Senior Managing Director John Murdock, Senior Director, Technology-Assisted Review

2 Predictive Coding: Going Beyond the Mystique Three common myths exist about the ease and universal applicability of predictive coding. These three myths can be summed up as: Predictive coding is really easy to use just push the start button, sit back and wait for the program to finish. Predictive coding is what s hot right now, and it can be applied to just about every case, whatever the size and complexity of the project. There s really only one benefit of predictive coding the cost savings you can achieve compared to a manual review. Landmark cases that challenged the use of predictive coding, including Da Silva Moore, have resulted in rulings favorable to the use of predictive coding i. Law review studies, including one from Maura Grossman and Gordon Cormack in 2011, provide evidence that manual review is imperfect, and potentially less accurate than machineassisted review ii. Yet many legal professionals remain hesitant to move ahead and implement a technologyassisted review process due to its relative newness and conflicting information about its pros and cons. Like so many hot trends in legal technology, objectively evaluating the benefits of predictive coding can be a complex undertaking. Yet the three predictive coding myths imply that this multi-dimensional review process is an easy, one-size-fits-all panacea, regardless of when and how it is used. To move beyond these misconceptions, there are 10 key considerations which can help you think through whether predictive coding is the right solution for your particular case, the ways it could be used, and whether it should stand alone or be part of a hybrid approach that includes predictive coding in conjunction with other tools. 1. Is your matter appropriately sized? Knowing the size of the matter is really just the starting point. The real key is calculating the number of documents needing review by attorneys for purposes of 1) training the predictive coding engine and 2) for statistically validating the model used to predict on that document set. That is, humans need to review a certain portion of the documents in order to train the predictive coding software on the topics and documents that are relevant, not relevant, etc. While normally you can get started with a small number of training documents, the total number of documents needed in the end will depend on the desired performance, the effectiveness of the tool on the document population, the number of/complexity of the issues being predicted on, and the level of confidence needed in the statistical validation. At some point the confidence level you desire in the statistical validation and the number of training documents required means that in some small cases (those with tens of thousands of documents) it might be most efficient to simply review the documents using effective analytic tools. i ii Da Silva Moore v. Publicis Groupe, No. 11 Civ (ALC) (AJP), 2012 U.S. Dist. LEXIS (S.D.N.Y. Feb. 24, 2012) Maura R. Grossman & Gordon Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective And More Efficient Than Exhaustive Manual Review, XVII Rich. J.L. & Tech 11 (2011). 1

3 2. Are you comfortable knowing the recall and precision rates? In the past, the defensibility of a production was largely dependent upon the workflow used during the e-discovery process and confirming there were documents produced to cover the major issues/expected document types. Predictive coding generally requires you to measure the effectiveness of the process to have confidence in the outcome, and as part of that to measure the likely proportion of the total documents which are responsive and the percentage of those which are then recalled by the tool. Different considerations in every case will mean there is no one right answer as to the appropriate recall level, but measuring the recall does mean you are now affirmatively aware of how many documents you are finding and how many you are (for defensible reasons) missing. While scoring trends in such a way that the documents missed are sometimes of less interest, it is likely that at least some highly responsive documents will be missed. (Of course, the same is true, perhaps even more so, for the traditional review processes which don t rely on predictive coding). If the recall is raised dramatically to capture virtually all responsive documents, the resulting precision would drop to the point that you may either need to review almost all the documents, a costly endeavor, or in the case of production without review risk turning over an unacceptable number of non-responsive documents. It s important to emphasize that this is already true for keywords (which also miss responsive documents), or human review (which also misses responsive documents) the difference here is not in the recall levels, but that the recall levels are necessarily measured when using predictive coding instead of that measurement being optional. 3. How subtle are the distinctions between responsive and non-responsive documents? There are many cases where the facts are straightforward and deciding whether electronically stored information (ESI) should be coded responsive or non-responsive (in response to a subpoena, for instance), does not require specialized reviewers to evaluate. In other instances, human review of the subtleties of communication will be essential. If the materials are very nuanced with extremely subtle differences between responsive and non-responsive documents, the precision and recall rates are likely to be lower, making these cases generally unsuitable for predictive coding. 4. Is the data feature rich? A review comprised mostly of communications and Word documents would be considered feature rich, and a strong candidate for the use of predictive coding software. A document set comprised of mainly financials or repetitive board documents with similar language would be considered flat and would present a challenge to any classification engine. The fewer language distinctions between responsive and non-responsive documents, the fewer opportunities there are for the classifier to recognize and effectively score documents. 5. Could predictive coding be used for issue coding for the matter? For many cases, review attorneys may apply multiple issue codes to the same document because it has content that touches upon several issues relevant to the case. Classifiers generally score documents on a spectrum based on various levels of confidence the document is or is not responsive to the issue being predicted on. If there are multiple issues that need scoring, a model will need to be built for each issue, which means providing enough examples documents responsive to the issue for the Classifier to learn from. The larger the number of issues, the more documents will need to be coded for the purpose of training, and the more subtly nuanced the issues, the more training documents will likely be required. (Of course, the more issues and the more nuanced they are the more expensive human review for those same issues is also). 2

4 6. Should predictive coding be used to identify documents which are sensitive in nature? For cases involving highly sensitive privileged documents that should never go out the door of the company, human review and analytics provide an effective supplement to predictive coding technology. That s because no system whether linear human review handled over months, or predictive coding review conducted over days will provide absolute accuracy, so multiple methods will need to be deployed in order to ensure sensitive data is kept secure. In addition, because predictive coding leverages the language in the documents, it may not always be appropriate as a standalone tool for matters where the primary goal is to protect privilege because it may result initially in too many false hits. Privilege tends to focus on relationships and their context, such as the multiple recipients in the To: and From: fields of an . The language in a privileged document may look identical to the language of a responsive document with the only difference being the parties on the communication. As a result, using predictive coding to achieve the high recall that lawyers are ethically obligated to pursue with respect to privilege can result in a high number of falsely classified documents compared to using targeted keyword searches. While classifier models can focus the classification on metadata (such as the To: and From: fields), the precision is sometimes so low as to need to review a much higher number of false hits than with keywords. Generally using multiple tools is the right approach here. 7. Do you need a plan for materials arriving on a rolling basis? Predictive coding is most straightforward when there is a known universe of data which won t change, since it is thus a one time process. Of course, in most civil litigation the universe of documents changes over time. In situations where new information is constantly being added a plan is needed as to how to handle the new materials. Generally this would involve a process of confirming the accuracy of the model against the newly collected set of documents, and further training the model if needed to improve performance against that new set. However if the case involves very many ongoing and very small incremental collections, this can get unwieldy. 8. Will there be collaboration with opposing counsel? As discussed above, many attorneys, government agencies and judges are unfamiliar or uncomfortable with the use of predictive coding. For matters in which all parties must agree on the e-discovery process, or discuss it during the meet and confer, it may be necessary to educate the other side in the process. Some cases have gone so far as to suggest collaboration with the other side on processes such as conducting QC on the training set, which raises considerations as to whether the other side would have access (likely temporary) to documents they would normally not be entitled to receive such as ones that are non-responsive or privileged. 9. Will high-quality reviewers be used to train the predictive coding software? The initial training sets should be the gold standard of coding, the most accurate coding possible in such a review since both the correct and incorrect calls in the training set will be magnified when propagated across the entire document set. While disagreement docs can be used to QC the reviewer s work (documents where the model is very confident a document should be coded one way and the human reviewer coded it the other), it is best to have those most familiar with the case do the initial training sets, or have very close supervision of the review team. When deciding to use predictive coding, it is important to have available the most sophisticated attorneys on the matter to assist with the review and to work collaboratively with outside counsel as part of the process and on the quality control of the training set. 3

5 10. Do you have access to experienced predictive coding professionals? Predictive coding is more than just software. It s a process that requires legal, technical, project management and statistical expertise. From deciding what issues in the case to use predictive coding on, assessing the number of documents to review in the training set, to designing the iterative workflow and even testifying on defensibility, predictive coding professionals can help you find the right balance between reducing costs and minimizing risk. As in all new technologies, it is important that the people using the tool have broad understanding of how it works, its limitations, and how to best achieve the desired goals. Using these considerations, you may find that your project is an excellent candidate for the multiple benefits of predictive coding, including increased efficiencies, more accurate analysis, improved quality control and significant cost savings. Alternatively, predictive coding may be just one part of a larger solution and hybrid workflow, rather than the sole review process for achieving your objectives. You may just use predictive coding to get rid of the non-responsive documents but then apply a more traditional approach to coding the remainder of the data. It s important to note that predictive coding can be an invaluable quality control tool even if it s not used in the review process. For example, predictive coding can be used to test human review coding decisions after the first pass of review, helping to ensure greater coding consistency. The three common myths about predictive coding that it s easy, relevant for every case, and solely focused on cost savings is inhibiting the use of a proven technology that we have used to provide a great amount of cost savings and value to clients. If you d like to learn more about how predictive coding could help you, please contact us at sophie.ross@fticonsulting.com or john.murdock@fticonsulting.com. 4