IBM Software White Paper. Information lifecycle governance in a big data environment

Transcription

1 IBM Software White Paper Information lifecycle governance in a big data environment

2 2 Information lifecycle governance in a big data environment A key tenet of big data is that you should keep as much data as you can, for as long as you can, because you never know what insights might be drawn in the future from data that is not currently useful. However, this keep-everything approach to big data can result in significant cost and risk to the business. On the other side of the corporate fence, information lifecycle governance (ILG) stakeholders work diligently to minimize the potential for problems. Their goal is to defensibly delete data and preserve only that which is required by law, regulation, litigation or business needs. How do you balance the need to reduce the cost and risk of keeping information against the need to retain enough useful data to inform current and future analytical models? This white paper describes steps to coordinate big data s drive to keep everything for possible future analysis with information governance s mission to defensibly delete as much as you can as quickly as possible. The importance of governance in big data management Because information may have different analytics value at different times, governance is even more important in a big data environment than in traditional data management models. As a result, organizations may define and enforce retention policies to protect sensitive data, govern data quality and defensibly dispose of legacy data. Intelligently constructed policies address both the need to minimize the cost and risk of keeping data and the need to retain and archive data for future reference and analysis. To achieve this equilibrium, you need to first understand your data. That process may lead to changes in how you deal with information. For example, the legal department must identify and evaluate the risks related to the information you propose to preserve, including the likelihood and impact of those risks. You may also encounter conflicts based on the ways different departments treat and approach their data. Essentially, the conflict centers around one basic and often the hardest evaluation: Does the real and perceived potential value of preserving the data outweigh the actual cost and apparent potential risk of preserving it? Fundamentally, resolving the conflict is straightforward: information must be governed. In environments where data ages rapidly and becomes obsolete quickly, an information governance based approach can heighten overall data quality. Also, success with big data and analytics depends on a supply of trusted information; a solid ILG foundation helps improve the level of confidence in the information. Organizations can strike a balance between the needs of big data and ILG by following a six-step process: 1. Understand your information and its current and future lifecycle 2. Define and enforce an overall retention schedule 3. Throw out the trash 4. Extract analyzable data 5. Sanitize before you analyze 6. Delete unneeded source information

3 IBM Software 3 Understand and adjust these steps based on your business Rarely will any methodology fit the needs of a business perfectly. It is critical that you adjust and expand these steps to fit your particular situation. Further, you may be incorporating types of data into your big data environment for example, streaming data from continuous sensor feeds that require specific steps not presented here. It is also important to note that these steps are not necessarily sequential. Some of them can and should be undertaken in parallel. The steps should also be customized and refined based on your big data and analytics strategy. Keep in mind that big data, analytics and ILG, which may be new approaches for your organization, should be considered as part of a larger enterprise transformation initiative. Step 1: Understand your information and its current and future lifecycle One of the most important steps in this process of harmonizing big data with ILG sounds deceptively simple: you need to understand your information. It is quite likely that many in your organization believe they understand their information. But it is equally likely that they understand only part of what comprises the overall lifecycle of their information, how the information is used and how it is integrated across the organization to support critical business operations. In some environments, just a handful of stakeholders may understand the potential business risk of keeping data, as well as the organization s legal obligations regarding what data must be kept and for how long. Understanding the overall information lifecycle is critical to determining the form in which data should be preserved and for how long; how data will be used for decision making as part of ongoing analytics; and what should happen with the data after it goes through the analytics process. To better understand your information, consider some of the following factors: Information ownership: Who is in charge of your data? Effective information governance is essential if organizations are to take full advantage of big data. Yet many enterprises struggle to get the most out of the small data they already hold let alone take charge of complex big data sources outside their direct control. Good governance is no longer optional. Without ownership and accountability, responsibility for data and information within organizations could slip through the cracks. Types of information: Have you considered all possible types of information that your organization generates, uses and keeps? What about paper stored on-site and off-site, , office documents, Microsoft SharePoint sites, voic , webinar recordings or video recordings from security cameras? How could those sources be useful in a big data and analytics environment? How can they pose business risks? How costly is it to keep that information? What data do you actually need to keep based on regulatory requirements? The information lifecycle: Where and how does the information originate? Is this information duplicated elsewhere? Should you even base your analysis on this information, or is there a better source? How long is this information retained? Where are all the places it might be found, and from where might you need to retrieve it under court order?

4 4 Information lifecycle governance in a big data environment Utility to the business: Where in the business is the information used? What decisions are made based on this information? How long does the business use the information? If it is used periodically on an ongoing basis, how much is it used, which parts are used and for what purpose? Utility in analysis: What analysis is performed using this information? What analysis could be performed using this information, perhaps in conjunction with other information for which you may or may not have access? How long is the analysis relevant does the value of the data decline with age? Risk of keeping: How has this type of information been used against your organization before? How has this type of information been used against others in your industry? If someone were to sue your organization, how would their attorney view your information in a completely different context and mislead a jury? What does your legal department say about the risk of keeping this information? Obligation to preserve: Is this information relevant to any current or foreseeable legal actions? What laws and regulations govern the use and preservation of your information? How long do they require you to preserve the information? Changes with big data and analytics or with ILG: How will the information be stored, altered, preserved and used in a big data environment? How will that alter the factors of interest to ILG? According to your ILG program, how long must you keep the information? How soon will it be eligible for destruction? How will retention time affect the information s usefulness to your big data and analytics environment? Infrastructure and data placement: Can you keep the data where it originated and run analytics there, instead of moving the data to a new environment? Do you need new infrastructure to support ediscovery in your existing big data environment? How does data in a cloud environment affect the need for governance overall and ILG in particular? What impact does a cloud environment have on data privacy concerns? Governance as a way of improving analytics: How will analytics quality be improved if data quality is higher than before? Do you really need all the data for analytics, or do you use a relatively small percentage to obtain the highestvalue insights? Can you dispose of the rest of the data? Can you trust the data used for analytics? Big data and Is a part of your big data analysis? Would present a high risk in the event of litigation or regulatory audit? Would the results of your analytics present a risk as well? What can you do to reduce your exposure while maintaining your legal obligation and preserving the ability to derive insights from analytics? How will you manage that supports your business exchange correspondence and collaboration with vendors and suppliers? Step 2: Define and enforce an overall retention schedule In records management, a key component of ILG, retention schedules are used to categorize information in a way appropriate to the industry and individual organizations. The schedules apply various laws and regulations to information categories. These laws and regulations may dictate minimum preservation requirements or, in some cases, the maximum amount of time information can be kept. In addition, retention schedules usually incorporate adjustments to preservation requirements based on business needs.

5 IBM Software 5 Creating a retention schedule requires a significant amount of effort and cooperation across the organization. It can be difficult to define record categories that all stakeholders in a single division can agree upon, let alone define them for common functions or information types that apply across the enterprise. Remember that executive approval, up to the board level, may be necessary for implementing or modifying a retention schedule. Big data advocates may erroneously think retention schedules apply only to traditional systems, because that is where much of their information may originate. But if your organization is sued and has the data somewhere, you will be obligated to produce it, no matter where it is stored. As such, retention schedules apply to big data systems, too. Retention schedules also must account for different laws and regulations across the global enterprise, since the same type of information may be stored in systems across different continents, countries or regions. This means you should adjust your retention schedule accordingly and accommodate alternative retention periods based on the different rules. No matter what else is done in an ILG program, defining and enforcing the retention schedule allows you to manage what may have been uncontrolled data growth and ever-increasing risk. However, recall that retention schedules should be adjusted to include preservation requirements that are based on business needs. This is where the big data and analytics advocates and the ILG advocates may not agree. Big data and analytics teams usually recognize that much of the information they want to keep has a shelf life beyond which its utility is questionable. Negotiate agreement for retention periods in the areas where you can. In the areas still under dispute, executives may need to make the determination based on advice from the ILG team, the big data and analytics team, and corporate legal counsel. Step 3: Throw out the trash Since one of the promises of big data is that you can collect any data and analyze it later, it is tempting to do exactly that. However, even the big data team likely does not want poorquality information. Analysis is generally better when you know the origin of your information and can attest to its quality, but big data analytics can draw interesting conclusions using mixed-quality information that is not necessarily from the best source. Even when processed through big data tools and techniques, information sources may contain useless, redundant or erroneous data that could pollute the analysis. Given such information sources, avoid the idea of keeping everything in case later analysis might prove beneficial. If you know the information is problematic, get rid of it. Having a smaller set of relevant data enhances ediscovery and leads to higher data quality, which is a critical element in optimizing data analytics, while strong governance is a key contributor to improving quality. Moreover, ILG should be a top priority to help reduce the unchecked growth of big data, boosting application efficiency. Step 4: Extract analyzable data Big data and analytics environments may include unstructured information such as office documents and social media feeds. In this case, use content analytics techniques to extract data from those sources so you can delete the original files in line with the retention schedule. Bear in mind, however, that content analytics techniques are constantly evolving, so the business value of the original documents should be carefully evaluated against the risk of preserving them to determine whether they should be kept longer than regulations require.

6 6 Information lifecycle governance in a big data environment When source document deletion does occur, ensure the raw data in the content analytics subsystem cannot be used to substantively reconstruct the document. If it can, dispose of the extracted data according to the retention schedule. You can also analyze data directly at its source and then either present the results to drive appropriate actions and decision making from the business or incorporate the analyzed data into the big data environment for further integrated analytics. Step 5: Sanitize before you analyze When your big data and analytics system contains information that can uniquely identify individuals or organizations, determine whether there is any benefit to keeping the data elements that can be used for identification. Does the analysis require the identification of individuals or organizations, or will the data be used in aggregate? Is it sufficient to simply distinguish individuals, as opposed to identifying them? In these types of cases where your analytics initiative does not require unique identification of individuals, you can sanitize the information with one-way encoding during the process of ingesting it into the big data and analytics system. Depending on your retention schedule, you may need to preserve the original identifying data. If you are storing sensitive data such as , that data should be highly protected no matter what, and may be subject to data redaction to prevent unauthorized access. Each use case involving big data is a volume-to-relevance discovery process. That is why organizations must discover, analyze and act on terabytes to petabytes of data and provide critical visibility into big data to make informed business and legal decisions in effect, turning it into relevant, useful, trusted data. You should perform audits and review data reports to help confirm your organization is retaining information according to regulations. At the same time, the organization can proactively identify content that does not comply with corporate governance policies, helping to ensure these concerns are addressed before they become legal issues. Many organizations have large volumes of legacy, or dark, data, which has lost its business value but creates cost and risk. Organizations are no longer obligated to preserve this data. A high percentage of legacy data often comprises duplicates of files and documents of every type. Also, 60 to 70 percent of unstructured data typically has no value. 1 All this data is consuming storage costs while adding to information management and production expenses and risks. Organizations typically have no insights into their dark data, yet any breach can expose personally identifiable information (PII), highly confidential information (HCI) and Payment Card Industry (PCI) data. IBM solutions help you address, remediate and clean up vast amounts of legacy data. With capabilities to rapidly index in place, these solutions enable organizations to get to the facts of their data within days or weeks. From those facts, they can then remove trivial data, move obsolete or duplicated data, or archive vital data to lower-tier storage where it still can be accessed if needed.

7 IBM Software 7 After information is stored in the big data system, does it still need to be preserved in the source locations? If the business does not need the information in the other locations, and if the retention schedule does not require you to keep the information, then you should remove the information from the source systems. Figure 1. An IBM StoredIQ heat map helps quickly identify data based on category filters and overlays. Using IBM solutions, you can rapidly assess your data in place to facilitate active governance. Powerful visualizations allow you to view all your unstructured data and then take defensible actions (see Figure 1). You can see what types of data are stored across the enterprise, discover where the oldest or least-used data resides, and then spot and address critical data and compliance issues. Step 6: Delete unneeded source information Typically, information is generated by one or more source systems and then stored in a big data system for later analysis. However, before entering the big data system, the information may be transformed and anonymized, becoming a type of data different from the original source data one with a different retention schedule that requires enforcement. Remember, however, that the retention schedule specifies the amount of time your organization is obligated to keep certain types of information. The information stored in a big data system, as well as in any other system, could be deleted at any time if you have no automated records management capabilities to preserve it, or if you do not have a defined or enforced retention schedule. Consequently, information that must be preserved based on the retention schedule should not be deleted from source systems because that information is needed; any deletion of or tampering with the information should be based only on the retention schedule. Policies and procedures defined for information stored in the enterprise apply equally to information stored in the big data system. In essence, no matter where information is stored, it should be deleted according to the retention schedule. Otherwise, the information is discoverable and must be provided in the event of litigation, even if it is in a big data environment. To be seen favorably by the courts, organizations should have not only a records retention policy with an updated retention schedule, but also a records management system that enforces the policy. In fact, a policy without enforcement or with inconsistent enforcement is often considered by experts to be as bad as not having a policy at all. For example, in Haynes v. Dart, 2010 WL (N.D. Ill. Jan. 11, 2010), the court negatively considered a company s plea that it failed to institute a document retention policy in the form of a litigation hold.

8 Conclusion Although the goals of big data and ILG may appear to conflict, a balanced, joint evaluation and analysis can align both initiatives within your organization. The result is a big data program that provides both actionable, relevant insights and satisfies the compliance and risk mitigation aspects of mature information governance. The approach described in this white paper can provide solid guidance as you work to satisfy the valuable goals of both your big data and ILG programs. In the end, your organization will be able to take advantage of the benefits of new big data and analytics technologies in a secure, trusted and compliant fashion. For more information To learn more about IBM Information Lifecycle Governance solutions, please contact your IBM representative or IBM Business Partner, or visit: ibm.com/ilg Additionally, IBM Global Financing can help you acquire the software capabilities that your business needs in the most cost-effective and strategic way possible. We ll partner with credit-qualified clients to customize a financing solution to suit your business and development goals, enable effective cash management, and improve your total cost of ownership. Fund your critical IT investment and propel your business forward with IBM Global Financing. For more information, visit: ibm.com/financing About the authors Elizabeth (E.) Koumpan is an IBM senior certified IT architect for IBM Industrial Section Solutions Architecture. Scott Sumner-Moore is an executive IT architect in the Enterprise Content Management Practice for IBM Global Business Services. Amir Jaibaji is the Program Director for IBM Information Lifecycle Governance Product Management. Copyright IBM Corporation 2015 IBM Corporation Software Group Route 100 Somers, NY Produced in the United States of America January 2015 IBM, the IBM logo, ibm.com, Global Business Services, and StoredIQ are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the web at Copyright and trademark information at Copyright and trademark information at ibm.com/legal/copytrade.shtml Microsoft and SharePoint are trademarks of Microsoft Corporation in the United States, other countries, or both. This document is current as of the initial date of publication and may be changed by IBM at any time. Not all offerings are available in every country in which IBM operates. THE INFORMATION IN THIS DOCUMENT IS PROVIDED AS IS WITHOUT ANY WARRANTY, EXPRESS OR IMPLIED, INCLUDING WITHOUT ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND ANY WARRANTY OR CONDITION OF NON-INFRINGEMENT. IBM products are warranted according to the terms and conditions of the agreements under which they are provided. Each IBM customer is responsible for ensuring its own compliance with legal requirements. It is the customer s sole responsibility to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law. 1 Compliance and Governance Oversight Counsel (CGOC) Summit 2012 Survey. Please Recycle WVW12356-USEN-00