CI Information Hub. Incorporating Text Analysis into Business and Competitive Intelligence

Transcription

1 CI Information Hub Incorporating Text Analysis into Business and Competitive Intelligence March 12, 2008 Iknow LLC 100 Overlook Center, 2nd Floor Princeton, New Jersey T: (609) F: (609) Copyright 2008 by Iknow LLC. This document may not be copied, modified, reproduced, republished, transmitted, posted, or distributed in any form without prior written permission from Iknow. All rights not expressly granted herein are reserved. Unauthorized use of this material may violate copyright, trademark and other applicable laws, and could result in criminal or civil penalties. Iknow is a registered trademark of Iknow LLC. This trademark may not be used in any manner without prior written consent from Iknow LLC. All other company and product names may be trademarks of their respective companies.

2 Pharmaceutical Industry Today s Context The pharmaceutical business is unlike any other. Our goal is not entertainment, enjoyment or prosperity. It is the health of patients. PhRMA member companies are devoted to applying biomedical innovation to create new medicines that will enhance or save the lives of patients around the world. Being in the healthcare business brings awesome responsibilities. Every day, our member companies face difficult, fundamental questions. The answers to those questions profoundly affect patients lives. Which diseases should we study? How can we best advance research? Where is the balance between risk and benefit? William C. Weldon, Immediate Past Chairman Pharmaceutical Research and Manufacturers of America (PhRMA) Copyright 2008 by Iknow LLC. 2

3 Four Categories of Information Analysis Lists (Raw) Structured Unstructured

4 Our focus today is on unstructured information Analysis Lists (Raw) Structured Unstructured

5 Some Examples of Corporate Knowledge Assets Copyright 2008 by Iknow LLC. 5

6 Our focus today is on unstructured information Analysis Lists (Raw) Structured Unstructured

7 Today s Agenda CI Information Hub I. Competitive Intelligence (CI) Business Process Information Discovery Analysis and Synthesis Delivery II. Text Analysis Entity Extraction Taxonomy Categorization / Clustering Summarization III. CI Information Hub Demonstration Text Analysis Search Summary Q&A Copyright 2008 by Iknow LLC. 7

8 About Iknow Legal Name Description Mission Expertise Year Founded Corporate Headquarters Iknow LLC Business management and technology consulting firm focused in the knowledge management (KM) domain. Iknow helps companies leverage and transform their intellectual assets into sustainable knowledge-based competitive advantage. Document management, content management, business process management (BPM) and workflow automation, corporate and enterprise portals, business and competitive intelligence, distance learning/e- Learning, collaboration and groupware, digital asset management, text analysis, taxonomy and metadata management, search, and other new and emerging information technologies Princeton, New Jersey, USA Copyright 2008 by Iknow LLC. Company confidential materials. Do not copy or distribute. 8

9 Iknow s Technology Expertise Copyright 2008 by Iknow LLC. 9

10 Knowledge Value Chain Source: The Knowledge Agency 2007 Copyright 2008 by Iknow LLC. 10

11 Section I Competitive Intelligence Business Process Stage 1 Stage 2 Stage 3 Discovery Relevant content is identified, collected, and converted it into documents. The document s metadata is generated and the document is categorized and indexed. Analysis and Synthesis Input documents are analyzed and synthesized and a CI report is created. This stage typically involves: Analysis tools Collaboration Workflow routing for review and approval Publishing the final CI report Delivery CI reports and other filtered content are made available to end-users through both push and pull delivery approaches. Push approaches deliver information through contentbased notification. Pull approaches involve userdirected search. 11

12 Source Identification and CI Report Template Creation Key Intelligence Topics Source Analysis CI Report Analysis CI Report Templates Information Sources Internet websites, including blogs and wikis Corporate intranet, including shared drives, erooms, and other collaboration News feeds Subscription services s Internal reports Purchased (external) reports Internal databases Purchased (external) databases Other field intelligence (e.g., a valuable comment from a supplier) 12

13 Stage 1: Information Discovery Information Sources Document Collection, Extraction and Conversion Document Preprocessing Automatic Tasks: Generate and/or extract document metadata Categorize the input document using one or more approaches Manual Tasks: Remove duplicate or non-relevant documents Fill in missing metadata and taxonomic entries Edit incorrect entries Validated Input Documents 13

14 Example of Document Metadata Copyright 2008 by Iknow LLC. 14

15 Stage 2: Information Analysis and Synthesis CI Report Templates Validated Input Documents Manual Tasks Access content from a variety of input documents Analyze content Synthesize content Produce draft reports Route reports for review and approval Perform review Load approved reports into document repository Approved and Published CI Reports (Output Document) Support Tools Search Analysis Charting Graphing Mapping Visualization Structuring Workflow 15

16 Stage 3: Information Delivery Approved and Published CI Reports Validated Input Documents Content Management System Notification and Routing Customer preferences Personalization campaign management Search Knowledge repositories Databases Indexes Push delivery based on an individualized personal profile. Pull delivery based on queries. Copyright 2007 by Iknow LLC. Company confidential materials. Do not copy or distribute. 16

17 Enabling Technologies Stage 1 Stage 2 Stage 3 Discovery Analysis and Synthesis Delivery Portal / Dashboard X X X Content Management X X X Text Analysis X X X Search X X X Analytical Tools X Collaboration X X Workflow Automation X X Intelligence Analysts End Users Copyright 2008 by Iknow LLC. 17

18 Section II Text Analysis Stage 1 Stage 2 Stage 3 Discovery Relevant content is identified, collected, and converted it into documents. The document s metadata is generated and the document is categorized and indexed. Analysis and Synthesis Input documents are analyzed and synthesized and a CI report is created. This stage typically involves: Analysis tools Collaboration Workflow routing for review and approval Publishing the final CI report Delivery CI reports and other filtered content are made available to end-users through both push and pull delivery approaches. Push approaches deliver information through contentbased notification. Pull approaches involve userdirected search. 18

19 Keyword-Based Technologies Keyword-based technologies have no understanding of the real content of the documents. For example, this paragraph: The study, of nearly 14,000 U.S. adults, found that higher blood levels of selenium were linked to a lower risk of death over 12 years, at which point the risk appears to increase. The findings, published in the Archives of Internal Medicine, support earlier studies linking selenium to lower risks of prostate, lung and colon cancers. And this paragraph: 12 14,000 a adults and appears Archives at blood cancers colon death earlier findings, found higher in increase Internal levels linked linking lower lower lung Medicine nearly of of of of of over point prostate published risk risk risks selenium selenium studies study support that The the The the to to to U.S. were which years Are considered to be the SAME!! Copyright 2008 by Iknow LLC. 19

20 Text Analysis Four Concepts 1. Entity Extraction. Entity extraction (also known as Named Entity Recognition (NER) and Entity Identification (EI)) seeks to identify and classify atomic elements in the text, called entities, into predefined categories. Examples of Common Entity Types Who: People, Positions, Social Security Numbers What: Companies, Organizations, Financial Indexes, Products (software, weapons, vehicles ) When: Dates, Days, Holidays, Months, Years, Times, Time Periods Where: Addresses, Cities, States, Countries, Facilities (stadiums, plants), Internet Addresses, Phone Numbers How Much: Currencies, Measures, Percentages Concepts (Global piracy, unstructured data ) Relations & Events (Person-Organization, Travel, M&A ) Categories (business, terrorism ) Copyright 2008 by Iknow LLC. 20

21 Text Analysis Four Concepts (continued) 2. Taxonomy. A taxonomy is a subject-based classification that arranges the terms in a controlled vocabulary into a hierarchy. The value of a taxonomy is that it allows related terms to be grouped together and categorized in ways that make it easier to find the correct term to use whether for searching or to describe an object. Example from MeSH All MeSH Categories Diseases Category Digestive System Diseases Liver Diseases Hepatitis Hepatitis, Alcoholic Hepatitis, Animal Hepatitis, Chronic Hepatitis, Viral, Animal + Hepatitis, Autoimmune Copyright 2008 by Iknow LLC. 21

22 Text Analysis Four Concepts (continued) 3. Categorization/Clustering. Automatic categorization is the process in which ideas and objects are recognized, differentiated and grouped into categories by a computer program. Ideally, a category illuminates a relationship between the subjects and objects. 4. Summarization. Automatic summarization is the creation of a shortened version of a text by a computer program. The output of this process contains the most important points of the original text. Summarization systems are able to create both query relevant text summaries and generic machine-generated summaries. Copyright 2008 by Iknow LLC. 22

23 Search Results Page with Text Analysis

24 Dynamic Summarization and Top Mentions

25 Entity Extraction and Clustering

26 Section III CI Information Hub Demonstration Today, we will demonstrate how a few steps in the Discovery Stage can be automated. Software Products Business Objects Text Analysis Raritan Technologies (search integration framework) Autonomy K2 (enterprise search) Content Sources Factiva Websites RSS Feeds Copyright 2008 by Iknow LLC. 26

27 Business Objects Text Analysis Uses sophisticated natural language processing. Combines lexicons with pattern recognition; based on deep understanding of language. Reads text documents in more than 220 file formats and in more than 30 major languages. Entity extraction analyzes the full text of documents, clustering results by people, places, organizations, concepts, and more. More than 35 pre-defined (out-of-the-box) entity types are available. Customize the system to cluster by industry-standard or company-specific taxonomies Workbench to create and test custom entities, relations, events, and taxonomies (using a hybrid learn-by-example and rules-based approach). Useful for answering complex questions, e.g., What companies are mentioned in conjunction with mine? What relevant M&A activities have occurred in the last week? What concepts are most commonly associated with my company in these news articles? What issues are my customers complaining about?

28 Raritan Technologies Search Integration Framework Raritan s Search Integration Framework is a software toolkit that is used to quickly develop feature-rich search applications. Reusable software components Pre-built connectors and adapters Benefits from using the Framework are: Easily integrate software products from many vendors Easily build customer-specific search applications Easily connect to hundreds of data sources, data bases, and applications Copyright 2008 by Iknow LLC. 28

29 Taxonomies In today s demonstration, we are using selected sections from the following four taxonomies. Medical Taxonomies ClinicalTrials.gov. Diseases taxonomy - 23 major classifications of diseases and conditions. Diseases hierarchy contains over 4,000 nodes and 100,000 rules. National Library of Medicine s Medical Subject Headings (MeSH). Comprehensive medical taxonomy contains over 300,000 nodes and rules. Business Taxonomies Library of Congress Subject Headings (LCSH). Class H (Social Sciences section) provides Business, Finance, Law and Sales and Marketing taxonomies. Factiva Company Taxonomy (for selected pharmaceutical companies). Note: Custom taxonomies can also be developed to match your company s unique businesses (e.g., molecules, drug names, product/brand names). Copyright 2008 by Iknow LLC. 29

30 Solution Highlights Advanced search and browse capabilities. Continuous update of information from selected web sites and online news sources. Data automatically classified by subject area. Document summaries that show you the "top things" contained within a full text search result Ability to save queries, set alerts, and to share queries within a work group. Advanced highlighting provides rapid insights into document relevance. Graphical reporting tools to provide instant visualization of key information trends. Role-based views focus on information relevant to a particular job function. Copyright 2008 by Iknow LLC. 30

31 Other Possible Applications Selected Examples R&D Marketing Call Center Manufacturing Purchasing Finance Legal Enterprise Competitive Analysis Sentiment Analysis, Buzz Tracking, Customer Intelligence, Competitive Analysis Common Cause Diagnosis, Regional Diagnosis Contract Analysis, Six Sigma Compliance, Warranty Claims Analysis Supplier Analysis, Bid Tracking and Analysis Regulatory Compliance, Fraud Detection, Insurance Claims Analysis e-discovery Data Fusion

32 Summary Today, we described an information hub for competitive intelligence. Reviewed the competitive intelligence business process Discussed how a variety of commercial-grade software products can be integrated together for intelligence applications Showed how a variety of source content can be processed using text analysis software. Benefits of this solution include: Quality Incorporates all corporate information multiple sources; multiple formats Text analytics helps make sense of what you're looking at Higher quality outputs and better business decision making Time Shorten the cycle time of finding and analyzing information Cost Significantly lower cost than manual processing Copyright 2008 by Iknow LLC. 32

33 Contact Information Iknow Dr. Bernard L. Palowitch, Jr. President (609) Raritan Technologies Mr. Barry Freindlich President (908) x10 Copyright 2008 by Iknow LLC. Company confidential materials. Do not copy or distribute. 33