Full Text Indexing and Search

Transcription

1 IBM Software Solutions Group Full Text Indexing and Search Bud Paton - World Wide Technical Consultant Greg Rochocki - CMOD Swat Team US 08/2014

2 Full Text Index and Search in Content Manager OnDemand IBM Corporation

3 Why Full Text Search? How could we search before FTS? Meta data searching only! IBM Corporation

4 Meta data searching Indexes are extracted from the document and stored in the database String type search operators are limited to equal and like Limited to exact matches Limited to what the indexing engine can extract AFP, Line and PDF only IBM Corporation

5 Searching Using a Viewer IBM Corporation

6 Searching from the viewer Exact matches only (no wild cards) Must retrieve the document before searching Only practical if user knows text exists within the document Different search interface for each viewer IBM Corporation

7 Server Based Text Search IBM Corporation

8 Server based text search Exact matches only Consumes library server resources No highlighting (dynamic summary) support Scans document text Limited to AFP, Line and PDF IBM Corporation

9 Design goals for Full Text Index/Search Support complex queries Boolean searches, wildcard Searches, optional terms, fuzzy searches, proximity searches, Weighted searches (boosting) Highlighting/Dynamic summary Query score Additional data types (XML, MS Office, etc ) Cross platform Database independent Performance IBM Corporation

10 Full Text Search feature highlights Separately priced feature Atthe core is a new server Full Text Search (FTS) Server Standalone install Supported on AIX, Sun, Linux, zlinuxzosand Windows Based on Apache Lucene text search engine library IBM Corporation

11 Full Text Search feature highlights FTS Exporter ships with OD server Supported on all platforms OnDemand server interfaces directly with FTS Server for search Searching the index supported by all OD servers Extensive data type support IBM Corporation

12 Full Text Search server IBM Corporation

13 Full Text Search server cont Document preprocessing Text extraction AFP, Line exceptions Language identification - Determines the language of the document Tokenization Creates tokens from extracted text using UIMA (Unstructured Information Management Architecture) -Watson runs on top of UIMA IBM Corporation

14 Full Text Search server cont (UIMA is a component software architecture for the development, discovery, composition, and deployment of multimodal analytics for the analysis of unstructured information and its integration with search technologies developed by IBM. An example is a logistics analysis software system that could convert unstructured data such as repair logs and service notes into relational tables. These tables can then be used by automated tools to detect maintenance or manufacturing problems IBM Corporation

15 Full Text Search Collections: Documents are grouped into collections Most actions are performed against a collection Add documents to collection Remove documents from collection Search a collection IBM Corporation

16 ARSDOC funtions to Full Text Index FTI_ADD function Use this function to index all of the text in documents. Use an SQL string or Named Query to query the database and determine the documentsto be indexed. FTI_RELEASE function Use to remove documents from a being indexed. You can query the database by using an SQL string or Named Query to determine the documents that should be removed from being indexed IBM Corporation

17 Integration Design points Off load full text index processing so as not to impact existing server performance. How to full text index data New versus old (legacy) Grouping documents into collections Support full text index on ALL existing OD clients IBM Corporation

18 Integration cont Off load Full Text Indexing and Search processing New table in CMOD database created to hold work items Exported created to process work items ADD, REMOVE, UPDATE work items IBM Corporation

19 How to FTI data New Data Can be FTI at load time if configured in CMOD Application Group At load time FTI Work database is updated with new work Export will send data over to FTI Server when timer is reached (in secs) Legacy data Arsdoccan index data ODWEK Entire load or SQL search or Named Qurery JAVA API s has new class/methods ODFolder FTIAddHits FTIReleaseHits IBM Corporation

20 Integration Grouping Documents into collections One to one mapping between OD data table an FTS collections Collection name is instance name plus table name More efficient searching when combined with segment table IBM Corporation

21 Integration-Segmentation This allows the FTS index to scale horizontally. During a query operation we can narrow the scope of documents that need to be searched. If the end user specifies a date range in addition to their full text search criteria, the OD segment tables are referenced to determine which collections need to be queried. There are really two searches going on at once, the normal parametric search by CMOD and the FTS by the FTS engine. Results are combined at hit list IBM Corporation

22 Integration Full Text Search is supported on ALL CMOD clients CMOD Folder enhanced to support four new data field types: Full Text Index Search Full Text Index Score Full Text Index Highlight Context of matching text Full Text Index Summary First 80 characters of document IBM Corporation

23 Configuration Enable the OnDemandserver for FTS Enable Application Group for FTS indexing Enable Folder for FTS searching IBM Corporation

24 Enable CMOD Server for FTI Enable the OnDemandserver for FTS Configurator(Windows) Ars.cfg(all other platforms) ARS_SPPORT_FULL_TEXT_INDEX= IBM Corporation

25 Enable the CMOD Application Group FTI server name and port must be specified Default port is 8191 Must specified if automatically or not IBM Corporation

26 Enable the Application Group cont Create new field Data Type must be Small Int(s) Check box for Full Text Index IBM Corporation

27 Application Group cont Permission have to be granted for people to use FTS IBM Corporation

28 FTS Folder setup There are four new Field types in the folder to allow people to do Full Text search: Full Text Index Search Full Text Index Score Full Text Index Highlight Full Text Index Summary IBM Corporation

29 Searching Scenarios Boolean database OR data base NOT log file Wildcard m?re(finds mare, mere, mire and more) (0 or more characters) Optional log %file (finds log and optionally file) Fuzzy analytics~ (finds analytics, analyze, analysis, etc ) Proximity IBM WebSphere ~7 (finds IBM and WebSphere within seven words of each other) IBM Corporation

30 Searching IBM Corporation

31 Results IBM Corporation

32 Best Practices for FTS IBM Corporation