CONFIGURATION MODEL FOR FOCUSED CRAWLERS IN TECHNOLOGY INTELLIGENCE

Transcription

1 CONFIGURATION MODEL FOR FOCUSED CRAWLERS IN TECHNOLOGY INTELLIGENCE GÜNTHER SCHUH Fraunhfer Institute fr Prductin Technlgy IPT, Germany ANDRÉ BRÄKLING Fraunhfer Institute fr Prductin Technlgy IPT, Germany (Crrespnding) TONI DRESCHER KEX Knwledge Exchange AG, Germany ag.cm Cpyright 2015 by Fraunhfer Institute fr Prductin Technlgy IPT and KEX Knwledge Exchange AG. Permissin granted t IAMOT t publish and use. ABSTRACT Due t a steady increase f cmpetitive cnstraints caused by nging glbalizatin and dynamically grwing markets, technlgy intelligence has becme an imprtant element f strategic business intelligence. The bjective f technlgy intelligence is t fcus n the systematic identificatin f future chances but als threats t cmpanies caused by new technlgies and further technlgy develpments. T perate technlgy intelligence efficiently, access t up t date, relevant, and sufficiently cmplete infrmatin is essential. Indeed, availability f infrmatin is higher than ever by reasn f digitalizatin. Hwever, it als causes the prblem f infrmatin verlad. The available mass f data has t be searched, assrted and assessed t identify the actual needed infrmatin. In additin, the entire infrmatin prcessing has t be cntinued permanently r t be repeated fr each new bject f investigatin, therwise the validity f the results is nt given any mre. Accrdingly, it appears reasnable t autmate this prcess by widely using smart sftware slutins. One f the prmising appraches is fcused crawling which nt just runs thrugh given data surces in the web, but als rates each data recrd t make an autnmus decisin, which infrmatin is relevant fr the further prcess, and which data recrds shuld reasnably be analyzed next. T implement such crawlers, different appraches exist in the field f infrmatin retrieval: Fr example, different rating and discvery algrithms. This paper presents the status qu f nging research t develp a cnfiguratin mdel fr fcused crawlers t fulfill the varying requirements f technlgy intelligence tasks. At first, the assessment criteria fr infrmatin in a technlgy intelligence prcess and the cnfiguratin pssibilities f fcused crawlers are described. As a result, a first apprach f a matching between the requirements f technlgy intelligence tasks and the cnsequences f different fcused crawler cnfiguratins is presented. Clsing, the paper explains hw this apprach will be imprved and validated in case studies prspectively. Key wrds: Technlgy Intelligence, Fcused Crawler, Infrmatin Overlad, Smart Sftware Page 832

2 INTRODUCTION Due t the nging glbalizatin, dynamically grwing markets, and the emergence f new technlgies, cmpetitive pprtunities and cnstraints are increasing rapidly. Derived frm this, the necessity f an effectively and efficiently strategic business intelligence prcess is made bvius. Related t technlgy ventures, especially technlgy intelligence is indispensable t systematically identify prspective chances but als threats by new technlgies and technlgy develpments, as explicated by Wellensiek (2011). Determinatin f infrmatin needs Infrmatin search Infrmatin assessment Cmmunicatin f infrmatin Result s Figure 1: The fur steps f the technlgy intelligence prcess accrding t Wellensiek (2011). The prcess f technlgy intelligence is divided int fur steps, as shwn in figure 1. First, the infrmatin needs have t be determined. They are affected by the cnducting cmpany s business and technlgy strategy. During the step f infrmatin search, as much relevant infrmatin as pssible related t the current infrmatin need is brught tgether using different infrmatin surces. On this basis, a further infrmatin analysis is perfrmed. Finally, the results f the analysis are prepared t be cmmunicated, e.g., as a final reprt. Because f the digitalizatin and netwrking in tday s infrmatin age, infrmatin is available easily. At first glance, this seems t be an advantage fr technlgy intelligence. Hwever, this als causes the prblem f infrmatin verlad. The fllwing example shws the dramatic increase f data: Until beginning f 2012, the US Library f Cngress cllected data in the amunt f 374 terabytes (US Library f Cngress, 2013). This is abut 73.5 millin times the cmplete wrks f William Shakespeare 1. The Internatinal Data Crpratin (IDC, 2014) estimated the digital universe in 2013 as 4.4 zettabytes, and frecasts it will reach 44 zettabyte in This amunt f data is bviusly nt manageable manually anymre. Therefre, the practical challenge is t make these data useful t technlgy intelligence by cmputer aided research. Fcused Crawlers prvide such an pprtunity t supprt research prcesses. They are a smarter implementatin f a web crawler. A simple web crawler cllects all available dcuments in cnnected surces (e.g., the internet) by extracting and fllwing all references in these dcuments. Whereas a fcused crawler nly cnsiders thse dcuments in its surces, that are relevant in the given research tpic, as described by Chakrabarti et al. (1999). The necessary relevance assessments are pssible by different statistical and linguistic algrithms. In additin, Batsakis et al. (2009) and Pant/Srinivasan (2005) pint ut that the sftware requires an apprpriate training related t the specific tpic. The effectivity and the efficiency f a fcused crawler depend n the selectin f its cmpnents in terms f adequate algrithm implementatins and training data sets related t the current tpic and task. Sme f the basic cncepts and appraches f fcused crawlers are described in Micarelli and Fabi (2007). 1 Based n the UTF 8 text file The Cmplete Wrks f William Shakespeare, which is available at Page 833

3 Accrding t Srinivasan et al. (2005), the sftware s effectivity is measured by its recall rati and its precisin, the efficiency is measured by the caused csts in the frm f resurces (CPU time, memry requirements). On the ther hand, the requirements are defined by the different basic activities f technlgy management (scanning, mnitring, and scuting) as well as by the determined infrmatin needs. Currently, there are many implementatins f fcused crawlers intended fr specific use cases. T use these implementatins in ther cntexts, usually cmpnents f their cnfiguratin have t be changed, e.g., new training data has t be defined r even algrithms have t be replaced. Therefre, fcused crawlers are nt easy and quick t use. In this article we present ur nging research prject abut the usage f fcused crawlers in technlgy intelligence prcesses. Due t the mentined cnstraints, the target f this prject is a cnfiguratin mdel t supprt the cnfiguratin f fcused crawlers by chsing apprpriate cmpnents that fit the different requirements f technlgy intelligence tasks. APPLICATION OF FOCUSED CRAWLERS IN TECHNOLOGY INTELLIGENCE As mentined befre, the requirements fr the usage f fcused crawlers in technlgy intelligence are defined by the different basic activities scanning, mnitring, and scuting. Wellensiek et al. (2011) describe these activities as fllws: i. Technlgy scanning is an undirected but cnstant search prcess. It aims t give an verview abut currently unknwn but ptentially relevant technlgy infrmatin. E.g., a cmpany can scan trending tpics like industry 4.0, big data r additive manufacturing independently f their current relevance t identify new pprtunities and cnstraints in time. ii. iii. Technlgy mnitring is a lng term activity t bserve a previusly identified relevant field f technlgy infrmatin (e.g., by a technlgy scanning) and perfrms a mre specific search fr new infrmatin related t this field. S a cmpany can mnitr a new technlgy that is very interesting but currently t expensive in prductin t becme an early adpter if the manufacturing cst drps. Technlgy scuting aims t get fast and detailed infrmatin abut a very specific technlgy by rder. A scuting is dne by a cmpany if an explicit decisin has t be made, e.g., abut a market entry r an investment. Each f these basic activities perfrms the fur steps f technlgy intelligence prcesses, as explained in Spath et al. (2010) and als shwn in figure 1: i. Determinatin f infrmatin needs: First, a search field strategy has t be defined as explicated in Schuh et al. (2009). The search fields depend n the cmpany s cre capabilities and als n its business and technlgy strategy. They can be used as rientatin guide, e.g. t specify the fields f bservatin, the required level f detail, and the used methds. Lichtenthaler (2005) examined the selectin f technlgy intelligence methds in 26 large technlgy intensive cmpanies in Eurpe and Nrth America. ii. Infrmatin search: Based n the defined infrmatin needs, feasible surces f infrmatin have t be chsen, the necessary amunt f data has t be stated and the actual research is Page 834

4 perfrmed. Surces f infrmatin can fr instance be patents, publicatins, and websites, but als cnferences, fairs, expert interviews and discussins. iii. iv. Infrmatin assessment: Afterwards, the cllected infrmatin has t be rated by its relevance fr the current activity t extract a reasnable and manageable subset. This subset has t be cnsidered and interpreted t deduce the prcess s key findings, e.g., abut a mnitred technlgies maturity. Cmmunicatin f infrmatin: Cncluding, the results have t be prepared in an apprpriate frm fr its audience. S they can be used in a management summary t supprt upcming decisins, but als t define a new, mre specific research prcess, e.g., if a scanning prcess identified pssible pprtunities befre. Determinatin f infrmatin needs Infrmatin search & assessment Cmmunicatin f infrmatin Result s Cnstant repetitin Figure 2:Technlgy intelligence prcess supprted by a fcused crawler (hatched area). The assumptin is, that a fcused crawler is useful in the steps f infrmatin search and assessment in a cmbined and repeated frm (see figure 2), because the fcused crawler rates its findings already during the nging research prcess, especially, t decide which references t fllw next, as described in Rawat/Patil (2013). Furthermre, the repetitin prvides a cnstantly updated database with manageable effrt. Just the final cnsideratin and interpretatin has still t be dne manually, f curse. But t get meaningful results, it is essential that the crawler is well cnfigured in terms f the selectin f cmpnents that meet the search field s requirements. These cmpnents and their implied crrelatin t the technlgy intelligence prcesses are described in the fllwing paragraph. ASPECTS OF A CONFIGURATION MODEL FOR FOCUSED CRAWLERS IN TECHNOLOGY INTELLIGENCE T define the cnfiguratin mdel, factrs and requirements related t bth the prcess f technlgy intelligence and fcused crawlers have t be identified. Fr that purpse, a basic fcused crawler infrastructure is intrduced subsequently befre pssible cnnectins and resulting factrs are discussed. Infrastructure f Fcused Crawlers Figure 3 shws an exemplary fcused crawler infrastructure which cnsists f 4 layers, see Pant/Srinivasan (2005). Based n the infrmatin needs, training data and the crawler cnfiguratin will be defined as input data. Afterwards, the data will be analyzed in the intelligence layer and a decisin is made, which references will be fllwed next. These references (e.g., URLs) will be depsited in the s called frntier, wherefrm the netwrk layer will get and fllw ne after anther, s the discvered cntents can be stred in the repsitry. The parsing and extractin layer extracts all necessary infrmatin frm the retrieved cntents, befre the representatin layer Page 835

5 cnverts them int a reasnable and machine readable frm. This frm is the basis fr the next analysis by the intelligence layer, whereupn the prcess begins again. Intelligence INPUTS Needs Crawling lp Frnt ier Representatin Parsing and extractin Histry & page repsitry Netwrking Result s Figure 3: A fcused crawling infrastructure accrding t Pant/Srinivasan (2005) and Schuh et al. (2014). All these layers can be implemented by different, in parts als cmbined algrithms. Mrever, the selectin and preparatin f the input data has a crucial impact n the results f the crawling prcess, cmpare Rawat/Patil (2013) and Srinivasan et al. (2005). We call all these interchangeable parts cmpnents which shuld be cmpsed by ur intended cnfiguratin mdel. A further explanatin f a fcused crawler s mde f peratin is (amngst thers) described in Pant/Srinivasan (2005), Zhuang et al. (2005), Pal et al. (2009) and Singh/Tyjagi (2013). Requirements twards Fcused Crawlers Accrding t Srinivasan et al. (2005) the results f a fcused crawling are evaluated by recall, precisin, and cst. Precisin describes the fractin f retrieved results that are relevant, and recall describes the fractin f relevant results that are actually retrieved, as stated by Manning et al. (2008, pp ). Cst includes parameters like CPU lad, memry usage, data strage, and similar parameters. Schuh et al. (2014) describe recall, precisin, and cst as interdependent: A better recall requires unavidably mre memry, and lw cmputing pwer can be cmpensated by reducing the precisin. As well, a high recall can cause a lw precisin, because strict relevance cnditins may result in s called false negatives (i.e., relevant dcuments which are rated as irrelevant by mistake). In cnsideratin f this interdependence, the described evaluatin parameters can be used t define a specific task s requirements twards fcused crawlers. Page 836

6 Infrmatin Needs and Infrmatin Search Crrespnding t Wellensiek et al. (2011), at first the search field has t be stated t describe the actual infrmatin needs, because it has an apprpriate impact n the subsequent research prcess. Schuh et al. (2009) enumerate fllwing parameters fr cmpany specific infrmatin needs: the basic characteristic f the cmpany, its technlgical base, the technlgical and scientific envirnment and the available ffer f infrmatin. Befre the actual research within the defined search field can start, the infrmatin surces have t be chsen, als accrding t Wellensiek at al. (2011). Hence, it is necessary t prve if and hw these surces can be accessed. Als, the level f infrmatin detail has t be defined matching the prcessed activity and its requirements. A cmprehensive netwrk cnfiguratin mdel t set up an ptimized technlgy intelligence netwrk by integratin f specific infrmatin surces is develped by Saxler (2011). Requirements twards Technlgy Intelligence Prcesses Schuh et al. (2009) mentin infrmatin cntent, earliness, exclusiveness, and the infrmatin s usefulness as base fr decisin making and thus as requirements twards technlgy intelligence prcesses. Regarding t the cmpany s technlgy strategy, these requirements have different specificatins. E.g., a cmpany which wants t be the technlgy pineer in its target market needs access t early and exclusive infrmatin. Step 1: Define fcused crawler requirements based n the given technlgy intelligence requirements. Infrmatin cntent Earliness + + Step 2: Chse a fcused crawler cnfiguratin based n the requirements determined in step 1. Recall factr Precisin Figure 4: First cncept f a fcused crawler cnfiguratin matrix. Figure 4 shws an early cncept which utlines a fcused crawler cnfiguratin matrix. Using such a cnfiguratin mdel, the requirements which are based n the current search field have t be translated int the specific task requirements twards fcused crawlers (step 1). Fllwing, the apprpriate fcused crawler cmpnents like algrithms and training data sets can be chsen t build a practical crawler slutin (step 2). Page 837

7 CURRENT FINDINGS The requirements twards technlgy intelligence prcesses and twards fcused crawlers can be harmnized with each ther. Fcused crawlers built f different cmpnents (relevance rating algrithms, cntent parsing and extractin algrithms, training data, etc.) can be assessed by recall, precisin and cst. After this assessment is dne related t the requirements and factrs f technlgy intelligence prcesses, an ptimized fcused crawler cnfiguratin t slve the specific task can be determined. CONCLUSION Because f bth, the amunt and the hetergeneity f the meanwhile available infrmatin, autmated r cmputer aided prcessing methds are indispensable. By a reasnable integratin f existing IT slutins, first appraches t ptimize research prcesses can be implemented. On this basis, new and cntinuative slutins can be develped. This demands the admissin f such slutins as part f the particular research field which intends t handle the challenges f infrmatin verlad. Already in this early state, the thughts abut fcused crawlers in technlgy intelligence shw that existing prcesses can be supprted significantly. OUTLOOK Next, the early cncept f a cnfiguratin matrix has t be advanced t a first usable draft, which allws an intuitive cnfiguratin f a required fcused crawler slutin. This matrix will be validated and further ptimized in at least tw case studies. On the ne hand, it will be used in a typical technlgy intelligence prcess as mentined in this article. On the ther hand, a lng term test will be passed with a by and by ptimized fcused crawler running t supprt research activities in the Cluster f Excellence Integrative Prductin Technlgy fr High Wage Cuntries. ACKNOWLEDGMENTS This wrk was perfrmed as part f the Cluster f Excellence Integrative Prductin Technlgy fr High Wage Cuntries, which is funded by the excellence initiative by the German federal and state gvernments t prmte science and research at German universities. REFERENCES Batsakis, S., Petrakis, E. G. M., and Milis, E., (2009), Imprving the Perfrmance f Fcused Web Crawlers. Data & Knwledge Engineering, 68(10), pp Chakrabarti, S., Berg, M. van d., and Dm, B., (1999), Fcused crawling: a new apprach t tpicspecific Web resurce discvery. Cmputer Netwrks, 31, pp Internatinal Data Crpratin, (2014), The Digital Universe f Opprtunities: Rich Data and the Increasing Value f the Internet f Things, [5 Nv 2104]. Lichtenthaler, E., (2005), The Chice f Technlgy Intelligence Methds in Multinatinals: Twards a Cntingency Apprach. Internatinal Jurnal f Technlgy Management 32(3/4), pp Manning, C. D., Raghavan, P., Schütze, H., (2008), Intrductin t Infrmatin Retrieval. Cambridge: Cambridge University Press. Page 838

8 Micarelli, A., and Gasparetti, F., (2007), Adaptive Fcused Crawling. In The Adaptive Web, P Brusilvsky, A Kbsa, W Nejdl (eds.), pp Berlin: Springer. Pal, A., Tmar, D. S., Shrivastava, S. C., (2009), Effective Fcused Crawling Based n Cntent and Link Structure Analysis. Internatinal Jurnal f Cmputer Science and Infrmatin Security (IJCSIS) 2(1). Pant, G., and Srinivasan, P., (2005), Learning t Crawl: Cmparing Classificatin Schemes. ACM Transactins n Infrmatin Systems 23(4), pp Pittsburgh: ACM. Rawat, S., Patil, D. R., (2013), Efficient Fcused Crawling based n Best First Search. 3 rd Internatinal Advance Cmputing Cnference (IACC), pp Ghaziabad, IEEE. Saxler, J., (2011), Gestaltungsmdell für Netzwerke zur Technlgiefrüherkennung. Aachen: Apprimus. Schuh, G., Orilski, S., and Wellensiek, M., (2009), Efficient Technlgy Intelligence by Search Field Strategies. The XX. Internatinal Sciety fr Prfessinal Innvatin Management ISPIM Cnference. Wien : ISPIM. Schuh, G., Bräkling, A., and Apfel, K., (2014), Identificatin f Requirements fr Fcused Crawlers in Technlgy Intelligence. Prceedings f PICMET 14. San Jse : IEEE. Singh, S., and Tyjagi, N., (2013), A Nvel Architecture f Mercatr: A Scalable, Extensible Web Crawler with Fcused Web Crawler. Internatinal Jurnal f Cmputer Science and Mbile Cmputing (IJCSMC) 2(6), pp Spath, D. (Ed.), Schimpf, S., and Lang Ketz, C., (2010), Technlgiemnitring. Stuttgart: Fraunhfer IAO. Srinivasan, P., Menczer, F., and Pant, G., (2005), A General Evaluatin Framewrk fr Tpical Crawlers. Infrmatin Retrieval, 8(3), pp US Library f Cngress, (2013), Annual Reprt f the Librarian f Cngress fr the fiscal year ending September 30, Wellensiek, M., Schuh, G., Hacker, P. A., and Saxler, J., (2011), Technlgiefrüherkennung. In Technlgiemanagement, G Schuh, S Klappert (eds.), pp Berlin: Springer. Zhuang, Z., Wagle, R., Lee Giles, C., (2005), What s There and What s Nt? Fcused Crawling fr Missing Dcuments in Digital Libraries. Digital Libraries, Prceedings f Jint Cnference f Digital Library (JCDL 05), pp Page 839