IR Research: Systems, Interaction, Evaluation and Theories

Transcription

1 IR Research: Systems, Interaction, Evaluation and Theories Kal Järvelin School of Information Sciences University of Tampere, Finland Information Studie s and Inte ractiv e Media SCHOOL OF INFORM ATION SCIEN CES

2 Outline On Information Retrieval On Information Retrieval Research Interaction in Information Retrieval Evaluation in Information Retrieval Research Beyond Evaluation: Theories

3 I On Information Retrieval Out There The goal of information retrieval in practical life is to access information in order to better carry out human tasks (instrumental) Finding useful information for task performance is often a challenge Tools augment human capabilities How well do we understand information-intensive tasks and the position of IR in their augmentation?

4 Augmenting Humans Situation Work Task Context Augmenting Task Performance Tools Methodology Knowledge Education/Training

5 Augmenting Humans vs IR Situation Work Task Augmenting Task Performance Context? Tools Methodology Knowledge Education/Training Retrieve Docs DB Web

6 Task-based IR: Means - Ends Situation Work Task Context Access Information Tools Augmenting Task Performance Education/Training Methodology Knowledge -- An Example Available Docs Colleagues Create Documents Find Docs Other Collections Retrieve Docs DB Information Seeking Education/ training... Web Remember/Use Other Sources Information Systems DB Colleagues

7 Sample Work Task Process in Biotech WN WT BR:gn WT WT (Kumpulainen & Järvelin 2010)

8 Information Ecology

9 Information Ecology Resources serve varying ends ecological niches Design implications design for the infoecological niche design for typical demand design for concerted use

10 II On IR Research The ultimate goal of IR research is to create ways to support humans to better access information in order to better carry out their tasks. -> IR research is a branch of technology IR research has a constructive aspect (to create novel systems) and an evaluation aspect (are they any good?). My search engine is better than yours!

11 IR Engineering... Focus on retrieval models for document and request representations, and matching / ranking Database Search request Documents Representation Representation Query Matching Result

12 Engineering focus...

13 IR Retrieval Exact-Match Best-Match Document- Based Network- Based Structure- Based Structured Document Non-Formal Feature- Based Formal Cluster- Based Browsing Spreading Activation Hypertext Navigation Probabilistic Language Fuzzy Set Logic-based Vector-Based (Based on Belkin and Croft, 1987)

14 Engines at work User performance and user experience Collaborative use of vehicles

15 III Interaction - Systems / Information Searchers interact with systems no single shots A long tradition in Information Science online searching strategies IR expert systems Much recent interest in IR The real need is to interact with information

16 Session analysis Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13

17

18 IV Evaluation in IR Research Hallmark of IR research / experimentation Evaluation is the systematic determination of merit of something using criteria against standards object and goal can be set in many ways real life difficult to investigate -- thus controllable surrogate goals are employed in evaluation work task - search task - query execution work task quality vs. ranked list quality

19 Standardization -- A Success Story Simplification of the problem -> standardization of research designs tremendous success in IR research due to shared test data comparable test results Cumulating knowledge? Armstrong & al CIKM 09: reported improvements in performance don t add up

20 Cranfield: Output Performance Representation Documents Search request Relevance assessment Representation Database Query Matching Query Result Recall base Evaluation Evaluation Result

21 Output vs. Outcome Performance Documents Search request F a Context Relevance assessment Representation Representation c t o r s Database Query Matching Query Result Human task Task outcome Recall base Evaluation Evaluation Result Evaluation Evaluation Result?

22 Broader Frameworks Socio-organizational& cultural context Work task context Seeking context Docs Request Seeking Task Work Task Repr DB Match Repr Query Seeking Process Work Process IR context Result Seeking Result Task Result Evaluation Criteria: D: Socio-cognitive relevance; quality of work task result C: Quality of info & work process/result, learning B: Usability, quality of information/process, learning A: Recall, precision, efficiency, quality of process

23 V Beyond Evaluation: Theories Science is about developing theories understanding and explaining, making hypotheses and testing them Theories systematic collections of theoretical and empirical laws Scientific laws empirical laws express verified relationships between variables Variables represent objects, properties or events are used in hypotheses, laws... Hypotheses state verifiable facts / relationships whose truth is unknown.

24 Theory of Ranking Docs Request Explanandum Repr Repr quality of ranking DB Query Explanans IR context Match Result representations and matching models

25 IR Matching Exact-Match Best-Match Document- Based Network- Based Structure- Based Structured Document Non-Formal Feature- Based Formal Cluster- Based Browsing Spreading Activation Hypertext Navigation Probabilistic Language Fuzzy Set Logic-based Vector-Based

26 IR Theory at ECIR ECIR Theory Session 7 Balancing Exploration and Exploitation in Learning to Rank Online ReFER: effective Relevance Feedback for Entity Ranking The Limits of Retrieval Effectiveness Learning Conditional Random Fields from Unaligned Data for Natural Language Understanding ECIR Theory Session 15 Learning for Ranking Aggregates Efficient Compressed Inverted Index Skipping for Disjunctive Text-Queries Within-Document Term-Based Index Pruning with Statistical Hypothesis Testing SkipBlock: Self-Indexing for Block-Based Inverted List Weight-based Boosting Model for Cross-Domain Relevance Ranking Adaptation

27 Theory of Searching Docs Request Repr Repr DB IR context Match Query Result

28 Theory of Searching Docs Request Repr Repr DB Match Query E = r t 2 IR context Result

29 Theory of Searching Docs Request Explanandum Repr DB Match Repr Query ranking, satisfaction, searching behavior (process) IR context Result Explanans knowledge of topic, system, process, collection, collection & system features

30 Theory of Information Access Seeking context Docs Request Seeking Task Repr DB IR context Match Repr Query Result Seeking Process Seeking Result Explanandum satisfaction, access behavior (process), quality of information, learning Explanans knowledge of and of information ecology, barriers, objective ecology

31 Theory of Information Interaction Work task context Seeking context Docs Request Seeking Task Work Task Repr DB Repr Query Seeking Process Work Process Match IR context Result Seeking Result Task Result Explanandum: satisfaction, task process/result & their quality; Explanans: task type and stage and the previous; performer s knowledge on task

32 Ingwersen, P & Järvelin, K. The Turn. Springer 2005.

33 Task complexity and searching A sample unit theory task complexity affects information types needed and through this the number of sources, their type and their location Task complexity Information types needed Number of sources Source types Location of sources Byström, K & Järvelin, K (1995) Task Complexity Affects IP&M

34 Vakkari 2001 Stage Domain knowledge Task performance process Mental model -differentiation - integration Use of information IR systems knowledge Specificity of info sought Expected contribution Vocabulary support Search Tactics Choice of channels Search keys Operators Query formulation support Relevance judgments: degree, criteria, P/R Relevance and contribution of docs found/used

35 Pandora s box

36 W Y D S I W Y D U

37 VI Finally The MINDS report 07: developments, challenges, agenda Heterogeneous Data IR systems Heterogeneous must seamlessly Context integrate and correlate information Search engines across a are variety context of media, free. Need sources, to understand and the formats. Beyond the Ranked List user, People the domain move and from the finding larger documents task. Search toward is not information the end interaction. goal. What Do People Really Do? Tools Little needed for information transformation: clustering, linking, highlighting Evaluation is known about how people use information retrieval tools. social networks, summarizing and arranging. Evaluation of new tools will require development of new We metrics are bound and to methodologies. research methodologies defined in the 1970s.

38 What s needed? Development of evaluation methodologies The tradition may become a straight-jacket Descriptive empirical studies Understanding feeds evaluation Theories in broader scopes They can guide tool development and evaluation Solving easy problems does not make great history

39 Pandora s box

40 Thank you!