Towards Planning and Execution for Information Retrieval

Transcription

1 Toward Planning and Execution for Information Retrieval Laurie S. Hiyakumoto and Manuela M. Veloo Computer Science Department Carnegie Mellon Univerity Pittburgh, PA, USA 523 hyaku, Abtract The problem of information gathering ha received coniderable attention from the planning community in recent year. However, reearch in thi area ha generally aumed a uer information goal i perfectly repreented by the query, and typically adopt a relational databae model for repreenting query operation and information ource. In thi paper, we preent a planning and execution framework intended to addre the more general information retrieval (IR problem in which querie are only approximate repreentation of the uer true information goal, and complete model of the content of information ource are not available. Our approach reformulate traditional IR technique uch a ynonym expanion and relevance-feedback a domain operator, explicitly modeling the uncertainty of action outcome and of atifying the uer information need, and taking into account reource contraint uch a time. A forwardchaining planning and execution algorithm guide action election, execution and replanning deciion uing a uerdefined utility function. We have implemented thi approach in the INSPIRE (INtegrated Sytem for Planning and Information REtrieval architecture. Introduction Traditional ad-hoc approache to information retrieval (IR are becoming increaingly inadequate for today large, dynamic, heterogeneou document collection, the mot obviou example of which i the World Wide Web. Current IR ytem place mot of the burden on the uer, relying on them to identify ource likely to contain relevant information, compoe an appropriate query, and ift through retrieved document to extract relevant information. Clearly, a document collection continue to grow, it will become impractical for uer to perform thee tak for all but the implet requet. Even today there are more ource than a peron could poibly acce in a reaonable amount of time, many of which contain redundant, irrelevant, outdated, or even erroneou information. Ideally, a uer hould be able to tate a high-level requet to an IR ytem and it would take care of the ret. Although we are far from achieving thi goal today, one promiing approach i to ue planning to automate more of the IR pro- Copyright c 2002, American Aociation for Artificial Intelligence ( All right reerved. ce. In addition to the obviou benefit of reducing uer effort, a planning approach ha two important advantage over traditional approache. Firt, it provide a tructured framework for learning which IR tool are bet-uited for different type of retrieval tak. Second, it free ytem developer to try radically different IR technique without worrying about their comprehenibility to novice uer. However, before a planning and execution ytem can uccefully apply to IR, it mut addre everal challenge preented by the tak: Incompletely-pecified goal: A query i often a poor approximation of the uer true information goal, and different information goal may be expreed by identical querie. Partial and incremental goal atifaction: For many information goal, retrieval i not an all-or-nothing propoition. Partial atifaction of a uer requet i common and may be achieved incrementally via multiple iteration. Incomplete knowledge of information ource content: It i virtually impoible to determine the full extent of information available from many large heterogeneou collection (e.g. the web page indexed by Google. Uncertain action outcome: The performance of IR technique varie from one ue to the next. For example, ynonym expanion will produce better reult on ome querie than other, and retrieval time will vary with network and machine load. Reource contraint: Mot uer are not willing to wait more than a few econd for a repone unle the information i extremely valuable to them. Uer interaction: Succe i uncertain until we actually execute a plan and receive feedback from the uer. An implementation capable of interleaving execution with planning i eential, a i upport for interaction with, and poibly intervention by the uer. Thi paper preent an integrated planning and execution framework that addree thee iue. Our domain operator are drawn from traditional IR technique uch a ynonym expanion and relevance feedback. We define an abtract repreentation of the uer goal and reource contraint uing a et of real-valued metric and a utilityfunction pecifying their relative importance. A forwardchaining planning and execution algorithm earche through

2 :name ( expand-query-with-ynonym :parameter ((?q query ytem-time-pent et-query-quality :precondition ( :ymbolic ( (not (applied-op-expand-with-ynonym?q :metric ( (< et-query-quality 0.8 :effect ( 0.70 (:ymbolic ( (del (not (applied-op-expand-with-ynonym?q (add (applied-op-expand-with-ynonym?q :metric ( (et-query-quality += 0.3 (ytem-time-pent += (:ymbolic ( (del (not (applied-op-expand-with-ynonym?q (add (applied-op-expand-with-ynonym?q :metric ( (et-query-quality -= 0.2 (ytem-time-pent += (:ymbolic ( (del (not (applied-op-expand-with-ynonym?q (add (applied-op-expand-with-ynonym?q :metric ( (ytem-time-pent += 0.5 :execute ( ynonym-expanion?q Figure : Operator pecification for expand-query-with-ynonym. the pace of belief tate, et of poible tate and their aociated likelihood, uing expected utility and belief tate uncertainty to guide action election, execution, and replanning deciion. Thi work contribute to planning reearch in two way. Firt, it identifie planning challenge poed by the general IR tak and preent a model that addree them. Second, it preent an approach to handling incompletely-pecified goal uing an integrated planning and execution proce. The remainder of the paper i organized a follow. Firt, we decribe the domain and problem repreentation we ue. Secondly, we preent our planning and execution algorithm. We then briefly decribe the INSPIRE architecture that implement the approach, and dicu ongoing work in learning parameter to etimate the efficacy of domain operator. We further dicu the relationhip between our work and previou reearch in planning and execution. We conclude with a dicuion of current and future reearch direction. The IR Planning Domain An IR planning problem conit of a domain definition and a problem tatement. The domain define the problemindependent knowledge of the IR tak which i common to different retrieval requet. The problem tatement define a problem-pecific earch context, the uer information goal, and reource contraint. In thi ection, we decribe our repreentational choice for each. Domain An IR domain conit of: A et of type defining clae of object (e.g., a QUERY A et of metric declaring reource, cot (e.g., (ELAPSED-TIME 5 and quality etimate that will be monitored within tate A et of operator defining the available atomic action Metric are declared a part of the domain decription, and initialized within a problem tatement. Like literal, they may be altered a either a ide-effect or a primary effect of action. In addition to repreenting conumable reource and execution cot, we alo ue metric to define pecial quality etimate that are ued to etimate how well the current tate of knowledge will achieve a particular ubgoal (e.g., (et-query-quality 0.3. Repreenting knowledge goal in thi way let u compare the relative value of different tate when the goal are only partially atified. (Dicuion of how thee quality etimate can be generated i deferred to a later ection. For now, we will jut aume the etimate are available to u. An operator i defined by it precondition and effect, plu an execution function that declare the procedure that will actually be called during execution, along with the operator binding to pa a argument. Precondition are conjunction of literal and metric expreion. Literal precondition may contain typed variable, negation, and inequality contraint on numeric feature. Metric precondition pecify inequality contraint on the metric value. Effect conit of literal to be added to or deleted from the tate, and update function to apply to metric. We can expre uncertainty in thee effect by declaring multiple effect et that enumerate all poible et of outcome and the joint probability ditribution for each. A complete pecification for a imple expand-querywith-ynonym operator i hown in Figure. Thi operator i ued to revie a query by adding ynonym of individual query term. For example, if applied to the ingleword query car, we obtain a new query containing car, auto, automobile, machine, motorcar. It take one variable of type QUERY, modifie two metric, and ha three poible outcome: a tate in which the query quality ha improved, one in which it ha remained unchanged (becaue no ynonym were available, and one in which the quality degrade. A metric precondition on etimated-query-

3 G i C i 0 :object ((Q query (LYCOS earch-engine (GOOGLE earch-engine :util ((ytem-time-pent C (uer-time-pent C2 (et-reult-quality G (et-query-quality G2 :init (:ymbolic ((available-earch-engine GOOGLE (available-earch-engine LYCOS (query-confidence Q 0.2 (query-term Q 2 :metric ((uer-time-pent 0 (ytem-time-pent 0 (et-query-quality 0.2 (et-reult-quality 0 0 ki :goal (:util-criteria (threhold 0.4 :ucc-criteria (threhold 0.8 :utilfn (( C (0 C2 (00 G (20 G2 Figure 2: Sample problem tatement pecifying the initial tate, goal criteria and utility function to ue. quality limit applicability to querie whoe quality i below the given threhold. We alo pecify a literal precondition and effect that retrict it to one-time application, a thi technique i generally only effective the firt time it i ued on a particular query. Problem Statement A pecific problem intance i defined by a et of available object and their type, an initial tate decription, and a goal pecification. The initial tate declare the et of literal fact that are known to be true, and aign initial value to each of the metric. A goal pecification define the three component needed to evaluate the ucce or failure of the planning and execution eion: A function that etimate tate utility from the current metric value in the tate A minimum utility value for ucceful termination A minimum atifiability threhold The utility function define the relative importance of the knowledge goal, cot, and reource. It i ued to etimate progre toward our information goal, and guide the action election proce by comparing the expected utilitie of the reulting tate. Thu, to be ueful, a utility function mut accurately reflect the uer goal by providing a relative ordering on the tate conitent with the uer preference, correctly mapping goal tate to high utility value and non-goal tate to low value. Our domain language currently limit utility function to weighted combination of function for individual metric in the domain, each of which produce a normalized value between zero and one: The utility threhold! pecifie the minimum utility value required for a atificing olution. Any executed equence with a utility value greater than thi i aumed to have achieved the goal. For example, the problem tatement hown in Figure 2 define a imple utility function compried of two cot metric and two quality metric. Each metric ha it own utility function (e.g., ytem-time-pent map to "$#, and the goal tatement declare the relative weight to aign to each. The diagram at the left of the tatement depict two very imple mapping; other are poible. In addition to a utility threhold, a goal declare a atifiability threhold between zero and one, indicating how confident we mut be in our ucce likelihood before we can declare planning ucceful. A lower value indicate a greater tolerance for riking a lower utility outcome and wated execution. We elaborate on the relationhip between thi and our planning and execution algorithm in the next ection. Interleaving Planning and Execution A previouly noted, the incompletely-pecified knowledge goal of information-eeking tak neceitate ue of a forward-chaining algorithm. Moreover, although we can etimate the likelihood that our plan ucceed, until we receive feedback from the environment (via evaluation of retrieved document and explicit uer-feedback, we cannot be certain we have actually achieved the goal. Thu, execution and replanning mut be an integral part of the planning proce.

4 / X Q U e / U / U / X U We ue an integrated planning and execution algorithm that perform a bet-firt earch acro belief tate. Our algorithm, preented in Table, i upplied with a domain model, and a problem tatement coniting of: an initial belief tate, a uer-defined utility-function, a utility ucce threhold!, and a value pecifying a confidence threhold for termination. GenerateOrExecutePlan(domain, belief tate, utility function, goal threhold, atifiability threhold. Initialize the current belief tate and candidate ucceor belief tate!!"$##%'&(#*, / 8 2. If ProbabiliticallySatifie(,, 0, UnexecutedActionSequence( If ( : Execute(PopFront( Update(,, 9 If (ChooeReplanOrContinue(, == replan ;8 GenerateOrExecutePlan(, 0, < * 9,, Ele goto 2. Ele return ucce. 3. Ele if (ReachedSearchLimit( or ( and UnexecutedActionSequence( return failure. 4. If (ChooeExecutionOrExpanion(, 5 5 expand - -?> ChooeBetOne( = /A@!!"$##%'&(#*,+ +. 0/ 5. Ele Goto UnexecutedActionSequence( 9 Execute(PopFront( Update( ;8,, 9 If ChooeReplanOrContinue( ;8, == replan GenerateOrExecutePlan(, 9,, 0, Ele goto 2. Table : The planning and execution algorithm. Beginning with the initial belief tate a the root and an empty plan, the planner evaluate all ucceor belief tate reachable from the current tate by applying a ingle operator, and elect the one with the highet expected utility. The current belief tate BDCE i updated to the projected belief tate, and the election proce repeat. The actual plan, the equence of operator application required to tranform the initial tate into the projected current belief tate B CE, i implicitly maintained within each belief tate by toring it generating operator and parent belief tate. At each tep, the algorithm conider the tradeoff between executing the firt unexecuted operator in the plan and continuing to plan with the uncertain outcome of the projected belief tate. If an execution tep i carried out, it i followed by an aement of the need for replanning. The algorithm terminate when all tep in the plan have been executed and the confidence and utility threhold for goal atifaction are met, or there are no additional action the planner can take. We now provide detail on the belief tate repreentation and upporting function. State Repreentation We repreent the tate-pace at two level: the individual tate level, at which individual operator are applied, and the belief tate level, at which the planning and execution algorithm operate. An individual tate conit of grounded metric and literal. (Note that we do not make a cloed-world aumption; the abence of a fact doe not imply it falehood. In turn, a belief tate conit of a finite et of individual tate repreenting all poible current tate and their correponding likelihood. Thi give u the ability to repreent the degree of uncertainty in our tate knowledge. During the planning and execution proce, new belief tate are generated by the Succeor( function hown in Table 2. Succeor(domain, belief tate, utility function. Generate the et of all applicable operator F for the current belief tate. F 5 G P*Q R Q S Q.TVUW XZYQ W\ T^WIU_$Q.HI KJ <.L(MON `ab e ]/.+ /dc 2. For each operator X]S F, generate a ucceor belief tate J, and calculate it expected utility fg J 5 P hir*ja0sk!yxzyu ju SkU lvlmu_.y okp YUWqX o h /.+ /dn / 5 okp YU*W X o X'aZab > a'u o U!YUq /$@ r U!YW p _!h / 5 X*T\T ot r l^um` /.+ r U!YW p _ h // / 5 TmX'WqU`Y Jwv X'_.Y p Qq` Jwv Jwv fg 5 h h / xyhi KJ <.L(MzN 3. Return the et of ucceor belief tate. Table 2: The Succeor algorithm. Given an applicable operator, the function imulate the effect of applying it to the current belief tate B CE. The reult i a new belief tate containing the et of all poible outcome tate and the probabilitie of eeing each outcome. Note that a pair of outcome tate within a belief tate may contradict one another. However, once we actually execute / ;

5 the plan tep leading to thi belief tate, only a ingle outcome tate will remain. An operator i conidered to be applicable in the current belief tate if all of it precondition are atified in every tate within the belief tate. Thi enure we have a valid plan regardle of which tate we are actually in. Although it i not explicitly repreented in Table 2, the generation of the ucceor belief tate work with fully intantiated operator. Figure 3 how a pictorial view of the generation proce. B a 3 a 3 a 3 5 A C 5 a a 2 P(e P(e2 P(e2 B a 2 B a P(8=P(e2.P(e P(e22 Figure 3: A pictorial view of the generation proce. When a new belief tate i created, we alo compute it expected utility. Thi value i equal to the weighted um of the etimated utilitie of each outcome tate compriing the belief tate. It i ued for action election by the function ChooeBetOne( that imply elect the belief tate with the highet expected utility. B CE i advanced to thi new tate, and the operator that generated the belief tate become the next tep in the plan. Execution The main advantage of executing in the IR domain i that it provide the planner with additional information that can be ued to etimate how well the retrieval proce i going, reducing the uncertainty and the number of poible tate the planner mut conider during forward projection. It may alo allow u to terminate earlier if we are fortunate enough to dicover our actual tate wa much better than our projection. The main diadvantage of execution in thi domain i that we cannot recover reource uch a time once they are conumed, potentially leading to a wore (le optimal reult than if we had continued planning. In the wort cae, where reource are everely limited, foolihly executing tep without ufficient lookahead may reult in failure to find any olution becaue we no longer have the reource available to complete the tak. Our ytem upport three different execution trategie, the firt two of which clearly repreent the extreme of poible approache: Conervative (Deliberative - Alway plan until we are forced to execute, i.e., no more planning i poible Reactive - Alway execute a oon a an unexecuted plan tep i available Informed - Bae our execution deciion on the feature of our current belief tate and alternate candidate. We are currently exploring everal different verion of informed trategie. Table 3 how a imple one. ChooeExecutionOrExpanion(belief tate, candidate. If (( == 6 or (Var(Utilitie(!YXZYU return execute 2. Ele return expand / Table 3: Sample deciion for execution v. planning. Thi function chooe execution when it i the only choice available, or when the variance of utilitie in the current belief tate exceed a fixed threhold. The intuition here i that planning i probably not a very good option in belief tate with bimodal utility ditribution coniting of very high and very low utility outcome. If executing put u in a high utility tate, we may be done without additional planning; if it put u in a low utility tate, then now we know that we are probably better off puruing other option. Either way, we benefit. In addition to conidering the ditribution of utilitie in our current belief tate and the number of alternative we have, other trategie include conideration of the potential that execution ha to change our view of the world (i.e., the number of alternative execution introduce, and how cotly the next tep i to execute. After executing an operator, an update function record the plan tep a having been executed, remove candidate at branche of the earch tree above the executed node, and recompute our current belief tate. Replanning Currently, for implicity, we alway chooe to replan after executing a tep. However, we eventually intend to replace thi with a true deciion point, taking into conideration: How conitent our new belief tate i with repect to the original plan. Whether operator that were previouly not applicable have now become applicable. The relative cot of updating the exiting belief tate tree veru the cot of replanning from cratch.

6 / Termination There are three different termination condition that may occur during the planning and execution proce: a olution may be found, the proce may hit a pre-defined earch limit, or it may fail becaue there are no additional action available to take. We limit out dicuion to jut the ucce condition a both failure cae are traightforward. The function ProbabiliticallySatifie( (Tabel 4 ue the atifiability threhold to determine when the goal i conidered atified. The intuition behind thi function i that if we have a plan that i already very likely to produce a ucceful outcome, additional planning may not be very ueful. It i probably more productive to witch into execution mode to obtain feedback verifying or diproving the poibility that we are done. ProbabiliticallySatifie(belief tate, utility function, goal threhold < *, atifiability threhold where:.h\m / 5 / % "q& / #" Table 4: Probabilitic goal atifaction. A defined earlier, thi uer-pecified threhold i a value between zero and one that indicate how willing the uer i to rik a uboptimal olution and wated execution. A threhold of one indicate a uer who i not willing to tolerate any rik (a trong atifiability requirement; a mall non-zero value indicate a uer who i very rik tolerant (a weak atifiability requirement. It i worth pointing out that although thi threhold influence how uboptimal the reult may be, it doe not make any guarantee for optimality. Even in the rik-avere cae, we are till only looking for atificing olution. Implementation We have implemented thi approach a part of the INSPIRE (INtegrated Sytem for Planning and Information REtrieval architecture pictured in Figure 4. INSPIRE conit of three major component: the planner (IRplan, the execution manager (IRexecute, and the uer interface. The uer interface module i reponible for handling all the direct interaction with the uer. It alo take care of tranlating the uer input into an abtract repreentation that the planner can ue. Upon receiving a new requet in the form of a text query, the interface tore it in the data-repoitory, and generate a new problem tatement containing the identifier by which the query wa indexed, the query feature and other tate information, and a goal tatement. Thi in turn initiate the planning and execution algorithm contained within the IRplan module. When an execution deciion i made, the planner end an execution requet to the IRexecute module. The text claifier domain model uer interface data repoitory IRplan execution lexical manager utilitie retrieval utilitie IRexecute... earch engine and document collection Figure 4: The INSPIRE Architecture. diplay utilitie query expanion utilitie execution manager retrieve any data it need from the data repoitory and proceed to carry out the requeted tak. After execution, it ave any new data to the repoitory, generate a reult ummary for the planner and return control to the proce. At preent, the diplay tool required to upport potproceing operator are not implemented (a indicated by the dotted box. The reult returned to the uer conit of the firt few line from the 0 mot-highly-ranked document. We alo do not provide mechanim that allow the uer to interrupt the ytem. Uer-control i returned only after the planning and execution proce i finihed or when a feedback requet i executed. Etimating Domain Parameter One important apect of our approach, which we have yet to addre here, i the origin of the parameter ued in the domain model. Currently, we ue tatic prior value for the probabilitie of operator outcome and etimate of metric effect (including quality etimate. However, the development of better model and the addition of the infratructure neceary to upport learning i the major focu of our preent reearch. We briefly outline two of the more intereting iue we are currently conidering below. Categorization of information need: Uer of information retrieval ytem have very divere information need ranging from requet for a pecific document, to exploratory earche (Belkin et al The ability to pre-claify incoming requet into a et of general goal categorie could help u generate problem tatement with more precie goal requirement. Quality Etimate: Reearch in information retrieval ha long focued on defining ueful meaure of document relevance (Rijbergen 979. However, for the purpoe of planning, etimating the quality of intermediate

7 product and reource uch a querie and ource are alo important. Similar to (Kekäläinen & Järvelin 998, we are currently evaluating method for predicting query quality both in tand-alone requet and in the context of incremental earch. Related Work The challenge preented by the IR tak touch upon many different area of planning reearch. In thi ection we decribe a repreentative ample of thi work. The area that i perhap mot relevant to the current work i the problem of planning for information gathering. It ha been the focu of coniderable attention within the planning community in recent year (e.g.. (Golden 998; Knoblock 996; Levy, Rajaraman, & Ordille 996; Kwok & Weld 996; Barih et al A in the general IR tak, the goal of information gathering i to obtain information that atifie a uer query, often in the face of incomplete knowledge and reource contraint. Unlike the current work however, many of thee ytem are modeled after databae paradigm: they are retricted to querying tructured ource, require a model of ource content, and aume that the information goal i perfectly repreented by the query. Information gathering ytem uch a Sage (Knoblock 995, and XII (Golden, Etzioni, & Weld 994 interleave execution to gather information, but adopt the imple policy of delaying execution for a long a poible (baed on (Ambro-Ingeron & Steel 988. PUCCINI (Golden 998 adopt a lightly more relaxed approach to execution. It allow execution to occur even when it i uncertain that the plan will be atified, a long a the outcome i verified afterward. In addition to information gathering ytem that interleave planning and execution to upport ening, other have conidered thi problem in a more general planning context. Stone and Veloo (Stone & Veloo 996 decribe an extenion to the Prodigy planner to upport uer-guided execution. They alo ugget other execution policie baed on abtraction hierarchie and learning from oberving the uer. Nourbakhh (Nourbakhh 997 develop a et of three type of termination condition under which the execution can tart. They are baed on abtraction, aumptive planning, and relative partial plan quality. The problem of planning for uncertain outcome ha been addreed by everal reearcher within the context of conditional planning (e.g. (Peot & Smith 992; Pryor & Collin 996; Weld, Anderon, & Smith 998 and probabilitic planning (e.g. (Draper, Hank, & Weld 994; Kuhmerick, Hank, & Weld 995; Blythe 998; Blum & Langford 998. However, unlike the current work thee ytem generally work with completely pecified goal and don t include execution upport. Typically the probabiliitic ytem declare ucce when a probability threhold value i exceeded. Recently, there ha been increaed interet in heuritic earch technique for planning. One example i the GPT planner (Bonet & Geffner 2000, which ue forward chaining earch heuritic, allow incomplete information, and ha a belief-pace repreentation imiliar to the work here. We addre thi challenge within the context of information proceing tak. Deciion-theoretic approache to planning (e.g. (Haddaway & Suwandi 994; Haddawy & Hank 998; Williamon & Hank 994; Boutilier, Dean, & Hank 999 are alo imilar to the current work in that they ue a utility function to evaluate alternative plan and make it eay to take into account reource contraint. Other relevant work from outide the planning community include Microoft Lumiere project (Horvitz et al. 998, which infer uer need within the context of a oftware help ytem. Microoft ha alo worked with developing utility-baed model for uer interface (Horvitz 999. Concluion In thi paper, we have defined a et of challenge preented by the general real-world information retrieval tak and decribed an integrated planning and execution ytem deigned to addre them. We have preented a repreention that model the variou ource of uncertainty in the domain, and ue the uncertainty in the tate to guide our deciion for planning, execution, and replanning. From a planning perpective, the IR domain and our approach provide an intereting framework in which to addre a variety of difficult planning problem including: interleaving execution and planning, planning with incompletely pecified goal, partial and incremental goal atifaction, and conidering the role of the uer in the planning/execution loop. The INSPIRE architecture aim at addreing thee iue that are of great importance in bringing automated planning tool to upport uer goal deciion making and goal achievement. From an IR perpective, the INSPIRE architecture preent an opportunity to provide uer with better upport in the information retrieval proce, a well a to gain greater inight into the value of variou IR techhnique. Our approach ha been implemented and thorough evaluation with real uer and tak i part of our ongoing and future work. We are in fact currently addreing quetionanwering tak to further focu the general IR tak. The INSPIRE architecture i the ubtrate for our work. Although the quetion of interleaving planning and execution and planning under uncertainty have been previouly addreed, our reearch aim at grounding thee quetion within the tak of information proceing. Within thi challenging domain that include uer, we enviion learning the value of the parameter of our approach. Reference Ambro-Ingeron, J., and Steel, S Integrating planning, execution, and monitoring. In Proceeding of the Seventh National Conference on Artificial Intelligence, Barih, G.; DiPaquo, D.; Knoblock, C. A.; and Minton, S A dataflow approach to agent-baed information management. In Proceeding of the 2000 International Conference on Artificial Intelligence (ICAI-2000,

8 Belkin, N. J.; Cool, C.; Stein, A.; and Thiel, U Cae, cript, and information-eeking trategie: On the deign of interactive information retrieval ytem. Expert Sytem and Application 9: Blum, A., and Langford, J Probabilitic planning in the graphplan framework. In AIPS98 Workhop on Planning a Combinatorial Search, 8 2. Blythe, J Planning Under Uncertainty in Dynamic Domain. Ph.D. Diertation, Carnegie Mellon Univerity. Bonet, B., and Geffner, H Planning with incomplete information a heuritic earch in belief pace. In Proceeding of the Fifth International Conference on Artificial Intelligence Planning and Scheduling. Boutilier, C.; Dean, T.; and Hank, S Deciion theoretic planning: Structural aumption and computational leverage. Journal of Artificial Intelligence Reearch : 94. Draper, D.; Hank, S.; and Weld, D Probabilitic planning with information gathering and contingent execution. In Proceeding of the Second International Conference on Artificial Intelligence Planning Sytem (AIPS-94, Golden, K.; Etzioni, O.; and Weld, D Omnipotence without omnicience: Efficient enor management for planning. In Proceeding of the 2th National Conference on Artificial Intelligence (AAAI-94, volume 2, AAAI Pre. Golden, K Leap before you look: Information gathering in the puccini planner. In Proceeding of the Fourth International Conference on Artificial Intelligence Planning Sytem (AIPS-98, Haddaway, P., and Suwandi, M Deciion-theoretic refinement planning uing inheritance abtraction. In Proceeding of the Second International Conference on Artificial Intelligence Planning Sytem. Haddawy, P., and Hank, S Utility model for goaldirected deciion-theoretic planner. Computational Intelligence 4(3: Horvitz, E.; Breee, J.; Heckerman, D.; Hovel, D.; and Rommele, K The lumiere project: Bayeian uer modeling for inferring the goal and need of oftware uer. In Proceeding of the 4th Conference on Uncertainty in Artificial Intelligence, Horvitz, E Principle of mixed-initiative uer interface. In Proceeding of CHI 99, ACM SIGCHI Conference on Human Factor in Computing Sytem, Kekäläinen, J., and Järvelin, K The impact of query tructure and query expanion on retrieval performance. In Proceeding of the 2t Annual International ACM SIGIR Conference on Reearch and Development in Information Retrieval, Melbourne, Autrailia: ACM. Knoblock, C. A Planning, executing, ening, and replanning for information gathering. In Proceeding of the Fourteenth International Joint Conference on Artificial Intelligence (IJCAI-95, Knoblock, C. A Building a planner for information gathering: A report from the trenche. In Artificial Intelligence Planning Sytem: Proceeding of the Third International Conference (AIPS-96, Kuhmerick, N.; Hank, S.; and Weld, D. S An algorithm for probabilitic planning. Artificial Intelligence 76: Kwok, C. T., and Weld, D Planning to gather information. In Proceeding of the 4th National Conference on Artificial Intelligence (AAAI-96, volume, AAAI Pre. Levy, A. Y.; Rajaraman, A.; and Ordille, J. J Querying heterogeneou information ource uing ource decription. In Proceeding of the Twenty-econd International Conference on Very Large Databae, Bombay, India: VLDB Endowment, Saratoga, Calif. Nourbakhh, I Interleaving Planning and Execution. Ph.D. Diertation, Stanford Univerity. Peot, M. A., and Smith, D. E Conditional nonlinear planning. In Proceeding of the Firt International Conference on Artificial Intelligence Planning Sytem, Pryor, L., and Collin, G Planning for contingencie: A deciion-baed approach. Journal of Artificial Intelligence Reearch 4: Rijbergen, C. V Information Retrieval, 2nd edition. Dept. of Computer Science, Univerity of Glagow. Stone, P., and Veloo, M. M Uer-guided interleaving of planning and execution. In Ghallab, M., and Milani, A., ed., New Direction in AI Planning. IOS Pre Weld, D. S.; Anderon, C. R.; and Smith, D. E Extending graphplan to handle uncertainty and ening action. In Proceeding of the Fifteenth National Conference on Artificial Intelligence, Williamon, M., and Hank, S Optimal planning with a goal-directed utility model. In Proceeding of the Second International Conference on Artificial Intelligence Planning Sytem (AIPS-94, 76 8.