Identifying best applicants in recruiting using data envelopment analysis

Transcription

1 Socio-Economic Planning Sciences 37 (2003) Identifying best applicants in recruiting using data envelopment analysis Sharon A. Johnson, Joe Zhu* Department of Management, Worcester Polytechnic Institute, Worcester, MA 01609, USA Abstract Selecting the most promising candidates to fill an open position can be a difficult task when there are many applicants. Each applicant achieves certain performance levels in various categories and the resulting information can be overwhelming. We demonstrate how data envelopment analysis (DEA) can be used as a fair screening and sorting tool to support the candidate selection and decision-making process. Each applicant is viewed as an entity with multiple achievements. Without any a priori preference or information on the multiple achievements, DEA identifies the non-dominated solutions, which, in our case, represent the best candidates. A DEA-aided recruiting process was developed that (1) determines the performance levels of the best candidates relative to other applicants; (2) evaluates the degree of excellence of best candidates performance; (3) forms consistent tradeoff information on multiple recruiting criteria among search committee members, and, then, (4) clusters the applicants. r 2003 Elsevier Science Ltd. All rights reserved. Keywords: Data envelopment analysis (DEA); Recruiting; Candidate evaluation 1. Introduction Data envelopment analysis (DEA) is a linear programming-based procedure that measures the relative efficiencies and inefficiencies of peer decision-making units (DMUs). These DMUs produce multiple outputs by consuming multiple inputs. Since development of the first DEA model [1], the DEA methodology has been widely applied in areas such as banking, energy, flexible manufacturing cells, highway maintenance, individual physician practice, and telecommunications [2]. These applications have demonstrated that DEA is an effective tool for evaluating and managing operational performance in a wide variety of settings. *Corresponding author. address: jzhu@wpi.edu (J. Zhu) /03/$ - see front matter r 2003 Elsevier Science Ltd. All rights reserved. doi: /s (02)

2 126 S.A. Johnson, J. Zhu / Socio-Economic Planning Sciences 37 (2003) In this article, we present a novel use of DEA in recruiting processes. Evaluating candidates in such a process can be long and time consuming because of the: (1) large number of applicants, (2) difficulty in reaching consistent tradeoffs across various recruiting criteria, (3) need to reconcile the preferences of search committee members regarding candidates performance levels, and (4) lack of information about relationships amongst various performance measures. For example, in seeking promising candidates for faculty position openings in the Department of Management at Worcester Polytechnic Institute (WPI), we received more than 60 applications in each search. Each applicant had, of course, achieved certain performance levels related to research, teaching, and service. For the individual committee members evaluating these applications, it was difficult to balance and reconcile the various performance levels achieved by each applicant. Different committee members likely also had different preferences regarding research, teaching and service. These embedded conditions make the recruiting process complex and, perhaps, not entirely fair. During this process, some important performance measures may not be considered at certain stages because, for example, there is information overload. This, in turn, may result in a failure to select the best candidates. As we demonstrate in this paper, DEA can solve many of the problems associated with candidate evaluation in the recruiting process. It can thus help alleviate some degree of difficulty associated with reaching the final candidate pool. In recent years, major changes in the recruiting process have occurred given institutions ability to use the world wide web to disseminate job openings widely, and thus recruit candidates regardless of location [3]. This ability has led to an increase in the flow of resumes, increasing even further the need for effective candidate screening. Some companies are using computer systems to assist in the screening process; for example, searching for specific phrases or text in a resume. More sophisticated systems use artificial intelligence to recognize qualifications written in different ways [4]. Managers or recruiters still usually assess a candidate s fit [3], although some systems will rank those candidates who satisfy mandatory requirements in terms of the number of preferred requirements that each fills [4]. The DEA approach we describe is a sophisticated analytic tool, which can automate the screening process by assisting in the evaluation of applicants performance across many potential dimensions. 2. The DEA-assisted candidate selection process 2.1. Candidate selection process Fig. 1 depicts the proposed DEA-aided candidate selection process. We describe its use with data collected for a faculty search at Worcester Polytechnic Institute (WPI). In doing so, we illustrate specific steps in the methodology using a simplified example drawn from this faculty search. To begin the process, resumes are received and information about candidates entered into a database. Increasingly, this process is becoming more automatic as candidates apply online [3]. The pre-qualification step in the recruiting process is used to screen out applicants who do not have the basic required qualifications. For example, if a position requires a Ph.D. degree, then applicants who do not have a Ph.D., or are not in a Ph.D. program and near completion, will not receive further consideration. In this pre-qualification process, the search committee can

3 S.A. Johnson, J. Zhu / Socio-Economic Planning Sciences 37 (2003) Received Resumes Education Database for Applicants Experience/ Accomplishments 1. Pre-qualification 2. Selection of Performance Measures DEA 1.Selection of inputs/ outputs 2. Stage 1 analysis (see Figure 2) 3. Stage 2 analysis (See Figure 2) Reduced Pool of Candidates Committee Input "Best" Candidates Fig. 1. The DEA-assisted candidate selection process. incorporate any candidate criteria it might identify as useful. For example, in our search, we required an applicant possess a background in both industrial engineering and management. The next stage of the process involves evaluating the performance levels of all pre-qualified applicants. In order to carry out this evaluation, performance measures must first be specified. Typically, candidates present information on experience, education, and accomplishments in their

4 128 S.A. Johnson, J. Zhu / Socio-Economic Planning Sciences 37 (2003) resume. In the context of a faculty search, the following major criteria are thus selected for evaluating candidates: (1) research, (2) teaching, (3) experience, and (4) recommendations. Within each criterion, different performance measures are developed. For example, the number of publications and the amount of research funding are indicators of research quality and achievement. The number of conference presentations is a measure of the level of academic activity. The number of working papers indicates potential research output. In the area of teaching, student teaching evaluations might be used as a measure of teaching quality and achievement. In the context of experience, one might look at the reputation of the institutions from which an applicant received degrees. Other measures might include the number of years of experience since the Ph.D., years in industry, and/or years as a university professor. Finally, recommendation letters might play a role in the initial evaluation of candidates. As measures, one might look for phrases that characterize the applicant as the best ever, or consider the reputation of the person providing the recommendation. Developing performance measures is often a straightforward task; however, scoring each candidate and balancing the multiple performance measures can be complex because of the large number of pre-qualified applicants and the multiple measures. Each member of a search committee may have his/her own preferences across the performance measures. For example, one person may emphasize research while another emphasizes teaching. Also, over the course of reconciling conflicting opinions, it is likely that some performance measures are overlooked. Consequently, the evaluation may be incomplete with respect to all selected performance measures. This may subsequently bias the selection of the best candidates. The complexities described above occur partly because tradeoffs amongst the various performance measures are unclear. Fortunately, DEA can assist in identifying applicants across several performance levels such as education received and years of experience. Thus, we need not struggle to balance preferences across multiple performance measures. Rather, we can let applicants performance levels compete with one another. The use of DEA makes this a fair competition since no a priori preferences across performance measure importances are imposed DEA process The DEA process we describe consists of four activities, as shown in Fig. 2. First, inputs and outputs are determined based on the chosen performance measures. Next, we recommend applying two different DEA analyses. One is based on the classic variable returns-to-scale model of Banker et al. [5], while the other is based on the constant returns-to-scale model of Charnes et al. [1]. Within each process, the DEA has two stages. In the first, non-dominated candidates are identified using classic DEA formulations. As a result of this analysis, the candidate pool can be substantially reduced; perhaps by 40 70% in our experience, depending on the number of applicants, the number of performance measures, and the DEA model used. While the size of the candidate pool is reduced, it generally remains large compared to the number of candidates who would be invited for an on-site interview. In the second stage, recent developments in DEA are used to further characterize the candidate pool in order to identify the best performers. In our case, after applying the second stage DEA models, approximately 15% of the original applicant pool was identified as achieving best performance levels.

5 S.A. Johnson, J. Zhu / Socio-Economic Planning Sciences 37 (2003) DEA Process 1 Process 2 DEA Model I DEA Model II Stage 1: Identifying Non-dominated Candidates Bestperformance Levels Bestperformance Levels Benchmarking Shares Contextdependent Performance Stage 2: Further Characterizing Best Performers - Categorize "Best" Candidates - Incorporate Preference Information Reduced Pool of Candidates Fig. 2. Identification of best candidates. The final activity involves combining results from the two DEA analyses. It is important to note that DEA allows value judgments (preferences across different performance measures) to be incorporated. Thus, for each committee member, a new DEA run with preferences may be applied in the final activity to further screen the best candidates. This action may very well increase the confidence of each search committee member in selecting his/her best candidates. After applying DEA, we obtain a reduced pool of best candidates, who must be screened for best fit by a search committee or manager. The DEA process can also generate tradeoff information on the various performance measures (through the use of dual solutions), which can aid the search committee in selecting the final candidates.

6 130 S.A. Johnson, J. Zhu / Socio-Economic Planning Sciences 37 (2003) Selection of DEA inputs and outputs The first step in the DEA approach is to identify those inputs and outputs associated with the peer DMUs being assessed. In our case, we were seeking to determine the efficiency with which candidates in a job search (the DMUs in our application) met the performance criteria relative to the pool of candidates. The DEA inputs and outputs were selected from the various performance measures defined by search committee members. In the linear programs used to carry out DEA, one may fix the outputs in the constraints and seek to minimize the inputs; or, one may fix the inputs in the constraints and seek to maximize the outputs. In searching for the best candidates for a tenure-track faculty position, we used years of experience since the Ph.D. and years in a Ph.D. program as DEA inputs. The rationale is that someone would be expected to achieve a higher level of performance with more years of experience. We used several output measures, including the number of publications, the number of working papers, and the number of conference presentations. The number of courses taught was also used as an output. Because DEA was being used as a screening tool, we employed a simple measure of teaching quality based on the absence or presence of teaching evaluations. We assigned scores of 3 and 6, respectively, to applicants who did not or did send in teaching evaluations. Also, we assigned scores from 1 to 6 for recommendation letters based on the number of letters received with the application. 1 A candidate with six recommendation letters might thus be identified as a best performer even with low teaching and research performance levels if, on average, other candidates did not submit as many letters. In this regard, the model is compensatory in the way it generates candidate evaluations from specific criteria. 4. Process 1: The classic variable returns-to-scale DEA model Once all DEA inputs and outputs were determined, DEA models were used to implicitly characterize relationships amongst the inputs and outputs. These relationships generated a frontier of the best candidates performance levels. In this section, we discuss application of the variable returns-to-scale DEA model [5], which we refer to as DEA Model I. After the first stage application, the most frequently benchmarked performance levels were used to further categorize the non-dominated candidates Stage 1: Identifying non-dominated candidates In DEA Model I, we solved a linear program for each candidate to determine whether they exhibited best performance relative to other candidates. The mathematical formulation is described in Appendix A. Essentially, a composite candidate was created [1]; each input (output) associated with this composite candidate was the weighted average of the inputs (outputs) of 1 In the current study, the quality of some of the inputs and outputs was not evaluated. For example, in addition to the number of publications, we might wish to consider the quality of publications by categorizing and treating them as category variables in the DEA models.

7 S.A. Johnson, J. Zhu / Socio-Economic Planning Sciences 37 (2003) A5 (9,7) 6 Level-1 Level-2 Level-3 A3 (7,6) Number of Publications expected increase in number of publications A6 (7,3) A4 (8,4) 2 A2 (4,2) 1 A1 (2,1) Years of Experience Fig. 3. DEA analysis results for six potential candidates. Two performance measures are used: (1) number of refereed publications, and (2) number of years of professional experience. The frontier A1A3A5 results from applying DEA Model I, while the dotted line represents the result from applying DEA Model II. The levels indicate the results achieved from stratification. individual candidates. The same weights were used for both inputs and outputs. The constraints in the linear program (LP) require that the output of the composite candidate be greater than or equal to that of the individual. Further, the inputs of the composite candidate are required to be less than or equal to the inputs of the individual candidate. Solving the LP for an individual candidate determines a set of weights as well as the candidate s DEA score. If the score equals 1, the candidate produces as much output for a given level of input as does the composite candidate. If the score is greater than 1, the candidate produces less output than does the composite candidate for a given level of input; the candidate s performance level is thus judged to be inefficient. We provide a graphic view (see Fig. 3) of the six hypothetical applicants performance levels, labeled as points A1 A6. For each point, the numbers of years of experience and publications are reported in parentheses. We assumed here that more years of research/teaching experience should lead to more publications. Solving DEA Model I generated a best-performance frontier of A1A3A5 (in Fig. 3). The DEA model assumes that any convex combination between two or more observed performance levels leads to a level that could be observed, although no real applicant might show such a level. For example, applying the normalized weights l A1 ¼ 3 5 and l A3 ¼ 2 5 to A1

8 132 S.A. Johnson, J. Zhu / Socio-Economic Planning Sciences 37 (2003) A5 (6, 7) 7 6 A3 (6, 6) Number of Publications A6 (3, 3) A4 (6, 3) 2 A2 (3, 2) 1 A1 (3, 1) Teaching Fig. 4. DEA analysis results for six potential candidates using criteria for teaching and research (number of publications). and A3 yields: " # " # " # " publication ¼ 3 1 þ 2 6 ¼ 3 # : experience By comparing the above performance level with A2, we say that A2 would be expected to have at least one more publication. The resulting DEA score for A2 is then expected performance level ¼ 3 current performance level 2 : This score signifies that A2 s performance level is dominated by the performance levels of A1 and A3, indicating potential performance enhancement for A2. There is a performance gap between the expected performance level and the level of A2. For A1, A3 and A5, the DEA score is unity because there is no gap between expected performance level and the current level ; that is, the performance levels of A1, A3 and A5 are not dominated and thus represent the best levels. Now, suppose the performance measure years of experience is replaced by teaching. As shown in Fig. 4, the candidate represented by point A5 has the best performance level. In this case, all other candidates are expected to reach the performance level of A5. For example, for A6,

9 S.A. Johnson, J. Zhu / Socio-Economic Planning Sciences 37 (2003) Table 1 DEA scores, weights and benchmarking scores a Performance level DEA score Weights D j A1 1 l A1 ¼ A2 1.4 l A1 ¼ 0:6; l A3 ¼ 0:4; A3 1 l A3 ¼ A4 1 l A4 ¼ 1 0 A5 1 l A5 ¼ 1 0 A6 2 l A3 ¼ 1 a Three performance measures are considered: (1) number of refereed publications (research), (2) teaching, and (3) number of years of professional experience. the resulting DEA score is 2 (see Table 1). The 2 represents both the proportional increase of research and teaching levels. In this case, a one unit increase in research is expected for A6 after the proportional increase. Finally, if we consider all three performance measures, i.e., research, teaching, and years of professional experience, we have the set of DEA results shown in Table 1. The best performance frontier now consists of A1, A3, A4, and A Stage 2: Determining a benchmarking share To further prioritize the performance levels of the best candidates, we turn to the optimized DEA weights in DEA Model I. Recall that in Fig. 3, the performance of A2 is compared to A1 and A3 because a set of normalized non-zero weights is associated with A1 and A3. In other words, A1 and A3 may be viewed as benchmarks for A2. We define the following benchmarking score to measure the frequency of a specific best candidate s performance level: PdAN D j ¼ ld j ; I N where j represents a best candidate s performance level, and d represents a dominated performance level. N is the set of dominated performance levels and I N is the number of dominated performance levels in set N: In our example, with the three performance measures (research and teaching given the years of experience) presented in Table 1, we have four best performance levels and two dominated performance levels, A2 and A6. Thus, D 1 ¼ 0:6 2 ¼ 0:3 for A1, D 3 ¼ 0:4þ1 2 ¼ 0:7 for A3, and D j ¼ 0 for the other best performance levels, A4 and A5. In DEA Model I, the weights are normalized so that P j D j ¼ 1: Consequently, D j yields benchmarking shares among the best candidates performance levels. A larger value of D j indicates that there is a higher concentration of performance levels around this particular best candidate s performance level. To further classify the candidates, we divide them into two classes: those with a large benchmarking share, and those with a small benchmarking share. In defining large benchmarking share, one might require, for example, that the share be at least be 60%. The best candidates with a small benchmarking share have levels that are outliers relative to other candidates; such information allows the committee to further investigate these candidates.

10 134 S.A. Johnson, J. Zhu / Socio-Economic Planning Sciences 37 (2003) For example, for an advertised tenure-track position at the assistant professor level, perhaps one of the best candidates already has many years of academic experience, which is not consistent with the advertised appointment level. Benchmarking share helps to identify such unusual candidates. 5. Process 2: The classic constant returns-to-scale DEA model To further analyze applicants performance levels, a second set of DEA models can be applied to applicant data. In this section, we discuss the application of the constant returns-to-scale DEA model [1], which we refer to as DEA Model II. Following Stage 1 (see Fig. 2), the degree of excellence of the first-level applicants performance levels is evaluated using a context-dependent DEA [6] Stage 1: Identifying non-dominated candidates In DEA Model I, we did not assume a constant rate of number of publications per year. However, if one assumes such a rate, one obtains a new DEA model (which we call DEA Model II) where the normalization on weights (l A1 þ l A3 ¼ 1) is dropped. (see Appendix A for the mathematical formulation.) The resulting frontier is ray OA3 (represented by the dotted line in Fig. 3). In this case, A3 is the best candidate because A3 has the highest rate of publications per year. In fact, if the normalization on weights is dropped, then the scale efficiency of the performance levels is also considered. From a screening point of view, the use of DEA Model II results in fewer non-dominated candidates Stage 2: Evaluating the degree of excellence of best performance levels To further categorize the candidates with best performance levels, the next stage in the second DEA process involves applying two more recent DEA approaches: stratification DEA [6] followed by context-dependent DEA. Stratification DEA is illustrated using the two-performance level example in Fig. 3, research given years of experience. The mathematical formulation is presented in Appendix A. Using DEA Model I, we identify A1, A3 and A5 as candidates with best performance, or candidates in level 1 frontier. Next, we exclude these three best performance levels and apply the same DEA approach to the remaining three applicants performance levels, identifying a new best candidate frontier a level-2 frontier consisting of the ray A2A4. Proceeding in this manner, we divide the set of applicant performance levels into different subgroups or frontiers on the basis of DEA best-performance levels. Such action enables us to further study the degree of excellence of best candidates performance levels. In particular, we measure the best levels against the level-2 frontier using a context-dependent DEA model. Because DEA Model II yields a score of unity for candidates located on the frontier, these best levels are viewed as equal. Applying context-dependent DEA divides the best performers into two groups, using the second-level performance levels as

11 S.A. Johnson, J. Zhu / Socio-Economic Planning Sciences 37 (2003) potential competitors. Rather than use all applicants to develop the composite applicant, only those in the level-2 frontier are considered. An LP similar to either DEA Model I or II is solved for each best candidate. The mathematical formulation of the model is described in Appendix A. The smaller the contextdependent DEA score, the better the performance of a candidate. Solving the LP for an individual candidate yields a context-dependent DEA score. After solving such a model for each candidate in the level-1 frontier, these best candidates are divided into two classes based on their contextdependent DEA score; those with a score less than the average are grouped together as excellent performers while the remaining best candidates constitute the second group. 6. Categorizing the best performance levels in an effort to select final candidates As previously noted, applying DEA models I and II yields a set of candidates with best performance levels. Usually, DEA Model I generates more best candidates in terms of achievement than does Model II. In our application to data from a faculty search at WPI, less than 60% (28%) of the applicants performance levels were on the frontier of DEA Model I (DEA Model II). All applicants on the DEA Model II frontier were on Model I frontier. On the basis of benchmarking share and context-dependent performance, we were able to group the candidates performance levels into four categories, as shown in Fig. 5. Candidates who indicate a large benchmarking share and efficient performance relative to frontier level-2 are given first priority. Second priority is given to candidates in Categories II and III. Candidates who are on level-1 frontier in DEA Model I but not in DEA Model II will have an inefficient contextdependent score, and thus fall into Category III or IV. Applicants whose performance levels are in Category IV may be reserved for future consideration. Using our faculty search data, 13 candidates fell into Categories I, II and III; two candidates performance levels are in Category I. In carrying out the faculty search at WPI, we performed a traditional candidate selection process where each member of a faculty committee reviews the r!esum!es of the applicants and rates them according to listed criteria. The committee then gathers to examine all applicants profiles and develop a list of semifinalists. Candidates on the semifinalist list are often invited for campus interviews after detailed reference checks. We compared the results obtained from this traditional search method with that obtained using our DEA approach. All candidates identified in the traditional approach were included in Categories I, II, and III using the DEA process. The candidates identified in Category I were in the final list developed using the traditional approach. 7. Incorporating preference information In our proposed DEA-based methodology, we considered no preference information across performance. However, a search committee as a whole, or each member of the committee individually, may apply a DEA process in which preferences across the performance measures are added. For example, assume that one committee member emphasizes research over teaching and weights journal publications more heavily than conference presentations. Such preferences can

12 136 S.A. Johnson, J. Zhu / Socio-Economic Planning Sciences 37 (2003) Benchmarking Share Category III Large Benchmarking Share Category I Large Benchmarking Share Excellent Contextdependent Performance Category IV Category II Excellent Contextdependent Performance Context-dependent Performance Fig. 5. Categories for candidate performance levels. First priority is given to the candidates in Category I, i.e., those who have (a) a large benchmarking share, and (b) excellent context-dependent performance when compared to performance levels on the second-level best-performance frontier. Second priority is given to candidates who fall in Category II or III. then be incorporated into DEA Models I and II via assurance region [7] or DEA/preference structure models [8], among others. The above preferences on performance levels may be expressed as 1p w publication w teaching p3 and 2p w publication w teaching p4; where w represents a tradeoff weight. Such constraints on the tradeoff weights can be introduced into the dual forms of the DEA models. Clearly, this information changes the structure of the best-performance frontier. By applying such preference information to the faculty search data at WPI, only about 13% (rather than 28%) of the applicants indicated excellent performance using DEA Model II. If each committee member develops his/her own preference information, each may have a different set of finalists. Such information can then be used to generate new categories in addition to the four listed in Fig. 5.

13 S.A. Johnson, J. Zhu / Socio-Economic Planning Sciences 37 (2003) Conclusions In today s business world, hiring the best employee for a specific position can be a daunting task. The current article presents a DEA-assisted candidate selection process applied to an academic job opening scenario. The suggested use of DEA can be applied to other recruiting and hiring efforts as well. To ensure effectiveness, traditional recruiting processes are time consuming. Our proposed DEA-assisted process can potentially reduce the cost, and streamline the process of, reviewing thousands of resumes and conducting hundreds of interviews. This, in turn, allows recruiters and managers to focus on a smaller number of promising candidates who are better suited to specified job requirements. A structure for the decision-making process is created by first identifying performance measures. The methodology automates the evaluation of candidates given this structure, thereby ensuring consistency across candidates and performance measures. While the approach described in this paper is based on quantitative data, qualitative performance measures also exist in recruiting. Examples include student satisfaction with teaching and evaluations of research potential. Thus, the DEA methodology could be expanded to include both quantitative and qualitative measures. Also, the process could be further automated by embedding it within a decision support system. Such a system could support users in selecting performance measures and assigning preferences, as well as in generating graphical output for candidate comparisons. Acknowledgements The authors are grateful to the Editor-in-Chief, Dr. Barnett R. Parker, and two anonymous reviewers for their helpful suggestions on earlier drafts of this paper. Appendix A This section provides the mathematical formulations of the DEA models we employed to select best candidates in a recruiting effort. In our formulations, we use a vector notion for inputs and outputs where decision-making unit DMU j (j ¼ 1; 2; y; n) produces a vector of outputs y j ¼ ðy 1j ; yy sj Þ by using a vector of inputs x j ¼ðx 1j ; y; x mj Þ: We define J 1 ¼fDMU j ; j ¼ 1; y; ng as the set of all n DMUs (representing n job applicants and their performance levels). A.1. DEA Model I For each candidate k; we solve the following linear program: f ðkþ ¼ max lj ;fðkþ fðkþ P s:t: jafðj 1 Þ l jy j XfðkÞy k P jafðj 1 Þ l jx j px k P jafðj 1 Þ l j ¼ 1 l j X0; jafðj 1 Þ;

14 138 S.A. Johnson, J. Zhu / Socio-Economic Planning Sciences 37 (2003) where x k and y k represent input and output performance levels, respectively, for candidate k; and jafðj 1 Þ indicates DMU j AJ 1 ; i.e., FðÞ represents the correspondence from a DMU set to the corresponding (subscript) index set. This DEA model was first formulated by Banker et al. [5]. The best performance levels can be expressed as E 1 ¼fDMU k AJ 1 f ðkþ ¼1g: A.2. DEA Model II If we drop the restriction of P jafðj 1 Þ l j ¼ 1 in DEA Model I, we then have DEA Model II [1]. A.3. Stratification DEA model We iteratively define J lþ1 ¼ J 1 E l where E l ¼fDMU k AJ 1 f ðl; kþ ¼1g; and f ðl; kþ is the optimal value to the following linear programming problem: f ðl; kþ ¼ max lj ;fðl;kþ fðl; kþ; P s:t: jafðj l Þ l jy j Xfðl; kþy k P jafðj l Þ l jy j px k l j p0; jafðj l Þ: When l ¼ 1; this model becomes DEA Model II and consists of all the (radially) best-performance DMUs. The DMUs in set E 1 define the first-level best-practice frontier. When l ¼ 2; the model yields the second-level best-practice frontier resulting from the exclusion of the first-level DMUs. And so on. In this manner, we identify several levels of best-practice frontiers. Identification of these frontiers is accomplished by the following algorithm: * Step 1: Set l ¼ 1: Evaluate the entire set of DMUs, J 1 ; by DEA Model II to obtain the first-level efficient DMUs, set E 1 (the first-level best-practice frontier). * Step 2: Exclude the frontier DMUs from future DEA runs. J lþ1 ¼ J l E l : (If J lþ1 ¼ +; then stop.) * Step 3: Evaluate the remaining DMUs, J lþ1 ; by the CCR DEA model [1] to obtain a new set of frontier DMUs E lþ1 (the new best-practice frontier). * Step 4: Let l ¼ l þ 1: Go to step 2. * Stop rule: J lþ1 ¼ +; the algorithm stops. The above procedure can also be applied to DEA Model I. A.4. Context-dependent DEA model The DEA stratification model partitions the set of DMUs into different subgroups (levels) characterized by E l ðl ¼ 1; y; LÞ: Now, consider a specific DMU q ¼ðx q ; y q Þ from the first-level E 1 : Context-dependent performance versus the performance levels in the second-level (E 2 ) can be

15 S.A. Johnson, J. Zhu / Socio-Economic Planning Sciences 37 (2003) measured via the following: O q ¼ max l j ;O q O q P s:t: jafðe 2 Þ l jy j XO q y q P jafðe 2 Þ l jx j px q l j X0; jafðe 2 Þ: Note that O qp1: The above model can also be developed on the basis of DEA Model I. References [1] Charnes A, Cooper WW, Rhodes E. Measuring the efficiency of decision making units. European Journal of Operational Research 1978;2: [2] Seiford LM. Data envelopment analysis: the evolution of the state of the art ( ). Journal of Productivity Analysis 1996;7: [3] Kay AS. Recruiters embrace the internet. Information Week 2000;778: [4] Theaker M. Entering the era of electronic CVs. People Management 1995;1(16):34. [5] Banker RD, Charnes A, Cooper WW. Some models for estimating technical and scale efficiencies in data envelopment analysis. Management Science 1984;30: [6] Seiford LM, Zhu J. Profitability and marketability of the best 55 US commercial banks. Management Science 1999;45: [7] Thompson RG, Langemeier LN, Lee C, Lee E, Thrall RM. The role of multiplier bounds in efficiency analysis with application to Kansas farming. Journal of Econometrics 1990;46: [8] Zhu J. Data envelopment analysis with preference structure. Journal of the Operational Research Society 1996;47: