This article appeared in a journal published by Elsevier. The attached copy is furnished to the author for internal non-commercial research and

Transcription

1 This article appeared in a journal published by Elsevier. The attached copy is furnished to the author for internal non-commercial research and education use, including for instruction at the authors institution and sharing with colleagues. Other uses, including reproduction and distribution, or selling or licensing copies, or posting to personal, institutional or third party websites are prohibited. In most cases authors are permitted to post their version of the article (e.g. in Word or Tex form) to their personal website or institutional repository. Authors requiring further information regarding Elsevier s archiving and manuscript policies are encouraged to visit:

2 Omega 39 (20) Contents lists available at ScienceDirect Omega journal homepage: Knowledge dissemination in operations management: Published perceptions versus academic reality Jack R. Meredith, Michelle D. Steward n, Bruce R. Lewis Wake Forest University, Schools of Business, P.O. Box 7285, Winston-Salem, NC 2709, USA article info Article history: Received 7 January 200 Accepted October 200 Processed by Associate Editor Teo Available online 6 October 200 Keywords: Operations management Journal ranking Journal perception OM discipline abstract The channels for knowledge generation and dissemination in the business disciplines are many: presenting research at conferences, writing books, distributing working papers, offering insights in society newsletters, giving invited talks, publishing studies in academic journals, and many other venues, including even blogs and perhaps Facebook s. But the most important venue is probably published research in top-level academic journals. In the discipline of Operations, many studies and lists have been published that attempt to determine which of these journals are supposedly the top according to either citation analyses, the opinion of recognized experts, author affiliations, bibliometric studies, and other approaches. These lists may then, in turn, be used in different degrees to evaluate research. However, what really counts is what the academic institutions actually use for guidance in evaluating faculty research. Based on a new source of ranking data from AACSB-accredited schools, we compare published journal-ranking studies against that of academe to determine the degree to which the studies reflect academic reality. We present rankings of OM journals based on this new source of data and on an aggregate of the stream of published studies, and evaluate their consistency. & 200 Elsevier Ltd. All rights reserved.. Introduction The channels available for disseminating academic knowledge are extensive, ranging from informal blogs these days to formal presentations to published books, monographs, and articles in journals. In the field of business, the most important, arguably, are published articles in top journals (rather than grants, as might be the case in medicine or engineering) since these are frequently the most important basis for promotion and tenure (P&T) decisions. But beyond P&T and annual evaluations, such top publications are also often the basis for: research awards by universities, scholarly societies, governmental academies, and journal publishers; grants from federal, state, and private agencies such as the National Science Foundation; nominations for high-profile chairs, professorships, research grants, and fellowships; offers for joining, or perhaps taking joint professorships at prestigious universities such as Yale, Harvard, MIT, and such others; candidacy for high-visibility governmental positions such as advisory positions to the President or Governor, secretarial n Corresponding author. Tel.: address: stewarmd@wfu.edu (M.D. Steward). positions with the state or federal organizations such as the Treasury, membership on the Federal Reserve or Council of Economic Advisors, etc. and and even, at least partially, the requisite, ubiquitous business school/program rankings. Clearly, the role of top publications in each of these sets of decisions will be different, but their influence is often substantial. For example, annual evaluations may include aspects of teaching, service to the department (and school, university, discipline, and perhaps even local community), research in-progress, and publications, while P&T may be based much more heavily on publications. Yet, universities are aware of the legal complications that can arise when a faculty member receives outstanding annual evaluations only to be turned down for P&T some years later. Hence, for schools that wish to place an emphasis on top journal publications for P&T, the role of such publications must be heavily included in the annual evaluations as well [,2]. It is also clear that there are more stakeholders in top journal publications than just the university. Attaining a reputation as the national expert in a particular field or topic can lead to such accolades as being appointed Assistant Secretary of the Treasury, Economic Czar of the State, or Chair of the Federal Bankruptcy Committee for the XYZ Corporation. Such eminent appointments are a great boon to a university, enhancing its reputation, bringing in donations and endowments, increasing its student applications, /$ - see front matter & 200 Elsevier Ltd. All rights reserved. doi:0.06/j.omega

3 436 J.R. Meredith et al. / Omega 39 (20) and many other benefits. Hence, the university also has an interest in their faculty publishing in highly recognized journals, and may thus emphasize particular journals or fields over others. Understandably then, everyone from University Presidents to Deans to Assistant Professors are interested in knowing which journals the university considers to be the top journals, and especially the top journals in each Professor s own field or discipline. To determine these top journals, and the rankings of other journals in a field, numerous authors have used various approaches such as citation impact scores, surveys of recognized scholars, bibliometric analyses, author affiliation indexes, and other techniques. These studies commonly identify the same general set of journals, but their ranking may differ considerably. In the discipline of interest here operations management Holsapple and Lee-Post [3] detail many of these approaches and identify examples of their use in operations management. However, with such a range of rankings derived by a variety of methods, it is not clear whose ranking to use, or even which method to rely upon. All of the methods have some justification, but also some weaknesses and limitations, again well described in Holsapple and Lee-Post [3] and Lewis [4]. For that matter, it is not only the methods that confound the studies but a range of other factors as well such as the geographic region (e.g., US versus Europe), the time period considered (e.g., 4 years versus 25 years), the date of publication, the selection of scholars, and the set of journals under consideration. In this study, we analyze all the ranking studies of operations management journals since 990, and then compare them to a new source of data official in-house journal lists of AACSB-accredited business schools used for helping evaluate faculty publications. This is a source of data that has not been used previously in OM journal ranking studies. (Although we asked schools for journal lists that are used for evaluating publications, we cannot state how, or even if, they were actually used. Our research here informs those evaluating research, but does not report any criteria for that evaluation, such as three top-level journal articles or six solid publications. ) Certainly not all schools formally and explicitly use in-house lists to evaluate research. However, schools often do create formal in-house lists for evaluation. Van Fleet et al. [5, p. 839] suggest that, These rankings are designed to reduce difficulties in evaluating quality and to help faculty members identify target journals. Additionally, Vokurka [6, p. 345] argues that, The rankings of journals are important to academics because promotion and tenure decisions are based to a large extent on publication achievements. These decisions are based primarily on the journals in which research is published. Our focus is on identifying the most credible ranking studies in the sense of conforming to the reality of academic guidelines provided by schools that have journal rating lists and then comment on these studies and their various approaches. 2. Background and data The literature on journal evaluation in operations management has a long history, as well as being diverse. We include here a discussion of these studies, as well as a description of the AACSB survey of journal lists used as a guide for making academic promotion, tenure, and salary decisions. 2.. The journal ranking studies Published studies ranking OM journals over two decades from 990 to 2009 were selected. Table presents the 2 articles, the nature of the data that were used in each, and the specific table within each study from which the ranking data were extracted. Seven of the studies compiled journal rankings based on perception scores derived from survey data, three based their analysis on citation data, and two utilized data from other sources author affiliation index and behavior-based publication counts. For each study the final, overall ranking of quality was used. For cases where the ranking article ranked other dimensions, such as journal relevance, etc., we selected the data corresponding to the article s overall measure of quality. Table presents the specific table within each paper that the data was extracted from and a description of the nature of that data. As can be seen, seven studies were published in the Journal of Operations, three in Omega, and one each in Interfaces and Manufacturing & Service Operations. Eleven of the studies were published in the 4 year span between 996 and 2009, for about one such article per year, showing the increased interest in publication venues since the mid-990s. Two of the senior authors published another such study, the eight other studies being published by different author teams The AACSB-accredited school survey In order to capture the reality of the stature of OM journals, we collected the official lists used to help assess faculty research output at AACSB-accredited schools [4]. The AACSB was selected because it is recognized as the major accreditation entity for business schools worldwide. As Van Fleet et al. [5, p. 340] note: A list provides an explicit measure of how a department values research outlets. Moreover, such lists reflect the current state-ofthe-standings among competing journals in academic practice. To our knowledge, this data source has not been previously used in ranking OM journals. An was sent to AACSB-accredited universities requesting a copy of their official journal list, if such was used, for evaluating faculty publications at their school. The request was initially sent in November 2006, followed by two reminders in each of the two following months. Of the 545 institutions receiving the request, 206 responded, representing a 38% response rate. Table 2 offers general demographics of both the responding schools and the entire population of AACSB-accredited schools. To determine the representativeness of the set of respondents, the demographics of the responding schools were statistically compared to those of the entire AACSB population (see Table 2). For the categorical measures (i.e., affiliation, geographic region, degree level offered, and mission priority), one-sample chi-square tests were run, but only one test (public/private affiliation) was even marginally significant at the 0.05 level. One-sample t-tests were utilized for the continuous variables (essentially, different versions of school size), but no significant differences were found at the 0.05 level. Our conclusion from these tests was that the respondent set was representative of the entire AACSB population. While our focus is on OM journal rankings, not on the type of evaluative mechanisms that different schools use, we did find it of interest to compare responding schools that have internal lists versus the entire AACSB-accredited school population. Using the same approach of one-sample tests to compare the same demographic variables as above, we found that as compared to the population of AACSB-accredited schools, the schools that have internal lists have a statistically significantly larger faculty (98 versus 76 on average) and undergraduate enrollment, higher research focus as their mission priority (9% versus 2%), and accordingly, fewer schools that are private (9% versus 32%). These differences, and some others, are presented in Table 3. Of the responding schools, 83 (40%) provided their formal target journal lists. This is a fairly large proportion of schools that have

4 J.R. Meredith et al. / Omega 39 (20) Table The 2 published journal ranking articles. Year First author Journal Article title Data type Table Column 99 Barman Journal of Operations 996 Goh Omega International Journal of Science 996 Vokurka Journal of Operations 997 Goh Journal of Operations 999 Soteriou Journal of Operations 2000 Donohue Omega International Journal of Science 200 Barman Journal of Operations 2005 Gorman Manufacturing & Service Operations An empirical assessment of the perceived relevance and quality of POM-related journals by academicians An empirical assessment of influences on POM research Perception scores 2 Mean quality ratings Citation scores 4 Normalized total citation based The relative importance of journals used in operations management research a citation analysis Citation rankings 8 Overall ranking citations Evaluating and classifying POM journals Citation rankings 2 Overall rank age-adjusted across base years Assessing production and operations Perception scores 4 Mean quality ratings management related journals: the European perspective A multi-method evaluation of journals in the decisions and management sciences by US academics Perceived relevance and quality of POM journals: a decade later OM forum evaluating operations management-related journals via the author affiliation index 2005 Olson Interfaces Top 25-school professors rate journals in operations management and related fields 2007 Theoharakis Journal of Operations Insights into factors affecting POM journal evaluation 2007 Zsidisin Journal of Operations 2009 Holsapple Omega International Journal of Science Criteria development and assessment of purchasing and supply management journals Behavior-based analysis of knowledge dissemination channels in operations management Perception scores 3 Perceived quality rating (POM interest area) Perception scores 4 Mean quality ratings Author affiliation index 2 Author affiliation index Perception scores 2 Mean quality rating from top-25 business school professors Perception scores 4 Mean journal quality rating worldwide Perception scores 5 Mean rating based on: articles published in this journal regularly meet my journal evaluation criteria Behavior: publication 4 Intensity score counts Table 2 Demographics of AACSB schools: respondents vs. the population. Demographic characteristic Respondents Population One-sample test n Percent n Percent Statistic p-value Affiliation Private w 2 ¼3.86 p¼0.050 Public Geographic region North America w 2 ¼3.09 p¼0.23 Europe Other Degree level offered Undergraduate only w 2 ¼4.73 p¼0.094 Graduate only Both Mission priority Teaching w 2 ¼0.6 p¼0.894 Research Teaching and research equal Teaching, research and service equal Mean Std. dev Mean Std. dev Statistic p-value Size Full time equivalent faculty t¼.33 p¼0.84 Undergrad enrolment full time t¼0.06 p¼0.949 Graduate enrolment full time t¼0.58 p¼0.566 Undergraduate degrees conferred t¼0.29 p¼0.776 Graduate degrees conferred t¼.86 p¼0.064 Notes: The n s for the respondents and for the population do not add to 206 and 545, respectively, due to the exclusion of missing data. p-valuesr0.05 are in bold.

5 438 J.R. Meredith et al. / Omega 39 (20) Table 3 Demographics of respondent schools with an internal list vs. the population. Demographic characteristic Respondent schools with an internal list Population One-sample test n Percent n Percent Statistic p-value Affiliation Private w 2 ¼6.00 p¼0.04 Public Geographic region North America w 2 ¼7.45 p¼0.042 Europe Other Degree level offered Undergraduate only w 2 ¼6.7 p¼0.035 Graduate only Both Mission priority Teaching w 2 ¼.9 p¼0.008 Research Teaching and research equal Teaching, research and service equal Mean Std. dev. Mean Std. dev. Statistic p-value Size Full time equivalent faculty t¼4.0 po Undergrad enrolment full time t¼3.84 po Graduate enrolment full time t¼.47 p¼0.45 Undergraduate degrees conferred t¼4.2 po Graduate degrees conferred t¼.88 p¼0.064 Notes: The n s for the respondent schools with an internal list and for the population do not add to 83 and 545, respectively, due to the exclusion of missing data. p-valuesr0.05 are in bold. Table 4 Demographics of respondent schools with an OM list vs. all schools with an internal list. Demographic characteristic Respondent schools with an OM List Respondent schools with an internal list One-sample test n Percent n Percent Statistic p-value Affiliation Private w 2 o0.0 p¼0.992 Public Geographic region North America w 2 ¼0.03 p¼0.987 Europe Other Degree level offered Undergraduate only w 2 ¼.2 p¼0.27 Graduate only Both Mission priority Teaching w 2 ¼.03 p¼0.795 Research Teaching and research equal Teaching, research and service equal Mean Std. dev. Mean Std. dev. Statistic p-value Size Full time equivalent faculty t¼0.0 p¼0.989 Undergrad enrolment full time t¼.43 p¼0.64 Graduate enrolment full time t¼3.29 p¼0.002 Undergraduate degrees conferred t¼0.37 p¼0.74 Graduate degrees conferred t¼0.55 p¼0.583 Notes: The n s for the respondent schools with an OM list and all respondent schools with an internal list do not add to 37 and 83, respectively, due to the exclusion of missing data. p-valuesr0.5 are in bold. their own in-house lists. By comparison, a survey of departments [5] found that only 4% had internal journal lists and a survey of Accounting departments [7] found that only 3% had internal journal lists. The remainder of the respondents included 89 schools that indicated they did not have internally generated lists, 2 that stated they used external lists, such as the Financial

6 J.R. Meredith et al. / Omega 39 (20) Times, and 22 that employed Cabell s Directory of Publishing Opportunities. As noted earlier, these lists reflect how journals may influence university decisions on annual faculty evaluations, tenure and promotion decisions, research awards, and other research-oriented activities. Since journals are generally ranked, or at least categorized, at individual schools by their perceived quality [5], the metrics derived from these lists serve as a reasonable depiction of journal standing from an operational standpoint. Of the 83 school lists submitted, 37 classified journals in graded tiers and specifically categorized OM journals in groups labeled as: operations, operations management, production and operations management, supply chain management, and logistics. These school lists were the ones used in our calculations. Table 4 offers demographic information on these 37 schools, with a comparison to schools that did not specifically categorize OM journals. There are no significant differences between the two groups. 3. Results We next describe our methodology and assessment of statistical reliability. We follow that with a description of the results of our analysis of published journal-ranking studies, an analysis of our AACSB journal ranking data, and a comparison of the two. 3.. Journal ranking studies To assess the psychometric soundness of the published ranking studies, reliability analysis was performed. Reliability is concerned with the consistency of a measure in different contexts and over successive trials [8]. Various methods can be used to evaluate reliability. At some level, each method correlates scores from one source with scores from another source. Higher correlations demonstrate more consistency, or reliability, across the sources. An assessment of the reliability of the rankings of the OM journal studies was conducted by correlating the ratings of all possible pairings of the 2 studies listed in Table. The nonparametric Spearman Rho correlation coefficient was used due to the ordinal nature of the data and the small number of journals that are common between studies. The results of this reliability analysis are reported in Table 5 [3,6,9, 7 25]. The journal articles are arranged by the data source of each article (perception, citation, other), and shaded regions highlight the correlations within each data source. Of the 66 Spearman correlation coefficients shown in Table 5,4 are statistically significant (in bold) at the 0.05 level with the majority of these significant beyond the level. The magnitude of the statistically significant correlation coefficients ranged from to Pairs of articles within each data source type are significantly correlated with one another with rare exception, such as Zsidisin et al. [9], which is not significantly correlated with any other study. A possible reason for this is that Zsidisin et al. examines purchasing and supply chain management journals specifically, rather than OM (or POM) journals as the other studies rank. It is also notable that the perception studies tend to be highly correlated with all the other studies OM journal categories Our interest in validating the published OM journal ranking stream using the new data source of in-house journal lists actually used for evaluating faculty research requires that we identify which journals are specifically dedicated to OM. For our candidate list of journals that disseminate OM knowledge, we considered the set of journals that were included in both the published studies on the dissemination of OM knowledge and the school lists of journals that are recognized as contributing to OM knowledge. To be conservative at this stage, we only required that a journal be mentioned both in at least one published study and on at least one school s list. As seen in Table 6, this gave us a set of 7 journals; 7 of the 7 journals appeared on only one published study and only one school list. The journals are listed in order of the number of published studies they appeared on, since the published studies preceded the more recent AACSB school list. As can be seen, there is a close, but not perfect, correlation between the two lists. (The actual correlations are given later in the paper for the OM-dedicated journals.) Next, we followed Holsapple and Lee-Post s [3] approach to separating OM-dedicated journals from Interdisciplinary journals and reference-discipline ( Reference ) journals. The contrast between OM-dedicated and interdisciplinary journals is relatively straight-forward: whether virtually all the papers published in the journal are dedicated to OM topics or whether they range across various disciplines (either business or even broader) with OM being only a portion of the subject matter. The contrast between OMdedicated and reference journals is a bit more complex since the reference discipline may be either a topic area (e.g., Economics, Engineering, Information Systems) or a methodology area (operations research, statistics). It was concluded that if the papers in a journal were related to one of the reference discipline topic areas rather than OM, then it was a reference journal. For the more complicated situation of a journal with methodological papers but related to an OM topic, we classified that as OM-dedicated if the focus of the papers was on the OM topic but Reference Discipline if the focus was on methodological issues (e.g., operations research methodology, statistical methodology). These three categories (OM-dedicated, Reference, Interdisciplinary) are listed in the right column of Table 6; most of them are obvious but we describe our logic regarding some of the less obvious journals. We relied on five potential sources of evidence to help categorize the journals:. The specific Aims and Scope stated by the journal. 2. The type of society (e.g., engineering, economics, operations research, production research) if the journal is sponsored by such an institution. 3. The variety of membership groups within the society, if society sponsored. 4. The diversity of articles actually published in the journal. 5. Holsapple and Lee-Post s [3] categorization, if the journal appeared on their list. Since we refer to this article often, we abbreviate it H&L. In order to illustrate our logic we describe our categorization for eleven journals that were problematic to classify for one reason or another. Journal of Manufacturing and Operations (#30): This was a publication of INFORMS (the Institute for Operations Research and Science) that was discontinued many years ago. Its articles were then folded into Science. We designated it as an OM-dedicated journal. Production and Inventory Journal (#5): This journal was originally a publication of APICS, a practitioner society for production and inventory control professionals, but it ceased publication many years ago. APICS has recently reinstated the journal but with a different orientation, and primarily as an electronic journal directed to P&IC professionals and managers. Given the large number of papers that were published previously in the journal that fell in the OM domain, we have designated this, at least in its original form, as an OM-dedicated journal.

7 440 J.R. Meredith et al. / Omega 39 (20) Table 5 Reliability analysis: correlations between published study journal rankings arranged by data type (perception, citation, other). Spearman Correlation Coefficient (Number of Journals) p-value Perception Data Citation Data Other Study [Barman, 8] [Soteriou, 9] [Donohu, 20] [Barman, 2] [Olson, 22] [Theoharakis, 23] [Zsidisin, 9] [Goh, 24] [Vokurka, 6] [Goh, 25] [Gorman, 7] [Holsapple, 3] [Barman, 8] [Soteriou, 9] [Donohu, 20] [Barman, 2] [Olson, 22] [Theoharakis, 23] [Zsidisin, 9] [Goh, 24] [Vokurka, 6] [Goh, 25] [Gorman, 7] [Holsapple, 3] (7) (5) (7) (6) (9) () (7) (2) (7) (5) (6) (5) (9) 0.50 (9) (0) (5) (9) (2) (9) (7) (9) (5) (6) (8) (0) (5) (0) (4) (5) (5) (8) (0) (3) (6) (2) (7) (7) (8) () 0.02 (3) (9) () (8) (23) (22) (8) (8) (8) (9) () 0.65 () (4) (8) (4) (4) (3) () (30) (6) (9) (2) () () (7) (9) (2) p-valuesr0.05 are in bold. First author listed and reference number. Production and Inventory Review (and APICS News) (#7): This was another publication of APICS that was discontinued, but it was more of a Newsletter than a journal. This publication and the P&IM Journal were occasionally mis-referenced, one for the other. Interfaces (#7): This is another INFORMS journal and is more directed to practice and application than their academic journals. As might be expected, the articles cover the full range of business disciplines, leading us to classify it, as did H&L, as Interdisciplinary. Journal of Manufacturing Systems (#28): This is a publication of the Society of Manufacturing Engineers. Although the journal has occasional articles about the management of manufacturing, the focus is largely on engineering issues, hence we classified it as a reference-discipline journal, Engineering in particular. The society has numerous local chapters and eight technical communities but nothing related to management. The aims and scope of the journal does identify such OM issues as strategy, production planning, quality, and the supply chain as topics of interest, but the great majority of topics are related to manufacturing engineering. International Journal of Production Economics (#9): Although H&L classify this as an OM journal, close inspection of the heavily mathematical papers published in the journal leads to the conclusion that this is a reference-discipline journal, Economics in this case. The aims and scope of the journal note that it focuses on the interface between engineering and management but primarily through the economic environment. European Journal of Operational Research (#6): This journal publishes both articles directed toward extending the theory of operations research as well as applications across multiple areas, and far beyond business, such as agriculture, engineering, government, and so on. H&L classified the journal as Interdisciplinary, probably because of its range of applications, but inspection

8 J.R. Meredith et al. / Omega 39 (20) Table 6 7 journals both ranked in published OM studies and listed in school lists. Journal No. of times listed in 2 published studies No. of times listed in 37 school lists OM-dedicated (D), reference discipline (R), interdisciplinary (I). Journal of Operations 2 28 D 2. International Journal of Production Research 2 24 D 3. Science 2 24 I 4. International Journal of Operations and Production 2 22 D 5. Decision Sciences 2 20 I 6. European Journal of Operational Research 2 8 R 7. Interfaces 7 I 8. IIE Transactions 5 R 9. Operations Research 0 24 R 0. OMEGA International Journal of Science 0 23 I. Journal of the Operations Research Society 0 8 R 2. Naval Research Logistics 0 8 R 3. Journal of Supply Chain 0 7 D 4. Computers and Operations Research 9 6 R 5. Production and Inventory Journal 9 9 D 6. Production and Operations 8 28 D 7. Computers and Industrial Engineering 8 0 R 8. Harvard Business Review 7 I 9. International Journal of Production Economics 6 7 R 20. Annals of Operations Research 5 2 R 2. IEEE Transactions on Engineering 5 5 R 22. Manufacturing and Service Operations 4 9 D 23. Operations Research Letters 4 R 24. Sloan Review 4 I 25. Strategic Journal 4 R 26. Journal of Business Logistics 3 6 D 27. Transportation Science 3 6 D 28. Journal of Manufacturing Systems 3 6 R 29. Mathematics of Operations Research 3 6 R 30. Journal of Manufacturing and Operations 3 2 D 3. Business Horizons 3 I 32. Journal of Quality Technology 2 4 D 33. Industrial Engineering 2 3 R 34. Quality Progress 2 3 D 35. Simulation 2 3 R 36. Journal of the American Statistical Association 2 2 R 37. Technometrics 2 2 R 38. California Review 2 I 39. IEEE Transactions on Systems, Man, Cybernetics 2 R 40. Journal of 2 R 4. Journal of Scheduling 2 D 42. International Journal of Physical Distribution and Logistics 3 D 43. International Journal of Logistics 0 D 44. Production Planning and Control 8 D 45. Supply Chain : An International Journal 7 D 46. INFORMS Journal on Computing 6 R 47. International Journal of Flexible Manufacturing Systems 5 D 48. International Transactions in Operational Research 5 R 49. Logistics and Transportation Review 5 D 50. International Journal of Operations and Quantitative 4 D 5. International Journal of Service Industry 4 D 52. Journal of Heuristics 4 R 53. Mathematical Programming 4 R 54. Networks 4 R 55. Quality Journal 4 D 56. Supply Chain Review 4 D 57. Communications of the ACM 3 R 58. Journal of Global Optimization 3 R 59. SIAM Review 3 R 60. International Journal of Purchasing and Materials 2 D 6. Journal of Purchasing and Supply 2 D 62. Quality Magazine 2 D 63. Service Industries Journal 2 D 64. TQM Magazine 2 D 65. American Journal of Mathematics and Sciences R 66. Engineering Journal R 67. IEEE Transactions on Reliability R 68. Journal of the ACM R 69. Manufacturing Review D 70. Mathematical and Computer Modelling R 7. Production and Inventory Review D

9 442 J.R. Meredith et al. / Omega 39 (20) of its aims and scope ( papers that contribute to the methodology of operational research (OR) and to the practice of decision making ), as well as the papers actually published, leads us to classify it as a reference-discipline journal, namely, Operations Research. Omega: The International Journal of Science (#0): This journal is rather eclectic in the papers it publishes but notes in its aims and scope that it publishes in the specific fields or functions of management, and specifically with implications to the practice of management rather than purely theoretical articles. This focus on the implications to practice was emphasized in recent articles and editorials regarding the journal s audience and the nature of the types of problems authors might address [0 2]. Inspection of the articles appearing in the journal confirms that it covers all aspects of management, such as accounting, information systems, and OM. We hence designated this as an Interdisciplinary journal, which agrees with the H&L categorization. Naval Research Logistics (#2): As stated in the journal s Description, the journal focuses on operations research, applied statistics, and general quantitative modeling with special application to logistics. The papers reflect this orientation accurately and we thus designated this journal as a reference-discipline journal, in this case both Operations Research and Statistics. H&L came to the same conclusion. Decision Sciences (#5): This journal, the face of the Decision Science Institute, used to be a favorite for OM researchers to publish in when there were no OM-dedicated journals (i.e., prior to 980). In spite of the fact that the society, which has 20 interest areas, only one of which is OM, clearly states in its description that it publishes in all the functional areas of business, and does indeed publish across the spectrum of business disciplines, the journal still seems to carry the earlier image among academics of being an OM journal. Part of the reason may be that the OM academics still represent a large interest area in the institute (24% of the membership as of 2009). Nevertheless, the journal has always been an interdisciplinary journal, as H&L classified it, and so did we. Science (#3): This, like Decision Sciences, is also a mainstream journal of a society, INFORMS in this case. And it too publishes across a wide range of disciplines but not just those limited to business. Its editorial statement notes that its articles address management issues with tools from foundational fields (e.g., economics, statistics, psychology, operations research), as well as multidisciplinary research. The institute is comprised of ten societies of which one is Manufacturing and Service Operations, and constitutes somewhat less than 0% of the INFORMS membership (as of 2009). And like Decision Sciences, the articles range across a spectrum of disciplines (including the military, government, and others). Again, like H&L, we classified this journal as Interdisciplinary. Clearly, many journals have contributed to the dissemination of OM knowledge over the years. This is not unlike other business disciplines that develop and benefit from work published in a range of other interdisciplinary and reference-discipline journals. However, not all such journals would be considered to be journals dedicated to a specific business discipline, and that is also the case with OM. Hence, we have selected those journals shown as OM Dedicated from Table 6 and reproduced them in Table 7. As can be seen in Table 7, three OM-dedicated journals (JOM, IJPR, IJOPM) were listed in every ranking study, whereas no journal was listed by every single school on their lists. This may suggest that a bit of tailoring occurs in creating school lists of OM-dedicated journals to match faculty research interests. All OM-dedicated journals appeared at least once in both the collection of ranking studies and school lists. Table 7 30 OM-dedicated journals. Journal 3.3. Final journal rankings No. of times listed in 2 published studies Journal of Operations 2 28 International Journal of Production Research 2 24 International Journal of Operations and 2 22 Production Journal of Supply Chain 0 7 Production and Inventory Journal a 9 9 Production and Operations 8 28 Manufacturing and Service Operations 4 9 Journal of Business Logistics 3 6 Transportation Science 3 6 Journal of Manufacturing and Operations 3 2 a Journal of Quality Technology 2 4 Quality Progress 2 3 Journal of Scheduling 2 International Journal of Physical Distribution and 3 Logistics International Journal of Logistics 0 Production Planning and Control 8 Supply Chain : An International 7 Journal International Journal of Flexible Manufacturing 5 Systems Logistics and Transportation Review 5 International Journal of Operations and 4 Quantitative International Journal of Service Industry 4 Quality Journal 4 Supply Chain Review 4 International Journal of Purchasing and Materials 2 Journal of Purchasing and Supply 2 Quality Magazine 2 Service Industries Journal 2 TQM Magazine 2 Manufacturing Review Production and Inventory Review a No. of times listed in 37 school lists a These publications were discontinued; P&IMJ has recently been resurrected. Finally, we proceeded to rank the journals. The 37 tiered school lists for OM journals collected from AACSB-accredited schools were used to compute a composite ranking [3] for each journal. Initially, three obvious measures were calculated: () the percentage of times each journal was listed in the top tier across schools, (2) the percentage of times in the top two tiers, and (3) the percentage of times in any tier. However, because the number of tiers differed among the schools in the sample, a more sophisticated fourth measure a weighted mean percentile score was then computed for each journal based on its assignment in each school s graded tiers. This score accounted for the actual number of tiers within each school list, the number of journals in each tier, the tier placement of each journal, and the number of schools listing that journal. All journals in the same tier at a given school were given the same percentile measure (the mean for that tier) for that school. These mean percentiles were then aggregated across the schools in the sample by creating an average of the mean percentiles for each journal. The final weighted average mean percentile score was calculated by multiplying the average mean percentile by the number of schools listing that journal in one of their tiers. This group of four school-list metrics, shown in Table 8, reflects how journals are actually judged and used in practice.

10 J.R. Meredith et al. / Omega 39 (20) Table 8 Rankings from the AACSB school lists data of the 30 focal journals. Journal Overall ranking Weighted average mean percentile score % Times listed by schools in any tier (%) % Times listed by schools in top tier (%) % Times listed by schools in top 2 tiers (%) Journal of Operations Production and Operations International Journal of Production Research Manufacturing and Service Operations International Journal of Operations and Production Journal of Business Logistics Transportation Science Journal of Supply Chain International Journal of Physical Distribution and Logistics International Journal of Logistics Production and Inventory Journal Production Planning and Control Supply Chain : An International Journal Journal of Quality Technology Quality Journal Logistics and Transportation Review International Journal of Flexible Manufacturing Systems International Journal of Service Industry Supply Chain Review Quality Progress International Journal of Purchasing and Materials International Journal of Operations and Quantitative TQM Magazine Journal of Manufacturing and Operations Journal of Scheduling Service Industries Journal Journal of Purchasing and Supply Quality Magazine Manufacturing Review Production and Inventory Review However, the weighted average mean percentile score is the most nuanced of the four metrics in that it takes into consideration not only a given journal s tier placement at each school but also the number of tiers at the school and the number of journals graded by the school. Furthermore, in this metric each journal s score is weighted by the number of schools that graded that journal. We believe that this is the best school-list metric available to estimate a journal s actual standing in that it is based on the most information. Table 8 presents the final ranking of the 30 OM journals from the AACSB data based on their weighted average mean percentile scores. The other scores (percentages of tier inclusion) are also shown in the table. In order to verify the bona fides of these rankings we correlated them with the average of the journal ranks across the published OM journal ranking articles listed earlier. The correlation coefficient of (pr) with the published ranking aggregate shows a strong association, indicating that the rankings from the school lists are very consistent with the findings in the published stream. Additionally, we correlated the school rankings with each of the types of ranking studies (perception, citation, other). While each type of ranking correlated strongly with the school lists (reality), perception studies were slightly stronger in association (0.77, pr). Table 9 presents the correlations. The rankings of OM-dedicated journals from the school lists and the published ranking studies are produced side-by-side in Table 0. The ranking from the published ranking studies was established by using a double-weighted calculation. First, the rank for each journal in each study was divided by the number of journals ranked in that study. The underlying basis for this adjustment is to take into account the number of journals ranked in each study (i.e., a journal that ranked number 5 in a study of 200 journals should get a better ranking than a journal that ranked 5 in a study of 0 journals). Second, the average of these adjusted ranks was computed across studies and divided by the number of studies in which each journal appeared. This adjustment factored in the number of times each journal appeared in the collection of studies (i.e., a journal that was included in 0 studies presumably has more visibility and impact than one that was present in only one study). This final double-weighted average is the determinant of the journal s overall rank (the smaller the double-weighted average the better the rank). The primary reason for discrepancies between the aggregate rankings from the published studies versus the AACSB-accredited school lists is believed to be the timing of the two sets of data, with the AACSB data reflecting publication reality in early 2007 whereas the published studies go back almost two decades. 4. Discussion The comparison conducted in this study of actual lists used to assess faculty research and the published ranking stream is both needed (given that examining measures is a norm in scientific inquiry), and valuable, given the importance attached to publishing in the top journals in evaluation, promotion, tenure, award, nomination, and other such high-profile decisions. Several points of interest arise from the results of comparing the published stream of OM ranking papers to the actual in-house lists used by AACSBaccredited schools to evaluate faculty research. First, while the aggregate ranking from the published studies is significantly (and positively) correlated with the in-house lists, there is some dissimilarity in the rankings, even for the top journals. As noted earlier, the major reason is believed to be the difference in time periods between the published studies, on average about a decade. For instance, the journal Production and Operations has

11 444 J.R. Meredith et al. / Omega 39 (20) Table 9 Correlations between sources of journal rankings. Source Spearman correlation coefficient (number of journals) p-value School lists Published studies all Published studies perception Published studies citation Published studies all (30) Published studies perception (23) (23) Published studies citation (0) (0) (6) Published studies other (3) (3) (0) (5) Table 0 Ranking of 30 journals: school lists versus published ranking studies. Journals School lists Published ranking studies Journal of Operations Production and Operations International Journal of Production Research 3 2 Manufacturing and Service Operations 4 5 International Journal of Operations and Production Journal of Business Logistics 6 9 Transportation Science 7 8 Journal of Supply Chain 8 6 International Journal of Physical Distribution and 9 5 Logistics International Journal of Logistics 0 0 Production and Inventory Journal 7 Production Planning and Control 2 22 Supply Chain : An International Journal 3 7 Journal of Quality Technology 4 6 Quality Journal 5 25 Logistics and Transportation Review 6 7 International Journal of Flexible Manufacturing 7 29 Systems International Journal of Service Industry 8 22 Supply Chain Review 9 9 Quality Progress International Journal of Purchasing and Materials 2 2 International Journal of Operations and Quantitative TQM Magazine Journal of Manufacturing and Operations 24 Journal of Scheduling 24 3 Service Industries Journal Journal of Purchasing and Supply 27 4 Quality Magazine Manufacturing Review Production and Inventory Review 30 2 substantially improved in stature since it began publication in the early 990s, but it always takes time for a new journal to establish its reputation. Similarly, the even more recently initiated journal Manufacturing and Service Operations, sponsored by INFORMS,isranked4thintheAACSBlistand5thinthepublished studies list, reflecting the same delay in acceptance for new journals. And the British journal International Journal of Operations and Production, whose stature appears to have improved considerably over time in the published studies, suffers slightly in the AACSB list which is more heavily weighted with American schools (3 of the 37 schools with tiered OM lists). Other effects may also be seen in the data. For example, the large discrepancy in the published vs. AACSB rankings for the Production and Inventory Journal reflect the loss of stature when an organization decides to terminate a journal; and like the difficulty of establishing the stature of a new journal, the falloff in stature of a defunct journal is equally slow, having fallen only 4 ranks in a decade or so. Also, there may be a quantity effect, with International Journal of Production Research, which is published 24 times a year with a large number of papers in each issue, holding a high ranking on both lists. And the long-established Journal of Business Logistics, ranked 6th in the AACSB list but only 9th in the published studies list, may attest to the value of keeping a journal publishing in its area of strength (in this case, an area that has come into prominence in the operations field: supply chains and logistics) for an extended duration. On the other hand, the Journal of Supply Chain shows the opposite effect, holding the rank of 6th in the published studies list but only 8th in the AACSB list, possibly due to the many changes in the name of this journal over the years and the confusion this generates. Since this study is the first to present and compare rankings of OM-dedicated journals from this new source of AACSB in-house lists and published journal ranking studies, we can also get a sense of how OM-dedicated journals fare when ranked with related reference and interdisciplinary journals, as compared to ranking solely OM journals. Journal rankings offer one cue suggesting the value placed on different collections of knowledge and, whether explicitly or implicitly, provide incentives for scholars to publish in certain journals. If journals of distinct disciplines or research areas are combined, the focus and potential development of the fields involved becomes diluted. We thus suggest that there are benefits in separating journals into distinct disciplines and ranking journals within those disciplines. An academic discipline is a branch of learning or scholarly instruction [4]. As such, academic disciplines have become authoritative communities of expertise [5, p. 63]. The community of a discipline influences theory development and shapes the types of phenomena that scholars in that community attempt to understand. Journals are a vital component of the academic community as a key vehicle of communication among scholars. In fact, it has been claimed that a discipline s identity is primarily created by the journals that publish in that discipline [6]. As faculty are offered incentives for publishing in a field s top or premier journals, (e.g., greater mobility, research awards, enhanced reputation, merit pay, etc.), clarity in identifying these top journals is important. One only need reflect on past tenure and promotion