Rival, substitutable or complementary? Comparing the configurational and linear additive analysis of innovative outcomes of R&D consortia

Transcription

1 Rival, substitutable or complementary? Comparing the configurational and linear additive analysis of innovative outcomes of R&D consortia Jörg Raab, Remco Mannak, Marius Meeus, Sander Smit (all Tilburg University) Paper to be presented at the conference Qualitative Comparative Analysis (QCA) - Applications and Methodological Challenges International Conference at Goethe-University, November 22-23, 2013 Work in progress. Please do not cite or quote without permission! Contact: Jörg Raab, j.raab@uvt.nl

2 Abstract One of the most prominent questions in management and organization studies is the question, which factors in which configurations contribute to the innovative capacity of organizations, consortia and organizational networks. This is not only relevant for the theoretical development but has far reaching implications for government policies. In the paper, we will analyze the innovative outputs of 96 research & development consortia that were established between 1983 and 2004 in the Dutch water sector. These consortia were part of a government subsidized innovation program that stimulated joint research and product development between Dutch universities and Dutch companies. Within the program, 4-5 year research projects with multiple parties that had to include a project leader from a university and so called users, i.e. Dutch companies, were funded. Output was reported as a standardized rating in terms of incomes and product. Data has been collected on independent variables such as partner diversity in the project, subfield, consortium size, resource munificence, relational experience of the project leader as well as user activity level and quality of the researcher. In addition, about 51 interviews have been conducted with project leaders and users. We first performed an fsqca to explore possible paths which, however, did not deliver any informative results. We then conducted first a traditional linear regression, testing the additive net effects of independent variables mentioned above on innovative output and second a fuzzy set QCA analysis again with a configurational theoretical approach in mind with the variables that appeared to be of importance in the linear regression. We then assess the results and the analytical process to discuss advantages and disadvantages of both approaches and their combination in this particular field of study. We hereby contribute to two themes of the workshop: QCA and large N as well as QCA and mixed methods. 2

3 Introduction One of the most prominent questions in management and organization studies is the question, which factors contribute to the effectiveness of organizations, consortia and organizational networks. Even though effectiveness is an often disputed concept it is also called the ultimate dependent variable behind a majority of the current organization theories. After all, one would like to know what works and what does not. With regard to innovation, innovative capacity is a prominent outcome variable that characterizes organizations, organizational consortia and networks. This is not only relevant for the theoretical development but has far reaching implications for organizational and government policies alike that try to improve innovative capacity in order to stay on top of the competitive race between organizations and countries. The field of management and organization in general and the subfields that seek to explain innovative capacity or innovative outcomes in particular fit the general trend in the social sciences where a lot of studies exist that are conducted as small n (comparative) case studies and a lot of studies with large N (see Ragin 2000:24-25). In theoretical terms, the field is very much dominated by a general net effects thinking, where the influence of individual variables is looked at under ceteris paribus assumptions of some general variables such as size, resources, etc. One might find an occasional interaction effect of two variables usually theoretically conceptualized as a moderator model. However, from a practitioner perspective it seems almost obvious that it is the combination of several factors that lead to certain outcomes and given the specific circumstances of organizations, consortia or networks, different combinations might lead to similar outcomes. It therefore seems promising to approach this question from a configurational perspective and actively look for equifinality. This in turn should lead to much more realistic policy recommendations than results based alone on the assumption that individual variables independently affect innovative outcomes. Configurational ideas surfaced once in a while in management and organization studies in the 3

4 last three decades. For example, one of the major organization theories, contingency theory, is built on the idea that internal organizational configurations must have a fit with certain factors such as size, technology and environmental uncertainty in order to lead to organizational effectiveness. Meyer, Tsui and Hinings (1993) argued that we should understand configurational theory as being different from contingency theory and see it as a holistic understanding of configurations of organizational components that can be equifinal in their impact on organizational effectiveness. They also emphasized to apply configurational thinking to all levels of organizational analysis. In one of the seminal papers on the effectiveness of networks, Provan and Milward (1995) suggested a theoretical framework that was in principle based on a configurational approach. In 2005, Snow, Miles and Miles tried to reinvigorate interest in the configurational approach to organization design. However, the configurational ideas never really received traction in empirical research largely because of the dominance of correlation based methods and a lack of established alternative configurational/set theoretical methods. It is only recently, that a broader movement seems to gain momentum to establish configurational theory and methods in the mainstream empirical research in management and organizational studies (Fiss 2007, 2011; Fiss, Cambré and Marx 2013). Despite their theoretical and methodological potential, set-theoretical methods certainly also have their limitations and given their ongoing development still face many open questions. In the research presented here we therefore apply, compare and assess set-theoretical and correlational methods in examining the (set of) conditions that lead to R&D-consortium success. The main aim of the paper is to reveal benefits and limitations of both methods as well as the potential for their combination in a research situation of frequent occurrence: namely where important conditions are identified, but their interplay is unknown. In this study we examine the performance of 96 multi-partner R&D-consortia in the Dutch water sector. 4

5 We investigate the relation of consortium performance with the conditions subfield, resource munificence, consortium size, experience of the project leader, partner diversity, user activity and researcher quality. We consider the research setting as particularly applicable for the comparison of the correlational and set-theoretical approaches, because (I) relevant conditions could be identified (based on theory and qualitative empirics), but not whether the conditions provide alternative paths to success (equifinalty) or are part of a single path (conjunction). Accordingly, a set-theoretical exploratory approach seems to be more applicable than a correlational (hypothesis testing) approach (II) there are no ex ante threshold values for the calibration of (some of) the conditions. E.g. the interviews provide no insights in the distinction between low and high resource munificence, which speaks in favor of using a correlational approach. 5

6 Theoretical Background Innovation depends more and more on inter-organizational collaboration as a consequence of the wide dispersion of resources and skills over different organizations (Chen & Yun, 2008; Powell, et al., 1996; Schilling & Phelps, 2007). In order to share risks, resources and skills, organizations can form an R&D-consortium. Such an R&D-consortium is a social unit on its own (Das & Teng, 2002). Consequently, firm s incentives to participate in an R&Dconsortium can be twofold, namely to achieve the collective goal (e.g. new product development), or to achieve the individual goal as a consortium partner (e.g. to obtain and exploit knowledge of partners) (Cohen & Levinthal, 1989; Jones, Hesterly, Fladmoe- Lindquist, & Borgatti, 1998). To evaluate the different analytical techniques we concentrate on the consortium success (achievement of the collective goal) even though we realize that goal accomplishment at both levels can be mutually reinforcing, co-existing or provide a tension between the collective and individual level. Consortium success is conceptualized as a combination of both the outcomes in terms of new products and additional income for the participating companies which are the main goals of the R&D funding scheme. Independent variables/explanatory factors R&D-consortia in the specific funding scheme are composed of an individual project leader, a scientist affiliated with a Dutch university and so called users, i.e. industry partners who have an interest in the research to develop new products and bring them to the market. Project leader and industry partners jointly hand in a research proposal to the funding agency in order to receive funding. Industry partners also have to contribute either in kind or in cash to the research activities. Most of the time one or several PhD positions are funded with the grant money. Therefore, a research team is set up consisting of the project leader (researcher), one or several PhD students and the representatives of the industry partners. In trying to explain 6

7 consortium success, we included partner diversity in the project, relational experience of the project leader, user activity level, quality of the researcher, resource munificence, consortium size as well as sector subfield in the analysis. Partner diversity in the project A large literature has developed focussing on the effect of different dimensions of diversity of partners on team performance (Knippenberg et al. 2004, Harrison and Klein 2007). Reagans Zuckerman and McEvily (2001) argue that team diversity reduces internal density but increases network range, i.e. the potential variety of non-redundant information and knowledge with an overall positive effect on productivity in R&D teams. Harrison and Klein (2007) distinguish between variety, separation and disparity in teams. In terms of variety one can think of different experiences or knowledge (Harrison and Klein 2007) as they come into being through differences in educational and professional backgrounds of team members often also referred to as cognitive distance. Separation refers to differences in position or opinion on value, attitude, or belief among team members (Harrison and Klein 2007) for example with regard to the importance of technological progress vs. commercial success. Goal consensus/diversity also falls in this dimension. Disparity on the other hand corresponds to the idea that team members might be different with regard to socially valued assets or resources like status or income (Harrison and Klein 2007). In R&D consortia this could be differences in professional status or reputation of the industry partners. Recent research has shown that effects on team performance are different for each dimension for example in combination with the density of communication within a team (Curseu et al. 2012). We focus here on partner diversity as variety in terms of technological variety. Here, it is often assumed that a curvilinear relationship exists between variety and outcomes, i.e. too little and too much variety are less conducive to performance than an optimum in the middle, where cognitive 7

8 distance is not too large to be overcome by deliberation but diverse enough to create new insights for the parties (Gilsing et al. 2008). Relational experience of the project leader As R&D-consortia often are characterized by long time-horizons and high interdependencies (Das & Teng, 2002a), selection of reliable and complementary partners is essential for organization and consortium success (Emden, Calantone, & Droge, 2006; Vissa, 2011). Previous studies have emphasized the importance of prior experience for successful R&D-partnering (e.g. S. X. Li & Rowley, 2002; Powell, et al., 1996). Furthermore, it has been demonstrated that repeated collaboration experience, achieved through direct collaborations with a partner, provides the opportunity to evaluate the partner s capabilities as well as the partner s trustworthiness (Gulati, 1995; S. X. Li & Rowley, 2002). This is particularly important in R&D-consortia, where the technological risks are relatively high (Lavie, Kang, & Rosenkopf, 2010). In addition, repeated collaboration experiences can provide mutual understanding and can increase the opportunity to align the project goal and execution to the consortium members' capabilities and interests (Ring, Doz, & Olk, 2005). In this way, repeated collaboration experience can enhance the probability of organization and consortium success. However, it is important to take the amount of prior collaborations into account (Li, Eden, Hitt, & Ireland, 2008), as repeated collaboration experiences are not without constraints, but can cause creative lock-ins and inertia, increasing the risk of consortium failure (S. X. Li & Rowley, 2002; Polidoro, et al., 2011). The project leader is of great importance for these kind of R&D projects, since s(he) is mainly putting together the consortium, writes the grant proposal and coordinates the research we focus of his/her experience in collaborating with the specific industry partners. 8

9 We state that the potential benefits of repeated collaboration experience (e.g. the information about the partner s capabilities, the opportunity to align the project execution) and the constraints of repeated collaboration experience (lock-ins, reduced creativity), do not come in the same pace: Information benefits can be achieved through repeated collaboration in a single project, while lock-in constraints are more likely to arise over multiple joint projects. Therefore, the consortium success will be enhanced by repeated collaboration experience, until a threshold is reached. We therefore expect a curvilinear relationship between relational experience of the project leader and the productive outcomes. User activity level Whereas collaboration experience indicates the potential resources and skills that project leader and partner organizations contribute to the consortium, user activity indicates how the organizations effectuates those resources and skills. Cohen and Levinthal (1989) found that organizations in R&D-consortia engage in consortia in different ways. Some partners aim at co-development of new knowledge (active role), whereas other partners rather assemble existing knowledge (passive role). Accordingly, we state that consortium members can fulfill an active role, contributing to the collective goal, or a passive role, mainly monitoring the R&D-project. Active members are more likely to commit more own resources and engage more in the R&D process, since they have a keen interest in the development of new knowledge. However, a passive role fulfillment of an individual organization does not necessarily harm the collective interests, and can still provide benefits to the consortium. While active organizations contribute to the achievement of innovative outcome, passive organizations can provide critical mass to the consortium. In this way, passive organizations can enhance the consortium legitimacy (Human & Provan, 2000). In addition, the passive organizations can provide a safeguard against information redundancy and potential lock-in 9

10 (Uzzi, 1997). Accordingly, we expect that the mix of a few active organizations and critical mass of passive members provides the most optimal composition for consortium success. Given the consortium size, particularly the amount of active organizations is decisive for the consortium success: A lack of active members can hamper the organizing capacity of the consortium, while an overload of active organizations can become a liability, accompanied by goal ambiguity and management issues. Therefore we expect that a consortium is most likely to be successful when the amount of active organizations is moderate rather than low or high: Quality of the researcher Developing new products is a knowledge intensive process. Therefore not only the relational experience or the diversity or communication between partners play a role but also the level of skills and competences of both the organizations as well as the individuals involved. Human capital is therefore presumably of great importance, even though it is rarely looked at in team or R&D research. We focus here on the quality of the PhD students that are hired, since it is primarily them who do the day to day research in the R&D projects. We expect a positive linear relationship between the quality of the researcher and the productive outcomes. Resource munificence It is almost common sense that the amount of resources matter for the productive output of groups, organizations or networks. Most of the time, resource munificence is used as a control variable in linear regressions, since researchers are interested in other effects and therefore have to control for the amount of resources to isolate these effects. However, from configurational theoretical approach, resource munificence is most likely a necessary but not sufficient condition for effectiveness/consortium success as for example found by Provan and Milward (1995) for engineered networks. In a linear additive approach, one might expect an 10

11 interaction effect for example with quality of the research, i.e. the resources are more effectively used if quality of the researcher is high. One could also argue, however, that the relationship between resources and outcome is curvilinear, i.e. the effect is not increased anymore after a necessary level is reached. Consortium size Size is an important factor for all sorts of group processes and is usually included as a control variable in linear regressions on the effectiveness of teams for example. We also wanted to explore whether we would find different configurations for small and large consortia. One could for example argue that the level of user activity is dependent on the size of the consortium and therefore has a joint effect on the outcome. Sector subfield The water sector consists of the subfields Maritime technology, Delta technology, and Water technology. Even though they all belong to the water sector activities, technologies and type of projects differ quite substantially between the sectors. We therefore included it as a control in the regression and wanted to investigate whether different configurations would apply to different sectors. A configurational approach to R&D performance While relative clear cut theoretical arguments can be made for the net effect of any individual variable it is very difficult to come up with a sound theoretical reasoning for the joint effects of variables/factors but for interaction effects that exceeds two variables. It is therefore, not surprising that this is also what we usually find in studies on the outcomes of teams or R&D consortia. However, it seems relatively likely that factors jointly work together for certain 11

12 productive outcomes to occur. In addition, it is also likely that it is not only one configuration of factors that might lead to success. One could for example think that user activity might compensate for resources so that two paths might occur (equifinality). In addition, one could also argue that configurations of factors that lead to very successful outcomes are substantially different from those that lead to medium success or to failure, i.e. are not simply each other's negations (asymmetry). There is therefore ample theoretical reason to approach the analysis of outcomes of R&D projects from a configurational perspective and explore the data with fsqca. 12

13 Methods Research Setting In 2011, the Dutch Government announced a new innovation policy in order to stimulate 9 so called Top-sectors to maintain or achieve a world-leading position through science industry collaboration (Kamerstukken 2010/ nr. 1). One of these sectors is the Water sector in the Netherlands, including the subfield Maritime technology, Delta technology, and Water technology. With respect to the Water sector, the policy itself is not that pioneering: due to the fact that a large part of the Netherlands is located below the sea level, Dutch water research has a long history and has become internationally well esteemed (Karstens et al., 2011; Maritiem Cluster in de Topsector Water, 2011). The Water sector therefore is an excellent empirical setting to study collaboration and the effectiveness of R&D-projects. Between 1981 and 2004, more than 100 R&D-consortia in the water sector were funded by an agency that pursues a Dutch university-industry collaboration stimulation policy. The unit of analysis is therefore the R&D project. After excluding 6 cases because of missing values the following analysis is based on 96 cases. Measurement Outcome Consortium performance was operationalized as the sum score of product and income dimension. The funding agency (STW) requires project leaders and partners to indicate at the end of each project after 5 years what was achieved with regard to the development of new products and with regard to additional income. Below is an overview about the qualification of the outcomes in three different categories A, B and C. On the basis of this classification, we calculated a utilization score which ranges from 0 6 adding up the outcome for the product and income category. A is converted to a score of 1, B to 2, and C to 3. A value of 0 13

14 signals a complete failure mainly due to project termination, while a value of 6 indicates the existence of a tangible product (or a finished concept a firm can work on independently) and a (potential) large stream of additional income. It should be noted that a sum score of 3-5 can indicate different performance for product and income with the same sum score. Product Income Product Income Sum 0 the project has failed or in the research stage was terminated early because the project failed scientifically, or because no user can be found, this project has failed to generate income and no income is expected in the future A B C There is no concrete product. Further research is necessary to obtain a useable product. Preliminary conclusions have already been drawn, but various matters still need to be verified. We are still at the basic technology stage. Up until this moment, the principal form of output has been scientific Publication A preliminary model, principle or draft method is developed and usable. Before there can be talk of a final product, verification and fine-tuning are still necessary. The user cannot (yet) use the product completely Independently a tangible product exists, for example in the form of software, a working prototype, a process description, a patent: in short, there is a concept that is more or less finalized, which the user can start to work on independently the project has not yet generated any income. Contributions towards the research may well have been made, but as yet there is no revenue from knowledge exploitation. Future income is however not ruled out (part of) the knowledge was (or will be) sold incidentally. Here, income may also connote the fact that the result is of value to society there is (has been) a significant, constant or large stream of income, or there is a possibility that such might be realized within the next five years. For example if principal agreements have already been drawn up Independent variables/explanatory factors were measured in the following way: Partner Diversity: Number of different observed BIK-types in start year of project for industry partners/users. Every firm has a BIK-qualification, which can change over time. BIK = Bedrijfsindeling kamers van koophandel; more recently SBI = Standaard Bedrijfsindeling. 14

15 The BIK/SBI is based on the activity classification of the European Union and on the classification of the United Nations (International Standard Industrial Classification of All Economic Activities, ISIC). Classification used on the subindustry level such as wood or paper industry which both belong to the industry category manufacturing. Experience Project Leader: Number of projects of project leader that started before focal project, and included one or more of the current users in the user committee. Researcher Quality: Average number of publications per observed researcher in the period project start year through start year +5 [google.scholar no citations or patents]. User Activity: Number of active contributions: in person, in cash and/or in kind by users. Accordingly max 3 contributions per user. Resources: Financial resources as provided by the funding agency (research grant) in euro corrected for inflation. Size: Number of firms in user committee, on interim and/or utilization report. Water subfield: Maritime technology (reference); dummies for delta technology; water technology. Analysis In analyzing the data, we proceeded in the following way. First, the descriptive statistics and correlations of the variables were calculated and the threshold values of the different factors determined. Second, a necessity test is performed for all conditions, followed by the fuzzy-set analysis with the truth table algorithm for all conditions. As will be visible in the results section, the findings of this first fsqca were very inclusive and very difficult to interpret. Third, regression analyses are performed for the full set of conditions as well as an identified subset. Fourth, fuzzy-set analyses are conducted for the selected subset of conditions. Fifth, the results of the fuzzy-set QCA are compared with the results of a configurational 15

16 regression analysis. Finally, an additional fsqca are conducted for different outcome thresholds, condition calibrations and solution frequency thresholds. Findings Tables 1and 2 show the descriptive statistics and the correlations. The correlations table shows that size correlates with the amount of resources, user activity and extremely high with partner diversity. The two dummies for subsector correlate negatively relatively high indicating that R&D projects to a large extent are located either in the one or the other subsector. With regard to the dependent variable consortium performance, only researcher quality shows a positive correlation on the.05 level. Insert tables 1 and 2 about here In terms of calibration, i.e. determining the threshold values for the fsqca analysis, the crossover point is fixed at the mean [besides the delta and water technology dummy]. The threshold values for full (non-)membership are based on the cross-over to the 5% upper- and lower bound in terms of number of cases. These statistical threshold values are chosen as qualitative evidence for specific threshold values are lacking. Insert table 3 about here As table 4 on the necessity test show, none of the conditions meet the consistency threshold of 0.9 as prescribed by Schneider and Wagemann (2012, p 278). Accordingly the model does not include any conditions that are necessary to achieve consortium success. 16

17 Insert table 4 about here Below the results are shown for the fuzzy set QCA with the truth table algorithm, including all conditions. The analysis tests the presence of one or more sufficient paths that lead to successful consortium performance. For all fsqca, the table reports the intermediate solution and core conditions resulting from the parsimonious solution. Analyses are based on a solution frequency threshold defined as: more than 1 case per solution and at least 75% of the total number of cases (as indicated in the fsqca manual p78, Ragin, Krass and Davey 2008); and a consistency threshold of 0.75 (e.g., Ragin, 2006, 2008 in Fiss, 2011). In case of Prime Implicants, all PI are selected. The analysis reveals 13 different paths which is extremely complex and consequently not very informative. A Parsimonious Solution (technical term for one of the three fsqca outcomes) is not achieved. In case solutions with frequency 1 are also included, the solution becomes even less parsimonious, including 31 different paths. Accordingly the QCA with all conditions does not result in any informative findings. Insert table 5 about here The regression analyses presented in table 6 below show that researcher quality has a statistically significant positive effect on performance (the curvilinear effect is marginally statistically significant in model 3 and 6; statistically significant in model 5). User activity has a statistically significant curvilinear effect (diminishing marginal returns) on performance while partner diversity has a statistically significant negative effect on performance, but the effect is not robust in model 4 and 5 (without control variables. The curvilinear effect for partner diversity is marginally statistically significant in model 3. All other conditions are not related in a statistically significant way to performance. The explained variance is highest for 17

18 model 3 and 6 with 24% respectively 22%. We therefore can conclude that researcher quality, user activity (curvilinear effect) and partner diversity seem to be the most relevant variables in explaining consortium success. Table 6: Regression analyses on performance Model 1 Model 2 Model 3 Model 4 Model 5 Model 6 Intercept 3.517*** 3.047*** 3.033*** 3.070*** 3.152*** 3.148*** Delta technology dummy Water technology dummy Resources Resources^ Size Size^ Experience Project Leader Experience PL^ Partner Diversity * Partner Diversity^ User Activity * ** 0.478** User Activity^ * * ** Researcher Quality 0.049* 0.170* 0.047* 0.175** 0.176** Researcher Quality^ * N ANOVA (F) * ** 2.222* R^ R^2 Change * ** *** p<0.001; ** p<0.01; * p<0.05; p<0.10 For the purpose of the methodological comparison, we conducted a fsqca with the subset of the conditions partner diversity, user activity and researcher quality (see table 7 below). Therefore, the outcome of the regression analyses is used to define the subset of initial conditions that are relevant to this specific research setting. 18

19 Table 7: QCA on performance with selected conditions Solution Configuration Partner Diversity User Activity Researcher Quality Consistency Raw coverage Unique Coverage Overall solution consistency: Overall solution coverage: present not don't care Core Periphery Solution frequency threshold at least 12 cases (79% of all cases), consistency threshold at least The fsqca with the subset of conditions results in an overall solution consistency of and overall solution coverage of which is acceptable. User activity, partner diversity and researcher quality are core conditions, as shown in the parsimonious solution. The solution consists of three paths: Performance = User Activity * ~Researcher Quality + Partner Diversity * ~Researcher Quality + Researcher Quality * ~Partner Diversity * ~User Activity. The overall solution is a typical example of equifinality, as the three core conditions provide separate paths to consortium performance in absence of one or more of the other conditions. The raw coverage of all three paths is relatively high as they represent a substantial amount of cases. Interestingly, one paths figures partner diversity as a core condition, even though it had 19

20 a small negative linear effect and a statistically significant curvilinear effect in the regression. The path "Partner Diversity * ~Researcher Quality" has a very low unique coverage, however. This seemingly contradictory result between regression and fsqca findings that partner diversity has indeed a more negative influence on performance but there are a few cases, in which in the absence of researcher quality, i.e. low quality, partner diversity becomes positive. We then took the results of the fsqca analysis and conducted another regression with the interaction terms based on the three paths identified in the QCA analysis. The next section shows the results of the configurational regression analysis to test if these results are in accordance with the fsqca results (see table 8) below. The first regression model includes the three conditions separately and therefore is most comparable to the parsimonious solution of the fsqca that showed that the three core conditions are separate paths to success. Only for researcher quality, the regression analysis is in accordance with the parsimonious fsqca solution. The second and third models show the two-way and three-way interaction effects. The effects of partner diversity and user activity both are (marginally) statistically significant after inclusion of the interaction terms. However, the interaction terms are not completely in accordance with the fsqca outcome: the interaction term of partner diversity and researcher quality is negatively statistically significant, comparable to the fsqca result. However, the interaction term between partner diversity and user activity is negatively statistically significant as well, while this is a don t care term in the fsqca. On the contrary, the interaction term between user activity and researcher quality is not statistically significant, while the fsqca outcome suggests a negative interaction term. It must be noticed that only model 2 provides a statistically significant fit to the data, emphasizing a configurational approach in this research setting. The explained variance is.132, which is relatively low. The 20

21 solution coverage of the fsqca on the other hand is 0.733, which is in our view quite acceptable. Insert table 8 about here After having received some substantial insights in terms of core conditions and possible paths for consortium success and having explored how correlational and set-theoretic methods could be combined in general we now turn to the analysis of the robustness of the results with regard to the thresholds chosen for the fsqca analysis. In the previous section, (I) the crossover point of individual conditions was fixed to the mean, (II) the solution frequency threshold was fixed to more than 1 case per solution and at least 75% of the total number of cases; and (III) the consistency threshold was fixed to In this section, we will discuss the results of variation in the cross-over point and the solution frequency threshold. We also look in particular at the outcomes for very high performance and at the threshold variation conditions with a curvilinear effect. QCA on very high performance First, the cross-over point for the outcome is increased from 3.56 ( performance ) to 4.5 ( high performance ) and eventually to 5.5 ( very high performance ). The results are shown in table 9 below. Insert table 9 about here The fsqca for high performance results in the following path High Performance = Researcher Quality * ~Partner Diversity * ~User Activity. 21

22 This path is 'more selective' than the paths to performance. The paths User Activity * ~Researcher Quality and Partner Diversity * ~Researcher Quality both lead to performance but not to high performance. Only the path Researcher Quality * ~Partner Diversity * ~User Activity leads to both performance and high performance. This is a more parsimonious solution than for 'performance' which is not a contradiction, because you explain a different subset of cases. The fsqca for very high performance (cross over point set at 5.5 out of 6) results in no solution given the solution frequency and consistency thresholds.\ Threshold variation conditions with a curvilinear effect As shown in table 6, the results of the regression analyses suggest that the effect of user activity on consortium performance is rather curvilinear than linear. In this section, we discuss the potential implications of conditions with a curvilinear effect for the calibration of the condition. In case a condition has an inverted u-shaped relation to the outcome, traditional calibration with a single cross-over point between non-membership and membership can be insufficient. Two cross-over points are needed to distinguish medium membership from low and high membership, as shown in figure 1. In this study, we recoded the condition user activity in medium user activity and used two cross-over points to distinguish the membership of medium user activity from the 25% non-membership lower bound (cross-over point 0.50) and the 25% non-membership upper bound (cross-over point 3.5). As many curvilinear effects do not follow an inverted u- shaped pattern but rather a diminishing marginal returns pattern, we computed another condition high user activity including only the 25% upper bound as members (3.5 as cross over point). Finally, we conducted separate analyses with the condition medium user activity included and the condition high user activity included. 22

23 Inverted u-shaped effect Diminishing marginal returns Non member Member Non member Non member Medium High Figure 1: conditions with a curvilinear effect The fsqca analysis shows the following results: Table 10: QCA on performance with curvilinear user activity effects Solution: with Medium User Activity Solution: with High User Activity Configuration Partner Diversity User Activity (Medium/High) Researcher Quality Consistency Raw coverage Unique Coverage Overall solution consistency: Overall solution coverage: present not don't care Core Periphery Solution frequency threshold subsequently at least 8 cases (85% of all cases) and at least 11 cases (77% of all cases), consistency threshold is at least The analysis reveals the following paths. For Medium User Activity (between 1 and 3) Performance = Researcher Quality * Medium User Activity + Researcher Quality * Partner Diversity + Partner Diversity * Medium User Activity. 23

24 For High User Activity (larger than 3): Performance = Researcher Quality * ~High User Activity + Partner Diversity * ~High User Activity. In both fsqca, the core conditions are researcher quality and partner diversity, while user activity is a peripheral condition. A remarkable distinction, however, is the fact that the overall solution of the first fsqca includes two paths with the presence of medium user activity while the second fsqca includes two paths with the absence of high user activity. The curvilinear effect therefore is also visible in the fsqca. It seems that after a certain threshold, an increase in user activity does not lead to any additional performance effects or even becomes too much of a good thing, i.e. returns become slightly negative under the condition of diversity or researcher quality. Threshold variation solution frequency In this section, we test alternative solution frequency thresholds. In line with Fiss (2011) the solution frequency threshold is fixed at a minimum of 3 cases. The consistency threshold is fixed at 0.75 (e.g., Ragin, 2006, 2008 in Fiss, 2011). Insert table 11 about here The fsqca with all conditions includes six paths, but is still little parsimonious and little informative. Table 12 below shows the results of the fsqca with the selected conditions researcher quality, partner diversity and user activity with alternative thresholds (see table 7 for the original analysis with minimum threshold 12 cases). 24

25 Table 12: QCA on performance with selected conditions Solution Configuration Partner Diversity User Activity Researcher Quality Consistency Raw coverage Unique Coverage Overall solution consistency: Overall solution coverage: present not don't care Core Periphery Solution frequency threshold at least 3 cases (100% of all cases), consistency threshold at least The three core conditions are still observed, but no longer conditional on the absence of the other conditions. Accordingly, three (more parsimonious) independent paths are found. Consistency and coverage levels are comparable. This implies that there is a group of cases that share the core conditions but differ on the peripheral conditions, which makes them disappear in the paths, if the frequency threshold in the truth table is lowered. Peripheral conditions therefore account only for the large and not the smaller subgroups. Insert table 13 about here When including all cases (minimum of 3 as threshold), also for very high performance a more parsimonious solution emerges with researcher quality as the single core condition (see table 9 for the original analysis). Naturally, coverage levels are a bit higher and consistency scores a bit lower. 25

26 Insert table 14 about here Lowering the solution frequency to three does now reveal a path to very high performance (including 4 cases) while originally no path appeared. In those cases, very high performance was achieved through a combination of researcher quality, user activity and a relatively homogenous partner set (absence of partner diversity). In case of the analysis of the curvilinear effect, the alternative threshold also yields a more parsimonious solution with the core conditions researcher quality and partner diversity for medium user activity and three separate paths with singly core conditions researcher quality, user activity and partner diversity for high user activity. Insert table 15 about here Medium User Activity is no longer a peripheral condition in the paths to performance of the first fsqca. High User activity is now an independent path to consortium performance in the second fsqca, and the absence of high user activity is no longer a peripheral condition. Also here, consistency levels are slightly lower and coverage levels slightly higher (see table 10 for the original analysis). Discussion and Conclusion In this admittedly experimental study we attempted to explore the influence of variables such as partner diversity in the project, consortium size, resource munificence, relational experience of the project leader as well as user activity level and quality of the researcher on the performance of R&D projects in the Dutch water sector. Even though theoretical arguments about the potentially positive influence of these variables on the performance of 26

27 R&D projects can be found in the literature, no prior information was available as to how these factors might work together in configurations to bring about certain outcomes. We therefore started to analyze the data with a fsqca of all conditions, which however, did not yield any informative results. We then conducted a linear regression and found researcher quality, partner diversity and user activity to be core conditions. We subsequently ran a new fsqca with these core conditions. The fsqca showed that three separate paths existed with these factors as core conditions and the other two either as negated or as don t care conditions. A lower frequency threshold yielded more parsimonious solutions but confirmed the existence of three separate paths. The fsqca analysis also revealed that there are cases where partner diversity in the absence of researcher quality leads to project performance. This was unexpected given the results in the linear regression, where partner diversity had a negative (curvilinear) effect on project performance. For high and very high performance, however, it is researcher quality in combination with a homogenous partner set (absence of partner diversity) that leads to the outcome. Substantially, we therefore conclude that in our case only researcher quality, user activity and to some extent partner diversity separately lead to performance in R&D projects with researcher quality as the most important condition. These findings imply that it is relatively separate strategies that are successful. One could suspect that the different strategies would be related to the project leader, i.e. project leaders consciously or unconsciously favour a certain way how to run these R&D projects. For example, one may be good in spotting and attracting very good young scientific talent (researcher quality) another may have more a talent for bringing in partner firms and getting and keeping them actively involved. Unfortunately, we could not corroborate from our qualitative data that the differences were tied to strategies of the project leader and we currently do not have an explanation for these findings. 27

28 What is very interesting theoretically, however, is that contrary to our initial assumption, core conditions form, if at all, relatively weak configurations with other conditions. This might indicate that key factors are only loosely coupled in forming configurations, i.e. form relatively weak INUS conditions. It is often assumed that conditions are tightly coupled to each other in a configuration, i.e. we portrait it, as if two conditions necessarily jointly occur to lead to an outcome. The results described above, might therefore indicate that there are cases where especially researcher quality has an individual main effect and then there are cases where it co-occurs either with the absence or presence of other factors. With regard to methodological insights we would like to discuss the following four issues: comparative advantages and disadvantages of correlational and set-theoretic methods, calibration of curvilinear effects in fsqca, robustness of results in fsqca and implications for large N fsqca. Comparative advantages and disadvantages of correlational and set-theoretic methods In analyzing the data, it became clear that the combination of all original conditions would not lead to any informative results. Further following an exploratory logic, regression helped in identifying possible core conditions and reducing the set of factors on the basis of which to run a new fsqca. This would have been neither possible with fsqca except for a lengthy trial and error procedure with changing sets of conditions. The fsqca on the basis of a reduced subset revealed interesting patterns that could not have been detected with correlational methods and clearly contribute to more advanced theorizing in terms of configurations and equifinality. Also the results produced by fsqca could not be reproduced through interaction terms in the linear regression. This is a clear advantage of set-theoretic methods, because it contributes to the formulation of viable policy recommendations as 28

29 managerial alternatives. We did however, not go as far as to really integrate QCA and regression as suggested by Fiss, Sharapov and Cronquist (2013), which could be a future step. From a set-theoretic perspective the selection of factors and construction of a sub set on the basis of results of a regression might be questionable since it is the assumption of the configurational approach that the existence of INUS conditions are a major possibility and should be accounted for accordingly. Interestingly, the selected subset did lead to informative results. However, it should be checked whether different subsets would lead to similar results, i.e. separate paths around core conditions as a characteristic of the data. Another issue regards the analysis of data in network settings and over time as was the case in this study. Correlational methods are based on the assumption that observations are independent of each other. The R&D projects analyzed in our case are partly connected to each other by overlapping membership (and presumably interaction) between project leaders and users, i.e. both can be members of multiple consortia. In addition, many projects are at least partial continuations of previous ones or developed out of previous ones. Therefore, one of the assumptions for regression is violated while that problem does not exist for settheoretic methods in a technical sense. fsqca therefore seems to be better suited as an analytic technique in circumstances of relative cohesive project environments. Proposition 1a: Correlational methods are more suitable than set-theoretical methods to distill a subset of empirically relevant conditions out of set of theoretically relevant conditions. Proposition 1b: Set-theoretical methods are more suitable than correlational methods to examine the configurations of (a pre-selected subset of) conditions that lead to an outcome and substantially contribute to advanced theorizing. 29

30 Calibration of curvilinear effects in fsqca Research has shown, that key variables in the explanation of the performance of teams and networks very often have a curvilinear or u shaped effect as for example diversity, technological distance (Gilsing et al. 2008) or the ratio between arm s length and embedded ties (Uzzi 1997). The question therefore arises, how to deal with this insight in an fsqca analysis. One of the possibilities we explored here, was to first check with correlational techniques whether any of the variables does indeed show such an effect. Then the data can be analyzed based on a specific calibration of this particular factor (user activity in our case). For that, two crossover points need to be determined in order to create separate sets for low values, medium values and high values and then one has to analyze to what extent the paths differ based on this distinction. In our case we could show that the paths differed between medium and high user activity. While medium user activity appeared to be a peripheral condition with researcher quality and partner diversity, high user activity was a negated condition jointly with the other two core conditions in two separate paths. Proposition 2: Conditions with a curvilinear effect on the outcome require calibration with not one but two cross-over points between membership and non-membership. Robustness of results in fsqca Similar to Skaaning (2011), we noticed the sensitivity of fsqca results to threshold choices during the analysis from the raw data to the fsqca results. We agree with Fiss, Sharapov and Cronquist (2013) that establishing the robustness of QCA results is a more important concern in large-n applications than in studies with a small N. As we demonstrated above, the frequency used for the cut off in the truth table has an effect on results, even though in our case the solution became more parsimonious and core conditions were the same. A similar assessment can be made for the choice of consistency thresholds. We followed the procedure to check the robustness as suggested by Schneider and Wagemann (2012: ) and could 30