Customer Acceptance Tests and New Product Success An Application of QCA in Innovation Research

Transcription

1 Customer Acceptance Tests and New Product An Application of QCA in Innovation Research Abstract We test the connection between Customer Acceptance Tests (CATs) and New Product (NPS) for the first time using Qualitative Comparative Analysis based on data from a European tool manufacturer. We link results from 41 CATs conducted between 2009 and 2013 to NPS as rated by Product Managers and show that (1) an overall product evaluation of seven on a scale from one to ten is a necessary condition for NPS, that (2) tool features related to durability, performance, design & comfort and weight are a necessary condition for a superior overall product evaluation and that (3) product advantage vs. competition is a necessary condition for NPS. Our results support Product Managers how to assess results of future CATs and help them to determine what a good or bad customer feedback is. Keywords: Track: QCA, Methodology, Innovation INNOVATION & NEW PRODUCT AND SERVICE DEVELOPMENTS 1

2 1 Introduction Developments of increased competitive pressure on a global scale, shortened development cycle times and hyper-differentiated product offers have created an environment in which, according to Apple CEO and Steve-Jobs-successor Tim Cook, the rule applies: To not innovate is to die. (Grobart, 2013). Mastering survival in this surrounding is both vital and difficult, as is illustrated by high flop rates of new products (Ernst, 2002). The activity of Customer Acceptance Testing (CAT) within Cooper s Stage-Gate process, being one approach to handle the task described, is a mean for ensuring the effectiveness of development efforts. This is achieved by testing prototypes for alignment of product features with customer requirement (Cooper, 1990). Despite the stage s theoretical importance and an obvious, intuitive link between Customer Acceptance and New Product (NPS), empirical literature on the predictive quality of CATs is scarce, resulting in the research questions: Is there an empirically observable connection between results of Customer Acceptance Tests and New Product? If there is, what are the product features that drive it? This paper s aim is to systematically research this issue by first providing a literature review on Customer Acceptance Testing and the causal factors of New Product (NPS). Based on this review, hypotheses regarding the relationship are derived and tested in the context of 41 innovation projects that took place at a European tool manufacturer. Thus, the empirical part starts with background on the QCA method, followed by information on the sample as well as methods applied for sample generation. In a fourth part, results of QCA are discussed. We close by drawing conclusions, pointing towards important limitations and naming managerial implications. To the best of our knowledge, Qualitative Comparative Analysis (QCA) was used for the first time trying to assess supposedly asymmetrical relationships in innovation research. 2 Literature Review 2.1 Customer Acceptance Testing Customer Acceptance Tests (CATs) are not part of Cooper s Stage-Gate process, but, taking a generic lens on the activity, a sub-form of product tests as they are commonly used in market research. In this type of product test, the quality of a prototype is tested subjectively by potential customers in order to make sure that product development efforts have moved the design and features towards customer requirements (Weis & Steinmetz, 2002). However, these product tests possess limited predictive validity regarding NPS, as several other factors to be discussed subsequently were shown to impact NPS as well. These factors are not controlled for in CATs, leaving amounts of NPS variance unexplained (Hammann & Erichson, 2004). Nevertheless, a strong argument suggesting a weak instead of an absent relationship is implied by the term Customer Acceptance Tests alone, as they capture customers acceptance towards a new product. Adoption behavior is out of reach to them, as they lack a quantitative and deterministic component (Tauber, 1975). Nevertheless, the notion remains that some trial and early repeat is a necessary condition to achieve some adoption (Tauber, 1975, p.71). This translates into the first hypothesis: H1: A positive overall evaluation in a Customer Acceptance Test is a necessary causal factor for New Product. 2

3 2.2 Antecedents to New Product Further factors that were shown to have an impact on NPS can be divided into two groups, depending on whether they are located within or outside of the firm. The group of external antecedents that exert influence on NPS engulfs the market environment in its full spectrum, e.g. the legal & political, etc. (Tsai, Huang, & Tsai, 2014). Furthermore, demand uncertainty seems to play a vital role, as it influences the way a company s strategy decision translates into economic success. Additionally, any competitor activity bears the potential to impact product introduction results in various ways. One may think of a rather competitor-oriented industry, such as the German food retail industry, in which any new offer to customers immediately is responded to by intense price wars (Gatignon & Xuereb, 1997). The list of internal factors names an open and innovation-friendly culture that animates peoples creativity (Voss, 1985), an organizational design avoiding silo-thinking through use of cross-functional teams (Graner & Mißler-Behr, 2014), a strategic fit between the New Product Development strategy, resources and markets (Gatignon & Xuereb, 1997), the mere presence of a structured process (Cooper, 1983), commitment through senior management providing resources (Balachandra, 1984) as well as the proficiency market activities are conducted with (Benedetto, 1999). Especially the relative product quality in general and the product s superiority regarding its features need to be emphasized, as they contribute to a greater perceived quality (Tsai et al. 2014). Thus, superiority of a whole product respectively its features against competitor products is found to have an impact on NPS (Calantone & Benedetto, 1988). Combined with Benedetto (1999), stating that relative product quality, amongst others, is an important antecedent of NPS, this translates into the second hypothesis: H2a: Product feature superiority is a necessary causal factor for a superior overall evaluation. With superior product quality contributing to NPS, H2a can be justifiably extended to H2b: Product feature superiority is a necessary causal factor for NPS. With product advantage as a whole strongly contributing to NPS (Cooper und Kleinschmidt 1987), too, hypothesis H3 reads as follows: H3: Product advantage, i.e. product superiority, is a necessary causal factor for NPS. 3 Empirical Research 3.1 Methodology To analyze the data, this study used QCA (Ragin, 2008), which builds upon the premise that relationships among variables can be interpreted as members of sets. To assess the set relations, causal conditions (here overall evaluations & physical features) and the outcome of interest (here NPS) have to be represented in terms of membership scores. QCA then describes cases that show desired values for the outcome in question by examining the degree to which causal conditions or configurations of these conditions are present. QCA links causal conditions and the outcome with relationships of necessity and sufficiency, with the first implying that a causal condition must be present for an outcome to occur and the latter implying that a causal condition (or a combination thereof) can bring about an outcome (Ragin, 2008). A condition is considered necessary or almost always necessary if consistency exceeds the threshold of.9 (e.g. Schneider & Eggert, 2014). Consistency of.85 can be interpret- 3

4 ed as partial support. Consistency here refers to the degree to which the empirical cases that share a causal condition agree in displaying the outcome. The measure of coverage focuses the amount of cases described correctly by a condition; given a high consistency, it should exceed.5 (Ragin, 2008). For necessity analysis, this study calculated consistency scores for each physical feature and examined whether any of these causal conditions are indispensable to account for a superior overall evaluation (H2a) or even NPS (H2b). QCA comes in two forms, with csqca coding set membership dichotomously as being either present or absent ( 1/0 ). The second form, fsqca, is able to reflect vagueness of memberships by introducing values between one and zero, indicating partial membership. These properties predispose csqca for the research of questions linked with matters of superiority, whereas fsqca can be used to account for vagueness inherent to terms. The value of.5, a breakpoint, in any case serves as a boundary to distinguish set affiliation. (Ragin, 2008). 3.2 Sample The sample for this study consisted of secondary data from 18 products that had been tested for customer acceptance in several markets. Two competitor products had always been included in the testing. 41 data sets of product-market-combinations emerged and were enriched by information from Marketing Managers regarding NPS. Markets in which the CAT projects were conducted were located in central and southern Europe, Great Britain, India, Japan, Russia and the United States. The time span of researched projects reached from 2009 to Not all product-market-combinations were rated on all categories, producing a sample size ranging from n=11 to n=41. Conveniently, in QCA this does not detract from the validity of results (Ragin, 2008). 3.3 Measures Original CAT data applied metric scales ranging from 1 to 10 to evaluate perceived quality, 1 rating the respective criterion worst, 10 rating it best. The item Overall, present in H1a and H1b, was operationalized using various breakpoints: Full (Non-)membership was set at values 5 and 10, with breakpoints being systematically lowered by.5 four times starting at 7.5. That way, the minimum score constituting a positive overall evaluation still able to maintain the relationship of necessity was identified. Overall evaluation superiority was coded 1 if co-tested products did not score higher; else 0 was marked. For H2a and H2b, 116 product criteria were rated (1 = poor quality, 10 = high quality), each product using different criteria. These were then grouped into 9 categories after 2 iterations of category development and revision with Marketing Managers. Table 1 depicts the results of the content analytical process. Mean scores were then calculated for all existing criteria falling in the respective categories. Categories then were used as causal factors for QCA. NPS was assessed following the recommendations by Hart (1993) and the typology of Griffin and Page (1996). Thus, items that measured NPS featured the achievement of market share, unit sales, profitability, sales volume, return on investment, all measured relative to the previously stated objectives as well as the overall success of the product (α =.974). A 5-point Likert scale was used with 1 indicating low achievement and 5 indicating high achievement of objectives. A mean across all items was calculated. For NPS, both csqca and fsqca scores set the breakpoint at 2.9 in order to express that projects rated 3 on the 1-5 scale were considered narrow successes. For hypothesis H3, product superiority was coded dichotomously (1 = product advantage present, 0 = product advantage absent). Category (Abbreviation) TABLE 1: Operationalization of Categories Operationalization 4

5 Overall (OE) Ease of Use (EOU) Health and Safety (H&S) Design and Comfort (DAC) Performance Indicators (PI) Durability (DUR) Balance (BAL) Weight (WGT) Size (SIZ) holistic evaluation of a prototype perceived degree to which tools allow intuitive, fail-safe operation & time-saving procedures perceived degree to which features intended to decrease risk of injury or reduction of stress caused by tool-usage are appreciated degree to which design elements and tool characteristics match ergonomic principles and allow a fatigue-proof working experience describe the perceived quality of a tool`s key features Tool appearance of being able to withstand outwear and material fatigue despite heavy and frequent usage perceived usefulness of tool mass distribution perceived usefulness of tool weight perceived usefulness of tool size 3.4 Results Results showed relationships of necessity for an overall evaluation of at least 7 on a scale from 1-10 for NPS. With regard to physical features of tools, design & comfort issues, performance indicators, tools durability and weight turned out to be necessarily superior to those of other tested tools in order to achieve a superior overall evaluation. This superior overall evaluation turned out to be a necessity for the achievement of respondents product preference. Product preference, in turn, appeared to be necessary to NPS, thereby linking constructs of superior overall evaluations, preferred products and NPS indirectly through necessity. None of the physical features was directly linked to NPS. Table 2 depicts the results. TABLE 2: Results from QCA Tests for Necessity of causal factors Hypothesis confirmed* Causal factor Outcome Consistency Coverage n H1 Superior Overall (Cs)New Product Positive Overall (Fs)New Product H2a Superior EOU Superior Overall Superior H&S Superior D&C Superior PI Superior DUR Superior BAL Superior WGT Superior SIZ H2b Superior EOU (Cs)New Product Superior H&S Superior D&C Superior PI Superior DUR

6 Superior BAL Superior WGT Superior SIZ H3 Preferred Product (Cs)New Product Ad hoc Superior Overall observation Preferred Product * : consistency below.85 : consistency between.85 and.9 : consistency above.9 4 Discussion Through the use of QCA methods, relationships of necessity shaping the path from positive overall evaluations over preferred products to NPS were identified. At the very heart of the process seem to lie the physical features of design & comfort, performance, weight and balance of the product, which can be considered necessities for positive overall evaluations. They should dominate co-tested competitor products in CAT projects. Furthermore, these insights were generated introducing the tool of QCA, allowing the detection of formerly undetectable relationships by working previously unworkable sample sizes. However, this paper bears limitations to be addressed subsequently. As products observed came from various product classes, product features varied largely. Thus, these features were classified according to product properties which potential buyers may ponder as relevant (depicted in Table 1). This ultimately led to comparison of features across product classes, which itself could be worthy of discussion. More so, research could suffer from informant bias, as respondents held the same hierarchical position within the company (Ernst, 2002). Of course, the described linkages through necessity leave room for further research regarding other factors of NPS. Thus, future efforts could focus factors named in the literature review section and identify further necessary conditions, ultimately creating a checklist of products for NPS. This could enable companies to reduce uncertainty in the process of picking potentially successful projects after the CAT stage. Last, a sample richer in variation could be used to identify combinations of project properties constituting sufficiency for NPS. In summary, our work contributes in several ways: First, we provide an example of how QCA can be systematically applied in the context of innovation research and put to use in the industry. From a managerial perspective, we find evidence that the value of 7 out of 10 in Customer Acceptance Testing is a threshold not to be underrun. In the tool industry, such evaluation appears to be driven by perceived superior quality of features related to design & comfort, performance, weight and durability. 5 References Balachandra, R. (1984). Critical Signals for Making Go/NoGo Decisions in New Product Development. Journal of Product Innovation Management 1(2), Benedetto, C. A. (1999). Identifying the Key Factors in New Product Launch. Journal of Product Innovation Management 16(6), Calantone, R. J.; & Benedetto, C. A. (1988). An Integrative Model of the New Product Development Process. An Empirical Validation. Journal of Product Innovation Management 5(3),

7 Cooper, R. G. (1983). The impact of new product strategies. Industrial Marketing Management 12(4), Cooper, R. G., & Kleinschmidt, E. J. (1987). New products: What separates winners from losers? Journal of Product Innovation Management 4(3), Ernst, H. (2002). Factors of New Product Development: A Review of the Empirical Literature. Journal of Management Reviews 4(1), Gatignon, H., & Xuereb, J.-M. (1997). Strategic Orientation of the Firm and New Product Performance. Journal of Marketing Research 34(1), Graner, M., & Mißler-Behr, M. (2014). Method Application In New Product Development and the Impact on Cross-Functional Collaboration and New Product. International Journal of Innovation Management 18(1), Griffin, A., & Page, A. L. (1996). PDMA success measurement project: Recommended measures for product development success and failure. Journal of Product Innovation Management 13(6), Grobart, S. (2013). Apple Chiefs Discuss Strategy, Market Share--and the New iphones (Retrieved from (Last accessed September 27, 2014). Hammann, P., & Erichson, B. (2004). Marktforschung. [market research] 5., neu bearb. Aufl. Stuttgart: Lucius und Lucius. (in German). Hart, S. (1993). Dimensions of success in new product development: An exploratory investigation. Journal of Marketing Management 9(1), Ragin, C. (2008). Redesigning social inquiry. Fuzzy sets and beyond. Chicago: University of Chicago Press. Schneider, M. R., & Eggert, A. (2014). Embracing Complex Causality with the QCA Method: An Invitation. Journal of Business Marketing Management 7(1), Tauber, E. M. (1975). Why Concept and Product Tests Fail to Predict New Product Results. Journal of Marketing 39(4), Tsai, K.-H.; Huang, C.-T. & Tsai, M.-L. (2014). Reviews of market drivers of new product performance: Effects and relationships. International Journal of Market Research 55(5), Voss, C. A. (1985). Determinants of in the Development of Applications Software. Journal of Product Innovation Management 2(2), Weis, H. C., & Steinmetz, P. (2002). Marktforschung. [market research]. 5., völlig überarb. und aktualisierte Aufl. Ludwigshafen (Rhein): Kiehl. (in German). 7