IEEE TRNSTIONS ON SYSTEMS, MN, ND YBERNETIS PRT Study into the Factor that Influence the Undertandability of Buine Proce Model Hajo. Reijer and Jan Mendling btract Buine proce model are key artifact in the development of information ytem. While one of their main purpoe i to facilitate communication among takeholder, little i known about the factor that influence their comprehenion by human agent. On the bai of a ound theoretical foundation, thi paper preent a tudy into thee factor. Specifically, the effect of both peronal and model factor are invetigated. Uing a quetionnaire, tudent from three different univeritie evaluated a et of realitic proce model. Our finding are that both type of invetigated factor affect model undertanding, while peronal factor eem to be the more important of the two. The reult have been validated in a replication that involve profeional modeler. Index Term Buine Proce Modeling, Proce model, Human information proceing, omplexity meaure. I. INTRODUTION SINE the 96, conceptual model are in ue to facilitate the early detection and correction of ytem development error []. In more recent year, the primary focu of conceptual modeling effort ha hifted to buine procee []. Model reulting from uch effort are commonly referred to a buine proce model, or proce model for hort. They are ued to upport the analyi and deign of, for example, proce-aware information ytem [], ervice-oriented architecture [4], and web ervice [5]. Proce model typically capture in ome graphical notation what tak, event, tate, and control flow logic contitute a buine proce. buine proce that i in place to deal with complaint may, for example, contain a tak to evaluate the complaint, which i followed by another one pecifying that the cutomer in quetion i to be contacted. Similar to other conceptual model, proce model are firt and foremot required to be intuitive and eaily undertandable, epecially in IS project phae that are concerned with requirement documentation and communication [6]. Today, many companie deign and maintain everal thouand proce model, often alo involving non-expert modeler [7]. It ha been oberved that uch large model collection exhibit eriou quality iue in indutry practice [8]. gaint thi background it i problematic that little inight exit into what influence the quality of proce model, in particular with repect to their undertandability. The mot important inight i that the ize of the model i of notable Hajo. Reijer i with the School of Indutrial Engineering, Eindhoven Univerity of Technology, Eindhoven, The Netherland, e-mail: h.a.reijer@tue.nl. Jan Mendling i with the School of Buine and Economic, Humboldt- Univerität zu Berlin, Germany, e-mail jan.mendling@wiwi.hu-berlin.de Manucript ubmitted pril 9, 9. impact. n empirical tudy provide evidence that larger, realworld proce model tend to have more formal flaw (uch a e.g. deadlock or unreachable end tate) than maller model [9]. likely explanation for thi phenomenon would be that human modeler looe track of the interrelation in large and complex model due to their limited cognitive capabilitie (cf. []). They then introduce error that they would not inert in a mall model, which will make the model le effective for communication purpoe. There i both an academic and a practical motivation to look beyond thi inight. To tart with the former, it can be virtually ruled out that ize i the ole factor that play a role in undertanding a proce model. To illutrate, a purely equential model will be eaier to undertand than a model that i imilar in ize but where tak are interrelated in variou way. Thi raie the interet into the other factor that play a role here. more practical motivation i that model ize i often determined by the modeling domain or context. So, proce modeler will find it difficult to affect the ize metric of a proce model toward creating a better readable verion: They cannot imply kip relevant part from the model. The aim of thi paper i to invetigate whether factor can be determined beyond the ize of proce model that influence it undertanding. In that repect, we ditinguih between model factor and peronal factor. Model factor relate to the proce model itelf and refer to characteritic uch a a model denity or tructuredne. Peronal factor relate to the reader of uch a model, for example with repect to one educational background or the perception that are held about a proce model. While inight are miing into the impact of any of uch factor beyond ize on proce model undertandability, reearch ha uggeted the importance of imilar factor in other conceptual data model [], []. To invetigate the impact of peronal and model factor, the reearch that i preented here take on a urvey deign. Uing a quetionnaire that ha been filled out by 7 tudent from three different univeritie, hypothetical relation between model and peronal factor on the undertanding of a et of proce model are invetigated. Some exploratory finding from thi data are reported in [], [4], which eentially confirm the ignificance of the two type of factor. The contribution of thi paper i quite different. We develop a ound theoretical foundation for dicuing individual model undertanding that i rooted in cognitive reearch on computer program comprehenion. We ue thi foundation to etablih hypothee on the connection between undertandability on the one hand, and peronal and model factor on the other hand. For thee tet we ue the before mentioned urvey
IEEE TRNSTIONS ON SYSTEMS, MN, ND YBERNETIS PRT data. Beyond that, we provide an extenive validation of our finding and our intrument addreing two major challenge. Firt, there ha been little reearch on contruct validity of undertandability meaure. We ue ronbach alpha to check the conitency of our quetion that are ued in calculating the undertandability core. Furthermore, we addre potential threat to external validity. We report on a replication of our urvey with practitioner and invetigate if the reult differ from that of the tudent. The ret of thi paper i organized in accordance with the preented aim. Section II introduce the theoretical foundation of proce model undertanding. We identify matter of proce model undertanding and repective challenge. Thi lead u to factor of undertanding. Section III decribe the etup of our urvey deign and the motivation behind it. Section IV then preent the data analyi and the interpretation. Section V dicue threat to validity and how we addreed them. Section VI conclude the article. We ue ppendix to ummarize our urvey deign. Start OR OR I B J ND K N II. BKGROUND Thi ection introduce the theoretical background of our empirical reearch. Section II- give a brief overview of the information content of a proce model, define a notion of undertandability, and ummarize related work on proce model undertanding. Section II-B invetigate potential factor of undertandability. We utilize inight from cognitive reearch into computer program comprehenion in order to derive propoition about the ignificance of peronal and model factor for undertanding. L ND M O. Matter of Proce Model Undertanding Before conidering a notion of undertandability we firt have to dicu matter that can be undertood from a proce model. We are focuing on o-called activity-baed or controlflow-baed proce model (in contrat to goal-oriented [5] and choreography-oriented language [6]). Figure how an example of uch a proce model in a notation that we will ue throughout thi paper. Thi notation eentially cover the commonalitie of Event-driven Proce hain (EP) [7], [8] and the Buine Proce Modeling Notation (BPMN) [9], which are two of the mot frequently ued notation for proce modeling. Such a proce model decribe the control flow between different activitie (, B, I, J, K, L, M, N, and O in Figure ) uing arc. So-called connector (, ND, OR) define complex routing contraint of plit (multiple outgoing arc) and join (multiple ingoing arc). -plit repreent excluive choice and -join capture repective merge without ynchronization. ND-plit introduce concurrency of all outgoing branche while ND-join ynchronize all incoming arc. OR-plit define incluive choice of a -toall fahion. OR-join ynchronize uch multiple choice, which require a quite ophiticated implementation (ee [8], []). Furthermore, there are pecific node to indicate tart and end. In thi paper we conider formal tatement that can be derived about the behavior decribed by uch a proce model, ignoring the (informal) content of activity label. Thi formal Fig.. Part of a proce model focu ha the advantage that we can unambiguouly evaluate whether an individual ha graped a particular model apect correctly. In particular, we focu on binary relationhip between two activitie in term of execution order, excluivene, concurrency, and repetition. Thee relationhip play an important role for reading, modifying, and validating the model. Execution Order relate to whether the execution of one activity a i eventually lead the execution of another activity a j. In Figure, the execution of J lead to the execution of L. Excluivene mean that two activitie a i and a j can never be executed in the ame proce intance. In Figure, J and K are mutually excluive. The concurrency relation cover two activitie a i and a j if they can potentially be executed concurrently. In Figure, L and M are concurrent. ingle activity a i called repeatable if it i poible to execute it more than once for a proce intance. In Figure, among other, K, N, and I are repeatable. Statement uch a Executing activity a i implie that a j will be executed later can be eaily verified uing the reachability graph of the proce model. reachability graph capture all tate and tranition repreented by the proce
IEEE TRNSTIONS ON SYSTEMS, MN, ND YBERNETIS PRT model and it can be (automatically) generated from it. For ome clae of model, everal relationhip can be calculated more efficiently without the reachability graph []. For intance, thee relation can be contructed for thoe proce model that map to free-choice Petri net in O(n ) time [], []. B. Factor of Proce Model Undertanding Throughout thi paper, we ue the term undertandability in order to refer to the degree to which information contained in a proce model can be eaily undertood by a reader of that model. Thi definition already implie that undertandability can be invetigated from two major angle: peronal factor related to the model reader and factor that relate to the model itelf. We dicu the relevance of both categorie uing the cognitive dimenion framework a a theoretical foundation. The cognitive dimenion framework i a et of apect that have empirically been proven to be ignificant for the comprehenion of computer program and viual notation [4]. There are two major finding that the framework build upon: repreentation alway emphaize a certain information at the expene of another one, and there ha to be a fit between the mental tak at hand and the notation [5], [6]. The implication of thee inight are reflected by cognitive dimenion that are relevant for proce model reading. btraction Gradient refer to the grouping capabilitie of a notation. In a ingle proce model, there i no mechanim to group activitie. Therefore, flow language are called abtraction-hating [4]. a conequence, the more complex the model get the more difficult it become for the model reader to identify thoe part that cloely relate. Preumably, expert model reader will be more efficient in finding the related part. Hard Mental Operation relate to an over-proportional increae in difficulty when element are added to a repreentation. Thi i indeed the cae for the behavioral emantic of a proce model. In the general cae, calculating the reachability graph for a proce model i NP-complete [7]. Therefore, a larger proce model i over-proportionally more difficult to interpret than a imple model. On the other hand, expert are more likely to know decompoition trategie, e.g. a decribed in [8], to cope with complexity. Hidden Dependencie refer to interdependencie that are not fully viible. In proce model uch hidden dependencie exit between plit and join connector: each plit hould have a matching join connector of the ame type, e.g. to ynchronize concurrent path. In complex model, ditant plit-join pair can be quite difficult to trace. In general uch interdependencie can be analyzed uing the reachability graph, but many analye can be performed alo tructurally (ee [9]). Expert modeler tend to ue tructural heuritic for invetigating the behavior of a proce model. Secondary Notation refer to any piece of extra information that i not part of the formalim. In proce model econdary notation i an important matter, among other in term of labeling convention [] or layout trategie []. For model of increaing complexity, econdary notation alo gain in importance for making the hidden dependencie better viible. On the other hand, it ha been hown that expert performance i le dependent on econdary notation a that of novice []. Peronal factor have alo been recognized a important factor in engineering and deign [], [4]. In particular, the matter of expertie i clearly etablihed by prior reearch on human-computer interaction. While reearch on perceptual quality and perceptual expertie i only emerging recently in conceptual modeling (ee [5], [6]), there are ome trong inight into the factor of expert performance in different area. level of profeional expertie i aumed to take at leat, to 5, hour of continuou training [7, p.56]. In thi context, it i not only important that the expert ha worked on a certain matter for year, but alo that practicing ha taken place on a daily bai [8]. Such regular training i needed to build up experience, knowledge, and the ability to recognize pattern [9]. Furthermore, the way information i proceed by human i influenced by cognitive tyle, which can be related to peronality. There are peron who prefer verbal over image information and who rather grap the whole intead of analytically decompoing a matter, or the other way round [4]. model enable reaoning through viualization, perceptional capabilitie of a peron are alo relevant [4]. learly, thee capabilitie differ between peron with different proce modeling expertie. We conclude for thi theoretical dicuion that model feature and peronal characteritic are indeed likely to be relevant factor of proce model undertandability.. Related work In thi ection we preent related work grouped into three categorie: model factor, peronal factor, and other factor. The importance of model characteritic wa intuitively aumed by early work into proce model metric. Such metric quantify tructural propertie of a proce model, inpired by prior work in oftware engineering on line of code, cyclomatic number, or object-oriented metric [4] [44]. Early contribution by Lee and Yoon, Nien, and Moraca [45] [47] focu on defining metric. More recently, different metric have been validated empirically. The work of ardoo i centered around an adaptation of the cyclomatic number for buine procee he call control-flow complexity (F) [48]. Thi metric wa validated with repect to it correlation with perceived complexity of proce model [49]. The reearch conducted by a group including anfora, Rolón, and García analyze undertandability a an apect of maintainability. They include different metric of ize, complexity, and coupling in a et of experiment, and identify everal correlation [5], [5]. Further metric take their motivation from cognitive reearch, e.g. [4], and baed on concept of modularity, e.g. [5], [5]. Mot notably, an extenive et of metric ha been validated a factor of error probability [9], a ymptom of bad undertanding. The different validation clearly how that ize i an important model factor for undertandability, but doe
IEEE TRNSTIONS ON SYSTEMS, MN, ND YBERNETIS PRT 4 not fully determine phenomenon of undertanding: additional metric like tructuredne help to improve the explanatory power ignificantly [8]. Peronal factor have been le intenively reearched a factor of proce model undertanding. The experiment by Recker and Dreiling operationalize the notion of proce modeling expertie by a level of familiarity of a particular modeling notation [54]. In a urvey by Mendling, Reijer, and ardoo participant are characterized baed on the number of proce model they created and the year of modeling experience they have []. Mendling and Strembeck meaure theoretical knowledge of the participant in another urvey uing ix ye/no quetion [55]. Mot notable are two reult that point to the importance of theoretical proce modeling knowledge. In the Mendling, Reijer, and ardoo urvey the participant from TU Eindhoven with trong Petri net education cored bet and in the Mendling and Strembeck urvey, there wa a high correlation between theoretical knowledge and the undertandability core. There are other factor that alo might have an impact on proce model undertanding. We briefly dicu model purpoe, problem domain, modeling notation, and viual layout. Model purpoe: The undertanding of a model may be affected by the pecific purpoe the modeler had in mind. The bet example i that ome proce model are not intended to be ued on a day-to-day bai by people but intead are explicitly created for automatic enactment. In uch a cae, le care will be given to make them comprehenible to human. The difference between proce model a a reult of different modeling purpoe are mentioned, for example, in [6]. Empirical reearch into thi factor i miing. Problem domain: People may find it eaier to read a model about the domain they are familiar with than other model. While thi ha not been etablihed for proce model, it i known from oftware engineering that domain knowledge affect the undertanding of particular code [56]. Modeling notation: In the preence of many different notation for proce model, e.g. UML ctivity diagram, EP, BPMN, and Petri net, it cannot be ruled out that ome of thee are inherently more uitable to convey meaning to people than other. Empirical reearch that ha explored thi difference i, for example, reported in [57]. ccording to thee publication, the impact of the notation being ued i not very high, maybe becaue the language are too imilar. Other reearch that compare notation of a different focu identify a ignificant impact on undertanding [58], [59]. Viual layout: Semantically equivalent model can be arranged in different way. For example, different line drawing algorithm can be ued or model may be plit up into different ubmodel. The effect of layout on proce model undertanding wa already noted in the early 99 [6]. With repect to graph, it i a well-known reult that edge croe negatively affect undertanding [6]. lo, for proce model, the ue of modularity can improve undertanding [6]. Given that, a we argued, the inight into the undertanding of proce model are limited, thi i probably not a complete et of factor. But even at thi number, it would be difficult to invetigate them all together. In thi tudy, we retrict ourelve to the firt two categorie, i.e. peronal and model factor. In the definition of thi urvey, which will be explained in the next ection, we will dicu how we aimed to neutralize the potential effect of the other categorie. D. Summary From cognitive reearch into program undertanding we can expect that peronal and model factor are likely to be factor of proce model undertandability. The impact of ize a an important metric ha been etablihed by prior reearch. Yet, it only partially explain phenomena of undertanding. Peronal factor alo appear to be relevant. Theoretical knowledge wa found to be a ignificant factor, but o far only in tudent experiment. Furthermore, reearch into the relative importance of peronal and model factor are miing. In the following ection, we preent a urvey to invetigate thi quetion and analyze threat to validity. III. DEFINITION, PLNNING, ND OPERTION OF THE SURVEY DESIGN Thi ection explain the definition, the planning and the operation of a urvey deign in peronal and model related factor of undertanding.. Definition ccording to the theoretical background we provided, both the characteritic of the reader of a proce model and thoe of the proce model itelf affect the undertanding that uch a reader may gain from tudying that model. Both type of characteritic can be conidered a independent variable, while the undertanding gained from tudying a proce model contitute the dependent variable. Beyond thi, there are other potential factor of influence which we wih to neutralize, i.e. model purpoe, problem domain, modeling notation, and viual layout (ee Section II-). To explore the relation that interet u, the idea i to expoe a group of repondent to a et of proce model and then tet their undertanding of thee model uing a elf-adminitered quetionnaire. Such a deign hare characteritic with a urvey where peronal and model parameter are recorded, but without predefined factor level. We ue a convenience ample of tudent. From the analyi perpective it can be claified a a correlational tudy that eek to identify tatitical connection between meaurement of interet. Similar deign have been ued to invetigate metric in oftware engineering, e.g. in [6]. oncluion on cauality are limited, though. We trived to neutralize the influence of other relevant factor. Firt of all, a et of proce model from practice wa gathered that wa pecifically developed for documentation purpoe. Next, to eliminate the influence of domain knowledge all the tak label in thee proce model were replaced by neutral identifier, i.e. capital letter to W. In thi way, we alo prevent a potential bia temming from varying length of natural activity label (ee [55]). Baed on the obervation by [57] that EP appear to be eaier to undertand than Petri net, we choe for an EP-like notation without event.
IEEE TRNSTIONS ON SYSTEMS, MN, ND YBERNETIS PRT 5 The participant received a hort informal decription of the emantic imilar to [64, p. 5]. Finally, all model were graphically preented on one page, without ue of modularity, and drawn in the ame top-to-bottom layout with the tart element at the top and end element at the bottom. Furthermore, in our exploration we wih to exclude one particular proce model characteritic, which i ize. we argued in the introduction of thi paper and our dicuion of related work, proce model ize i the one model characteritic of which it impact on both error pronene and undertanding i reaonably well undertood. Becaue it i our purpoe to look beyond the impact of thi particular apect, we have controlled the number of tak in the proce model. Each of the included proce model ha the ame number of tak. However, to allow for variation acro the other model characteritic, two additional variant were contructed for each of the real proce model. The variation were etablihed by changing one or two routing element in each of thee model (e.g. a particular -plit in a ND-plit). Having taken care of the variou factor we wih to control, at thi point we can refine what peronal and model factor are taken into account and how thee are meaured in the quetionnaire. Note that a ummary of the quetionnaire i preented in ppendix. For the peronal factor, we take the following variable into conideration: THEORY: peron theoretical knowledge on proce modeling. Thi variable i meaured a a elf-aement by the repondent on a 5-point ordinal cale, with anchor point I have weak theoretical knowledge and I have trong theoretical knowledge. PRTIE: peron practical experience with proce modeling. Thi variable i a elf-aement by the repondent. It i meaured on a 4-point ordinal cale. The cale ha anchor point I never ue buine proce modeling in practice and I ue buine proce modeling in practice every day. EDUTION: peron educational background. Thi categorical variable refer to the educational intitute that the repondent i regitered at. For the model factor, everal variable are included. Thee variable are all formally defined in [8, pp. 7-8], with the exception of the cro-connectivity metric that i pecified in [4]. The model factor can be characterized a follow: #NODES, #RS, #TSKS, #ONNETORS, #ND- SPLITS, #ND-JOINS, #-SPLITS, #-JOINS, #OR-SPLITS, #OR-JOINS: Thee variable all relate to the number of a particular type of element in a proce model. Thee include count for the number of arc (#RS) and node (#NODES). The latter can be further ubdivided into #TSKS on the one hand and #ONNETORS on the other. The mot pecific count are ubcategorization of the different type of logical connector, like #ND-SPLITS and #OR-JOINS. DIMETER: The length of the longet path from a tart node to an end node in the proce model. TOKEN SPLITS: The maximum number of path in a proce model that may be concurrently initiated through the ue of ND-plit and OR-plit. VERGE ONNETOR DEGREE, MXIMUM ONNE- TOR DEGREE: The VERGE ONNETOR DEGREE expree the average of the number of both incoming and outgoing arc of the connector node in the proce model; the MXIMUM ONNETOR DEGREE expree the maximum um of incoming and outgoing arc of thoe connector node. ONTROL FLOW OMPLEXITY: weighted um of all connector that are ued in a proce model. MISMTH: The um of connector pair that do not match with each other, e.g. when an ND-plit i followed up by an OR-join. DEPTH: The maximum neting of tructured block in a proce model. ONNETIVITY, DENSITY: While ONNETIVITY relate to the ratio of the total number of arc in a proce model to it total number of node, DENSITY relate to the ratio of the total number of arc in a proce model to the theoretically maximum number of arc (i.e. when all node are directly connected). ROSS-ONNETIVITY: The extent to which all the node in a model are connected to each other. SEQUENTILITY: The degree to which the model i contructed of pure equence of tak. SEPRBILITY: The ratio of the number of cut-vertice on the one hand, i.e. node that erve a bridge between otherwie diconnected component, to the total number of node in the proce model on the other. STRUTUREDNESS: The extent to which a proce model can be built by neting block of matching plit and join connector. ONNETOR HETEROGENEITY: The extent to which different type of connector are ued in a proce model. To illutrate thee factor, we refer the reader to Figure. Shown here i a model of a loan requet proce expreed in the EP modeling notation, which i elaborated in [8, pp. 9- ]. In addition to the tandard EP notational element, tag are added to identify equence arc, cut vertice, and cycle node. dditionally, the number of incoming and outgoing arc are given for each node, a well a a bold arc that provide the diameter of the model. ll thee notion are intrumental in calculating the exact value of the model factor that were preented previouly. For thi particular model, the value of the model factor are given in Table I. Having dicued the independent variable, we need to addre now how a proce model undertanding i captured. There are variou dimenion in how far comprehenion can be meaured, for an overview ee [65]. For our reearch, we focu on a SORE variable. SORE i a quantification of a repondent accurate undertanding of a proce model. Thi ratio i determined by the et of correct anwer to a et of even cloed quetion and one open quetion. The cloed quetion confront the repondent with execution order, excluivene, concurrency, and repeatability iue (ee Section II-) which are linked to cloed quetion (ye/no/i don t
IEEE TRNSTIONS ON SYSTEMS, MN, ND YBERNETIS PRT 6 Fig.. poitive client aement negative rik aement check client aement negative client aement reject loan requet loan requet i rejected loan i requeted record loan requet requet i recorded conduct rik aement poitive rik aement et up loan contract loan contract i et up ign loan contract loan contract i completed Sample proce model to illutrate the model factor Sequence arc ut Vertex / rticulation point ycle node Diameter in In-depth, out-depth out Reducible by tructured rule requeter i new client analyze requirement requirement are analyzed offer further product TBLE I METRIS VLUES FOR THE SMPLE PROESS MODEL IN FIGURE #NODES 7 V. ONN. DEGREE #RS 9 MX. ONN. DEGREE #TSKS 8 ONTROL FLOW OMPLEXITY 8 #ONNETORS 8 MISMTH 8 #ND-SPLITS DEPTH #ND-JOINS ONNETIVITY.74 #-SPLITS DENSITY.4 #-JOINS ROSS-ONNETIVITY.65 #OR-SPLITS SEQUENTILITY.45 #OR-JOINS SEPRBILITY.44 DIMETER 4 STRUTUREDNESS.556 TOKEN SPLITS ONNETOR HETEROGENEITY.89 know). The remaining quetion i open; a repondent i aked to identify and decribe any model problem, if the repondent feel that any uch problem exit. While the cloed quetion can add point (wrong anwer) or point (correct anwer) to SORE, the open quetion can add point (wrong anwer), point (partially correct anwer), or point (completely correct anwer). uch, the SORE value for any model evaluation by a repondent may range between and 9. Finally, our expectation on how the variou independent variable (peronal and model factor) affect the dependent variable (proce model undertanding), can now be decribed a follow. For the peronal factor, more theoretical knowledge or practical experience with repect to proce modeling i likely to poitively affect a peron undertanding of a proce model; le of thee factor have the oppoite effect. Furthermore, becaue all involved repondent received a proce modeling education at an academic level and tudent were not expected to have any extenive practical experience with proce modeling, we did not expect that the exact educational background would have any affect on a peron undertanding of a proce model. Thi et of expectation can be ummarized a hypothei H: The more the peron can be regarded to be an expert, the better will be hi or her undertanding of a model. Model factor have been hypotheized to have notable effect on their undertanding, ee [4], [8] for the related dicuion. In hort, the higher a proce model equentiality, eparability, or tructuredne the eaier it i to undertand uch a model; lower value have the oppoite effect. Similarly, undertandability of a proce model will alo increae by a lower number of node, arc, tak, and connector regardle of it kind on the one hand, or lower value for it diameter, connectivity, denity, token plit, average connector degree, maximum connector degree, mimatch, depth, control flow complexity, connector heterogeneity, and croconnectivity on the other. Higher value of thee model factor will have the oppoite effect. Thi et of expectation can be ummarized a hypothei H: The more complex the model i, the le it will be undertood. B. Planning and operation The urvey wa conducted in three phae. Firt, we collected a et of proce model from practice ued for documentation purpoe. From thi et, we originally elected eight that had an equivalent number of tak (5), applied the uniform layout to each of them, and then contructed two additional variant for each of thee. We then developed cloed quetion related to execution order, excluivene, concurrency and repeatability iue for each of the proce model. We alo included one open quetion that wa the ame for each model, i.e. If there i any problem with thi proce (e.g. proper completion, deadlock, etc.), pleae decribe it.. The correct anwer for all thee quetion were determined with the EP analyi tool introduced in [66]. Thi tool wa alo ued to calculate the et of proce model metric that we have decribed in Section III. For the firt verion of the quetionnaire, we conducted a pre-tet at Eindhoven Univerity of Technology, involving 5 taff member and 7 Ph.D. tudent of the Information Sytem group. The pretet led to a reduction of the model et to proce model, i.e. four proce model familie, and a reformulation of ome quetion. We dropped the more imple model to prevent fatigue. Second, we created ix verion of the quetionnaire, each with a different randomized order of the bae model and it variant. The purpoe wa to eliminate learning effect throughout the anwering. The quetionnaire wa filled out in cla etting at the Eindhoven Univerity of Technology, the Univerity of Madeira, and the Vienna Univerity of Economic and Buine dminitration by 7 tudent in total (ee Table II). Thi led to a total of 847 complete model evaluation out of a theoretic maximum of 876 (= 7 tudent x model). t that point in time, tudent were
IEEE TRNSTIONS ON SYSTEMS, MN, ND YBERNETIS PRT 7 TBLE II PRTIIPNTS IN THE SURVEY TU Eindhoven Uni Madeira WU Vienna tudent tudent tudent graduate level under-graduate level under-graduate level Petri net Petri net and EP EP following or completing coure on proce modeling at thee intitution. Participation wa voluntarily. The motivation for the tudent wa the fact that they felt to be in a competitive ituation with tudent from other univeritie, and that the quetionnaire could be ued a a good exam preparation. We captured the intitution with the categorial variable EDU- TION a the tudy program differed. Eindhoven tudent had been taught about oundne [67] (a general correctne criterion for workflow net), reachability graph, and related concept. Thee concept can be expected to help anwering the quetion of the urvey. Moreover, Eindhoven tudent were at the graduate level while the tudent from Madeira and Vienna were till in their third year of undergraduate tudie. The Eindhoven tudent were trained on Petri net, the Vienna tudent in EP, and the Madeira tudent had knowledge of both the Petri net and EP (ee Table II). The anwer were coded and analyzed uing the tatitic oftware package SPSS and STTGRPHIS. While the correct anwer to the cloed quetion could be counted automatically, all anwer to the open quetion were evaluated by both author on the bai of conenu. To determine the reliability of our meauring intrument of undertandability, namely SORE, we determined ronbach alpha for each of the proce model familie quetion et, leading to value ranging between.675 and.87. Thee value are conidered a acceptable, and comparable to other quetionnaire ued in the context of buine proce oriented reearch, ee e.g. [68]. Third, to validate our finding we repeated the urvey with a group of 8 profeional working within one of the world larget mobile telecommunication companie. t the time of the validation, all thee people were working at the Dutch headquarter of thi company in a unit concerned with upporting operational improvement project; proce modeling i one of their daily activitie. Their participation wa voluntary and wa part of an in-houe mater cla on Buine Proce Management, a provided by the author. While pot-graduate tudent (like the one participating in our tudy) have been found to be adequate proxie for analyt with low to medium expertie level [65], [69], thi validation i of importance conidering the inecure external validity of tudent experiment and urvey in information ytem reearch (ee [7]). IV. DT NLYSIS ND INTERPRETTION Thi ection preent the data analyi and interpretation tarting with dicuing apect of undertanding, continue with peronal and model factor, and cloe with an aement of their relative importance.. Data Exploration Firt of all, we teted for a normal ditribution of the SORE variable. The Kolmogorov-Smirnov tet howed that the normality aumption doe not hold (P-value =.). Therefore, tandard NOV technique are not applicable for the hypothei tet. Where poible, we will rely in the remainder on the non-parametric Krukal-Walli tet, which i an analyi of variance by rank. It i accepted a an alternative to NOV in cae the conidered variable i not normally ditributed [7]. Fig.. Boxplot of SORE per model more detailed undertanding of the ditribution of SORE can be gained from the boxplot in Figure. They repreent it ditribution for each tudent per model. Some imilar pattern emerge for all the model with the exception of model L: () the median SORE equal either 7 or 8, () 5% of the SORE value alway lie between 6 and 9, and () variou outlier, ome of them extreme, occur toward the lower end of the cale (thee are tagged with the repondent identifier). To tet for any tatitical difference in SORE acro the model we applied the Krukal-Walli tet at a 95% confidence level [7], [7]. When all model are compared with thi tet excluding model L, no ignificant difference between the model can be oberved with repect to SORE (P-value =.57). Thi confirm that the repondent undertanding i comparable acro all model, with the exception of model L. While model J, K, and L tem from the ame family, there i a variation a diplayed in Figure 4. The econd logical routing element from the top ditinguihe the three model from each other. For model L thi i an -plit routing element, for model J and K an ND-plit and ORplit repectively. The variation in SORE eentially tem from two quetion relating to thi part, which got few correct anwer for model L ( ): If T i executed for a cae, can U be executed for the ame cae?, and Note that the complete model L can be found in ppendix.
IEEE TRNSTIONS ON SYSTEMS, MN, ND YBERNETIS PRT 8 ND OR O P Q O P Q O P Q S T U S T U S T U W W W OR M OR M OR M End End End Fig. 4. Fragment of proce model J, K, and L (from left to right) an T, M, and O all be executed for the ame cae? Figure 4 how that the ditinguihing connector (NDplit and OR-plit) directly allow for the interleaved execution of T and U. But even for L the rightmot model in Figure 4 it i poible that T and U will be executed for the ame cae. However, thi can only happen after a cycle through M. It eem plauible that thi i overlooked by many repondent. Similarly, with repect to the econd quetion, many repondent may have failed to ee that T, M, and O can be executed in the rightmot model (which i clearly alo poible in the other two model). Thi initial analyi provide u with two important inight. In the firt place, the lack of ignificant difference in SORE acro mot model potentially point to the fact that model ize i indeed a primary factor that impact on model undertandability. The number of tak for all the model i, by deign, exactly the ame, and o are the level of undertanding of thee model. Furthermore, our detailed analyi of the exceptional model how that the change of a ingle element can have a ignificant impact on a model undertandability. So, depite the potentially dominant impact of ize, the earch for the additional impact factor eem indeed relevant, which i in line with the expectation of thi reearch (ee Section II-B). B. Peronal factor In thi ection we operationalize H a follow: There i no effect of each predictor variable THEORY, PRTIE, and EDUTION on the expected predicted variable SORE. The alternative hypothei tate: Some predictor variable do have an effect on the expected predicted variable SORE. Before we undertook our experiment, we had no reaon to expect difference in SORE between repondent with different academic background. ll repondent had received at leat a baic training in the ue of proce modeling technique at the time they took the quetionnaire. lo, the expoure to proce modeling in practice would be negligible for all involved repondent. To tet the abence of uch a difference, we computed the average SORE over the model. In Figure 5, the ditribution of the average SORE a gained by the repondent from the different univeritie i hown a boxplot. Fig. 5. Boxplot of average SORE for different value of EDUTION If no difference would exit between the three ditribution of total SORE, tudent can be aumed to perform imilarly acro the three univeritie. To tet thi, we again applied the non-parametric Krukal-Walli tet, becaue application of the Kolmogorov-Smirnov tet indicate that with a 95% confidence average SORE i not normally ditributed for any univerity. ontrary to our expectation, the application of the Krukal- Walli tet doe indicate that there i a tatitically ignificant difference among the ditribution at a 95% confidence level for the different type of education (P-value =.). In other word, difference exit in the ability of repondent to anwer quetion correctly acro the three univeritie. dditional pairwie Mann-Whitney tet [7] were conducted, taking into account the appropriate Bonferroni adjutment to control for Type error due to the three additional tet, i.e. uing a tricter alpha level of.7 (=.5/). Thee tet indicate that repondent from TU/e (Eindhoven) perform ignificantly better than repondent from the univeritie of Madeira (Pvalue =.) and Vienna (P-value =.), although the difference between the repondent from the univeritie of Madeira (UMa) and Vienna (WUW) i not ignificant (P-value
IEEE TRNSTIONS ON SYSTEMS, MN, ND YBERNETIS PRT 9 TBLE III PERSONL FTORS: KRUSKL-WLLIS NLYSIS OF DIFFERENES IN UNDERSTNDING (SORE) peronal factor P-value THEORY.9 PRTIE. EDUTION. ignificant at 9% confidence level, at 95%, at 99% =.7). retropective analyi of the coure offered at the variou univeritie revealed that the hour pent on actual modeling i the highet in Eindhoven, which may explain the noted difference. In particular, Eindhoven tudent have been explicitly and thoroughly taught about oundne [67], a general correctne criterion for workflow net, reachability graph, and related concept. n alternative explanation i that Eindhoven tudent are graduate tudent while the tudent from Madeira and Vienna are till in their third year of undergraduate tudie. But note that thi too indicate that a difference in theoretical knowledge i important. Interetingly, acro the different univeritie different modeling technique are taught. The Eindhoven tudent were trained in workflow net (baed on the Petri net formalim), the Vienna tudent in EP, and the Madeira tudent had knowledge of both the Petri net formalim and EP. So, interetingly, the choice of our EP-like notation doe not obviouly favor tudent who are familiar with EP. The analyi with the Krukal-Walli tet of the other invetigated peronal factor, THEORY and PRTIE, doe not identify any tatitically ignificant difference with repect to SORE. The outcome of the Krukal-Walli tet for all peronal factor conidered i ummarized in Table III. It hould be noted that THEORY and PRTIE both rely on elfaement where EDUTION can be determined objectively. With repect to the latter factor, we can conclude that it i potentially an important factor of influence in the undertanding of proce model, perhap hinting at the effect of difference in theoretical background in undertanding proce modeling. The particular notation on which one receive training doe not eem to be of any importance, but rather we would ugget that the knowledge of abtract proce modeling notion doe. ltogether, our analyi provide upport for hypothei H that the more the peron can be regarded to be an expert, the better will be hi or her undertanding of a model.. Model factor In thi ection we operationalize H a follow: There i no effect of a predictor variable capturing proce model complexity on the expected predicted variable SORE. The alternative hypothei tate: Some of thee predictor variable do have an effect on the expected predicted variable SORE. To determine whether model factor, a decribed in Section III-, are helpful to explain variation in model undertandability, we ued the model average SORE (ee Figure ) and determined Pearon correlation coefficient with all potential factor. Recall that the NOV tet i TBLE IV MODEL FTORS: ORRELTION NLYSIS WITH UNDERSTNDING (SORE) model factor corr.coeff. P-value #OR JOINS -..95 DENSITY -.68. V. ONNETOR DEGREE -.674.6 MISMTH -.48.54 ONNETOR HETEROGENEITY -.. ROSS-ONNETIVITY -.549.65 ignificant at 9% confidence level, at 95%, at 99% not applicable becaue of the non-normal ditribution of the SORE variable. lo, the Krukal-Walli tet could not be ued a an alternative becaue of the continuou cale on which many of the conidered potential factor are meaured (e.g. STRUTUREDNESS and SEQUENTILITY). Of all the correlation coefficient that were etablihed, ix of them diplayed the direction of the influence that we hypotheized upon with repect to the undertandability of proce model, i.e. #OR JOINS, DENSITY, VERGE ON- NETOR DEGREE, MISMTH, ONNETOR HETEROGENE- ITY, and ROSS-ONNETIVITY. For example, the correlation coefficient between ONNETOR HETEROGENEITY and average SORE equal., which correpond with the intuition that the ue of a wider variety of connector in a model will decreae it undertandability. However, a can be een in Table IV, only the correlation coefficient of DENSITY and VERGE ONNETOR DEGREE are ignificant at a 95% confidence level. To examine the value of the ditinguihed factor in explaining difference in SORE more thoroughly, we developed variou linear regreion model even though it hould be noted that the number of different model obervation i quite low for thi approach. We compared all 6 (= 6 ) linear regreion model that take a non-empty ubet into account of the factor hown in Table IV. To differentiate between the regreion model, we ued the adjuted R tatitic that meaure how the variability in the SORE i explained by each model. The bet adjuted R tatitic equal 79%, which i quite a atifactory reult. It belong to the regreion model that ue #OR JOINS, DENSITY, VERGE ONNETOR DEGREE, MISMTH, and ROSS-ONNETIVITY. Ue of the Durbin-Waton (DW) tatitic tet indicate no erial autocorrelation in the reidual at the 95% confidence level. viualization of thi regreion model can be een in Figure 6. Note that the outlying model L can be clearly identified at the bottom left corner. tated before, the number of model i too mall to make trong claim. Under thi provio, it i intereting to ee that the two factor which mot convincingly relate to model undertandability both relate to the number of connection in a proce model, rather than, for example, the generated tate pace. The VERGE ONNETOR DEGREE meaure the model average of incoming/outcoming arc per routing element, while DENSITY give the ratio of exiting arc to the maximal number of arc between the node in the model (i.e. when it would be completely connected). Both factor point to
IEEE TRNSTIONS ON SYSTEMS, MN, ND YBERNETIS PRT Fig. 6. Multivariate linear regreion model explaining the average SORE the negative effect of a relatively high number of dependencie in a model on a model undertandability. pparently, if ize i kept contant, factor related to complexity eem to be the mot ignificant one. ltogether, the finding tend to partially upport H that the more complex the model i, the wore will be the undertandability of it. Given the mall et of model, future reearch need to further invetigate thoe metric that were ignificant in thi urvey. D. Peronal veru model factor t thi point, we have een relation between both peronal factor and model factor on the one hand and the objective undertanding of a proce model a meaured through our SORE variable on the other. To invetigate which of thee domain ha the bigger influence, we developed variou linear regreion model to explain the variation in the average SORE for each model. For thi purpoe, we ued all 847 complete model evaluation at our dipoal. Uing the tepwie method a available in SPSS for automatically electing the ignificant factor, we developed different regreion model uing () peronal factor only, () model factor only, and () all factor. The idea behind thi approach i to ee which of the regreion model ha the better explanatory power. The reult are ummarized in Table V. The adjuted R tatitic diplay rather low value, but thi i not urpriing. fter all, peronal factor will not differ for any evaluation that the ame repondent i involved in. Similarly, model factor will not vary either for evaluation of the ame model. Moreover, the undertanding of the model i quite imilar a we have achieved by keeping the ize of the model different. Nonethele, thee reult give an important inight into the relative importance of peronal factor veru model factor in explaining model undertanding. can be een from Table V, the adjuted R value for the peronal factor i more than twice a high a that for the model factor. Thi i a clear indication that even though both type of factor may influence proce model undertanding, the influence of peronal factor i bigger. In addition, the combination of both type of factor give the highet explanatory power. V. THRETS TO VLIDITY One particular apect of the external validity of the preented reearch relate to the extent to which the ued model are repreentative for real-world model. explained, we countered thi threat by our choice of real proce model. While thi ha increaed the level of external validity, thi choice made it more difficult to manipulate the range of metric value range. In that ene, improved external validity may have actually lowered the internal validity. The other important apect which refer to a potentially limited external validity, relate to the involvement of tudent. tated, we involved a number of experienced proce modeler in a replication of the urvey (ee Section III-B). The average SORE that the profeional attained in thi replication ranged from.5 to 7.7, with an average value of 5.8. To compare, the SORE value for all tudent combined ranged from.75 to 8.58, with an average value of 6.9. To tet for any ignificant difference in proce model undertanding between the profeional modeler and the tudent, we applied the Krukal-Walli tet. Recall that we had already etablihed that the average SORE i not normally ditributed for the tudent population, which jutifie the choice for thi tet. The application of the Krukal-Walli tet indicate a tatitically ignificant difference among the variou population at a 95% confidence level (P-value =.). dditional pairwie Mann-Whitney tet [7] were conducted, taking into account the appropriate Bonferroni adjutment to control for Type error. In thi way, the cut-off equal.8, which i determined a the alpha level of.5 divided by the number of pairwie tet (6). Interetingly, the profeional model perform imilarly a the tudent from Madeira (UMa) (Pvalue =.684) and Vienna (WUW) (P-value =.75) but le than the Eindhoven (TU/e) tudent (P-value =.). While the median average SORE for the profeional wa 6.75, it wa 7.75 for the Eindhoven tudent. In other word, baed on our replication it doe not eem that tudent perform le with repect to cognitive tak than profeional with their greater experience, which i a common worry for thi kind of reearch. In comparion with one particular ubgroup of the tudent, a contrary effect could be oberved. In the context of thi tudy, thi may be an indication that knowledge of abtract modeling notion may be key in explaining proce model undertanding. In the wider context of empirical reearch, the outcome of the replication i encouraging with repect to the involvement of tudent intead of profeional modeler. VI. DISUSSION ND ONLUSION. Summary, reult and reearch contribution In thi paper we have motivated and invetigated the impact of peronal and model related factor on undertandability of proce model. Our main hypothee were that expert modeler will perform ignificantly better and that the complexity of the model affect undertanding. We explored different operationalization of thee hypothee and found upportive evidence. Furthermore, we calculated a combined regreion