The genetic architecture of hybrid incompatibilities and their effect on barriers to introgression in secondary contact

Transcription

1 ORIGINAL ARTICLE doi: /evo The genetic architecture of hybrid incompatibilities and their effect on barriers to introgression in secondary contact Dorothea Lindtke 1,2,3 and C. Alex Buerkle 1 1 Department of Botany and Program in Ecology, University of Wyoming, Laramie, Wyoming Department of Animal and Plant Sciences, University of Sheffield, Sheffield S10 2TN, United Kingdom 3 d.lindtke@sheffield.ac.uk Received March 14, 2015 Accepted July 8, 2015 Genetic incompatibilities are an important component of reproductive isolation. Although theoretical studies have addressed their evolution, little is known about their maintenance when challenged by potentially high migration rates in secondary contact. Although theory predicts that recombination can erode barriers, many empirical systems have been found to retain species-specific differences despite substantial gene flow. By simulating whole genomes in individuals of hybridizing species, we find that the genetic architecture of two contrasting models of epistatic hybrid incompatibilities and the context of hybridization can substantially affect species integrity and genomic heterogeneity. In line with theory, our results show that intergenomic incompatibilities break down rapidly by recombination, but can maintain genome-wide differentiation under very limited conditions. By contrast, intragenomic interactions that arise from genetic pathways can maintain species-specific differences even with high migration rates and gene flow, whereas introgression at large parts of the genome can simultaneously remain extensive, consistent with empirical observations. We discuss the importance of intragenomic interactions in speciation and consider how this form of epistatic fitness variation is implicated and supported by other theoretical and empirical studies. We further address the relevance of replicates and knowledge of context when investigating the genomics of speciation. KEY WORDS: Coadaptation, Dobzhansky Muller incompatibilities, gene flow, genetic pathways, simulation, speciation. Speciation involves the accumulation of genetic changes that contribute to reproductive isolation and the maintenance of species-specific traits in primary or secondary contact. Thus, understanding the process of speciation includes uncovering the genetic architecture of fitness variation in hybridizing species and how it is affected by migration and gene flow (Endler 1973; Butlin et al. 2012; Seehausen et al. 2014). Advances in this field involve theoretical questions (e.g., which genetic architectures are effective in reducing gene flow), as well as methodological questions (e.g., how to detect genetic changes important for speciation in empirical systems). Progress in speciation research resulted in revised or complementary evolutionary concepts, such as the genic view of speciation (Wu 2001), sympatric speciation (Bolnick and Fitzpatrick 2007), and the impact of hybridization on speciation (Arnold and Hodges 1995; Seehausen 2004; Abbott et al. 2013). Some of the evolutionary theory that predates these advances, however, has only begun to be reconsidered in the light of new data. For example, the classic Dobzhansky Muller model (following the formulation by Dobzhansky 1937; Muller 1942) for the evolution of hybrid incompatibilities, often considered a standard model for reproductive isolation (Gavrilets 2003; Coyne and Orr 2004), has limited efficacy in the face of gene flow (Barton and Bengtsson 1986; Gavrilets 1997; Bank et al. 2012). If some hybrids produce fertile recombinant progeny, Dobzhansky Muller incompatibilities (DMIs) can be eroded. Thus, DMIs are unlikely to be the primary cause of reproductive isolation under conditions that constitute high probabilities for interspecific recombination, for example, arising from overlapping species distributions and the presence of at least partially fit hybrids C 2015 The Author(s). Evolution C 2015 The Society for the Study of Evolution. Evolution 69-8:

2 D. LINDTKE AND C. A. BUERKLE Despite the limitations of classic DMIs and although several alternative concepts for the evolution of genetic hybrid incompatibilities exist in the speciation literature (e.g., Johnson 2010; Presgraves 2010; Nei and Nozawa 2011), the respective theory has received little formal exploration. One potentially important alternative model states that allele combinations within species become coadapted and recombination disrupts epistatic interactions in coadapted genomes, resulting in reduced hybrid fitness. For example, intragenomic coadaptation can result from protein protein interactions or regulatory genetic pathways (Johnson and Porter 2000; Edmands and Timmerman 2003; Ortiz-Barrientos et al. 2007; Livingstone et al. 2012). For simplicity, we refer to such intragenomic epistatic interactions as the pathway model in the following. Divergent genetic pathways can evolve easily and might have high relevance for the maintenance of species differences in conditions with high gene flow (see below), thus representing a potentially important alternative to classic DMIs. Given the known limited efficacy of DMIs and the need for more detailed theoretical investigations of alternatives, we focus on comparing the DMI and pathway models of epistatic hybrid incompatibilities, but do not study selection acting independently on multiple loci without epistasis (e.g., as in Barton 1983; Flaxman et al. 2014). Our particular goal is to investigate the maintenance of species barriers in conditions that include high migration, in contrast to previous work that addressed the evolutionary origin (but not maintenance) of barriers mainly under low migration conditions (including allopatry). We evaluated the efficacy of different genetic architectures in terms of maintenance of differentiation at directly selected loci, but (and unlike the majority of previous studies) also in terms of effects on linked and unlinked variants across the whole genome, as stronger isolation between species should result in a greater fraction of the genome that is protected from recombination and introgression (Barton and De Cara 2009). In addition to building our understanding of the efficacy of different genetic architectures of hybrid fitness, additional theoretical modeling is needed to build expectations for observable genomic variation in empirical studies of natural species boundaries. Although searching genomes for statistical extreme or distinctive patterns is increasingly feasible, it remains unclear how heterogeneous genomic differentiation can be linked to underlying processes (Seehausen et al. 2014). In this article, we therefore extend previous attempts (e.g., Gompert et al. 2012) and investigate the genomic outcomes of different genetic architectures of hybrid incompatibilities and how they are affected by the context of hybridization (e.g., time since contact and population demography). To address these questions, we use computer simulation to model whole diploid genomes subjected to selection and admixture in secondary contact zones. In the next section, we briefly review the concepts of our focal models of hybrid incompatibilities: DMIs and pathways. We then outline their differences, and provide an overview of our main goals and findings of this study. TWO MODELS OF EPISTATIC HYBRID INCOMPATIBILITIES To solve the problem of how hybrid sterility or unfitness can evolve without populations having to cross a maladaptive fitness valley, Dobzhansky (1937) and Muller (1942) recognized that at least two interacting loci are necessary. That is, given the ancestral, diploid two-locus genotype aabb and the innocuous substitutions to genotype aabb in one allopatric population and to AAbb in another, hybrids aabb that simultaneously carry derived, dominant alleles A and B will experience at least partially reduced fitness (Dobzhansky 1937; Muller 1942). The simplicity of the model made DMIs a very popular explanation for the evolution of reproductive isolation, although empirical studies supporting their importance for speciation remain rare (Brideau et al. 2006; Rieseberg and Willis 2007; Nei and Nozawa 2011). The analytical tractability of DMIs resulted in various theoretical studies of their origin, barrier strength and maintenance (e.g., Orr 1995; Gavrilets 1997; Turelli and Orr 2000). However, the classic DMI model required modifications to prevent the collapse of the barrier if F 1 hybrids were not always sterile. Particularly, if compatible ancestral genotypes reemerge through recombination, compatible variants can rapidly increase in frequency and the barrier is lost (Fig. 1A; Barton and Bengtsson 1986; Gavrilets 1997; Gompert et al. 2012). Accordingly, most theoretical work on DMIs includes adaptive (and often divergent) selection on derived variants, or recessivity of incompatible alleles (Fierst and Hansen 2010; Nosil and Flaxman 2011; Bank et al. 2012). This indicates that classic DMIs on their own are ineffective at maintaining a species barrier under conditions that allow recombination and gene flow. Although various interpretations of DMIs and corresponding epistatic fitness matrices exist in the literature (e.g., Agrawal et al. 2011; Maheshwari and Barbash 2011; Nei and Nozawa 2011; Nosil and Flaxman 2011; Bank et al. 2012), here we consider classic DMIs (as described by Dobzhansky 1937; Muller 1942) as a special case within other forms of epistatic fitness variation that gives rise to hybrid incompatibilities. We emphasize this distinction because of the variety of possible epistatic interactions in hybrids and the substantial differences in their expected outcomes (see below). An alternative model of epistatic hybrid incompatibilities that might have high biological significance arises from knowledge of coadaptation within genomes (Edmands and Timmerman 2003) and regulatory genetic pathways (Johnson and Porter 2000; Ortiz-Barrientos et al. 2007). The common principle of these intragenomic interactions is that a trait is expressed normally if all its genetic components interact properly (e.g., transcription factors, 1988 EVOLUTION AUGUST 2015

3 THE GENETIC ARCHITECTURE OF HYBRID INCOMPATIBILITIES A B Figure 1. Schematic illustration of two contrasting models of epistatic hybrid incompatibilities. Species 1 and species 2 evolve from a common ancestor and establish new mutations at two different loci. The derived genotypes can be recombined in secondary contact. Fitnesses of F 1 and recombinant hybrids differ between the two models, from full fitness (white circles) to reduced fitness (gray circles; Tables 1 and 2). (A) With DMI-type incompatibilities, F 1 hybrids have reduced fitness as they carry independently derived, incompatible alleles A and B at the two loci. Recombinant hybrids that do not carry the incompatible allele combinations can be fit. (B) With pathwaytype incompatibilities, fitness depends on the presence of compatible haplotypes A 1 B 1 or A 2 B 2 at coadapted loci. F 1 hybrids with balanced genotypes can be fit, whereas disrupted, unbalanced genotypes result in reduced fitness in recombinants. biosynthetic steps or protein substructures; Fig. 1B). Disruption of the interacting genetic components through interspecific recombination can result in low hybrid fitness, known as outbreeding depression (Lynch 1991). The potential importance of pathways in speciation is supported by various studies showing that: (1) regulatory genetic pathways are a widespread mechanism to generate phenotypic variation (Romero et al. 2012; Boyle et al. 2014), (2) divergent evolution of pathways can be induced by adaptation (Johnson and Porter 2000, 2007; Porter and Johnson 2002; Palmer and Feldman 2009; Chevin et al. 2014; Tulchinsky et al. 2014) or random genetic drift (Lynch and Hagner 2015), and (3) low hybrid fitness can result from incompatibilities between regulatory elements (True and Haag 2001; Landry et al. 2007; Hegarty et al. 2009; Renaut and Bernatchez 2011). Pathways might thus constitute an important alternative mechanism to DMIs for the maintenance of species differences in conditions with high gene flow. However, this hypothesis has not yet received much attention in theoretical work, a gap that we want to address with the current study. A critical difference between DMIs and pathways are the expected fitnesses of recombinant hybrids, which can be described by parameters in a fitness matrix or an adaptive landscape (e.g., Gavrilets 1997). With DMIs, recombinants with high fitness can emerge and thus negatively interacting (derived) alleles from different species will be purged from a hybrid population (Table 1, Fig. 1A). With the pathway model, recombinants with Table 1. Genotype fitnesses for the DMI model. BB bb bb aa aa 1 s 1 s 1 AA 1 s 1 s 1 Notes Genotype aabb is ancestral, and allele A is derived in one species at the first locus, allele B is derived in the other species at the second locus. Genotypes aab B and AAbb are fixed in species 1 and species 2. Combinations of alleles A and B result in fitness reduction by s. Table 2. Genotype fitnesses for the pathway model. B 1 B 1 B 1 B 2 B 2 B 2 A 1 A s/2 1 s A 1 A 2 1 s/2 1 1 s/2 A 2 A 2 1 s 1 s/2 1 Notes Genotypes A 1 A 1 B 1 B 1 and A 2 A 2 B 2 B 2 are fixed in species 1 and species 2. The presence of compatible alleles at interacting loci (A 1 B 1 or A 2 B 2 ) is required for full fitness. Missing compatible combinations result in fitness reduction by s/2. disrupted intragenomic interactions will have low fitness (Table 2, Fig. 1B). Thus, in contrast to DMIs, interspecific recombination and gene flow will not erode the species barrier at epistatic loci in pathways. This makes alternative pathways with coevolved allelic EVOLUTION AUGUST

4 D. LINDTKE AND C. A. BUERKLE differences a potentially effective mechanism for the maintenance of species differences in conditions with high gene flow, the setting where DMIs typically fail. The majority of previous theoretical work on epistatic hybrid incompatibilities investigated simple genetic architectures and the maintenance of incompatibilities in parapatric contact (e.g., Gavrilets 1997; Bank et al. 2012), or the build-up of reproductive isolation during adaptation (e.g., Johnson and Porter 2000; Agrawal et al. 2011; Nosil and Flaxman 2011). Further, many studies did not address recombination (e.g., Palmer and Feldman 2009) or used haploid models (e.g., Agrawal et al. 2011), although the barrier to gene flow is expected to be substantially affected by recombination in diploid organisms (Gavrilets 1997; Barton 2001). With few exceptions (e.g., Gompert et al. 2012), only directly selected loci and limited sets of linked or unlinked variants were studied, thus it remains uncertain to what extent different genetic architectures of selection cause heterogeneous genomic introgression in hybrids and affect species barriers. The goal of this study was therefore twofold: assessing two contrasting genetic architectures of hybrid incompatibilities in diploids to investigate (1) their efficacy as species barriers in spatially explicit hybrid zones with high migration rates, and (2) their genomic outcomes of selection. Our model differs from other studies in that we modeled a large, spatially explicit and ecologically homogeneous contact zone between previously diverged species that allowed for source-sink dynamics and substantial migration rates between pure species and hybrids (but see Gavrilets 1997), and that we investigated simple and complex architectures of epistasis and their effects on admixture by modeling whole diploid genomes rather than a limited set of loci (but see Gompert et al. 2012). We modeled intrinsic postzygotic isolation arising from DMIs and pathways with different selection strengths and migration rates, and monitored how genetic differentiation, introgression, and admixture were affected across time and space and along the genome. We emphasize two main findings. First, pathways constitute a promising model for the maintenance of species differences, particularly in the problematic conditions of high gene flow in sympatry and high hybrid fitness. We further confirm that these are the conditions when classic DMIs break down. Thus, pathways or analogous forms of intragenomic interactions require more attention as an important mechanism involved in speciation. Second, selection on hybrid incompatibilities resulted in heterogeneous genomic introgression and admixture that could differ among genetic architectures of epistasis. However, genomic patterns of variation also strongly depended on the spatial, temporal, and demographic context of hybridization, highlighting that empirical observations need to be interpreted cautiously. We discuss our main findings in relation to empirical studies and other theoretical work. Model and Methods To investigate the genetic architecture of hybrid incompatibilities and their effects on the maintenance of genome differentiation, we simulated whole genomes of individuals in secondary contact between previously diverged species. As our aim was to address barrier maintenance, we assumed that incompatibilities already evolved and started with fixed genomic differences between species and recorded admixture and introgression through time. We modeled a chain of demes with finite population sizes that were connected by migration using a stepping-stone model (similar to Gavrilets 1997). Infinitely sized populations of different species at each end of the chain acted as continuous sources of unadmixed individuals. We thus modeled the metapopulation (the chain of demes) as only receiving immigrants (a sink) without any emigration to the core of species ranges (the source), in line with hybridization often occurring at range margins (Bridle and Vines 2007; Abbott et al. 2013). We assessed the maintenance of species differences in a homogeneous environment using two models of intrinsic epistatic selection: intergenomic incompatibilities (DMIs) and intragenomic interactions (pathways). We chose these architectures to investigate two extremes of many other conceivable forms of epistasis that might contribute to speciation. In both models, some hybrid genotypes were assigned full fitness, thus pure species epistatic genotypes were not separated by fitness valleys (Tables 1 and 2). We note that the two fitness matrices can converge under less strict definitions that allow for separated fitness peaks. For example, by modifying our specification for classic DMIs of Table 1 to simultaneously allow for nondominant incompatibilities and fitness disadvantage of ancestral alleles (Table S1), the resulting fitness matrix can become very similar (and even identical) to our specification for the pathway model (Tables 2 and S2). In the following, we describe the basic simulation setup, selection models, and summary statistics on the simulation runs. Further details are in the Supporting Information. Simulation C code together with a brief description of the software (which we named dfuse) is available from the second author s website. SIMULATION SETUP We describe the basic simulation setup used for the majority of our simulation runs below. We subsequently modified this basic model and tested the robustness of our results (Supporting Information). We simulated secondary contact with sourcesink dynamics by connecting two species source populations (of infinite size) by three finite demes with adult carrying capacity of N c = 500. Neighboring demes were connected by migration of rate m/2, and we chose high m ={0.01, 0.2} for most runs to emulate hybrid zone dynamics. Every generation, unadmixed individuals from species sources immigrated into the 1990 EVOLUTION AUGUST 2015

5 THE GENETIC ARCHITECTURE OF HYBRID INCOMPATIBILITIES peripheral demes (i.e., deme 1 or deme 3), whereas individuals that emigrated from the peripheral demes outward were eliminated. We initialized demes with unadmixed individuals, with the central deme receiving an equal mixture from both species, consistent with sympatric contact between previously isolated species. We assumed a homogeneous environment (i.e., no ecological differences among demes). We modeled genomes of diploid individuals with 2N = 20 autosomal chromosomes, each of 1 Morgan length. The average number of crossovers per chromosome per meiosis was one (Poisson distributed). We stored ancestry of chromosome blocks and the history of interspecific recombination events as in previous simulations (Buerkle and Rieseberg 2008; Gompert and Buerkle 2011; Gompert et al. 2012), and placed markers every 2 cm to sample 510 loci (51 per chromosome; including directly selected sites) for locus-specific statistics every 10 generations. We modeled hermaphroditic, nonselfing individuals with discrete generations. Mating was random within a deme and viability selection (see section below) acted on progeny. The reproductive phase continued until surviving offspring filled the deme to progeny carrying capacity N p = 1000 or available maternal gametes (mean of five per individual) were depleted, whichever occurred first. Beyond reproduction, the life cycle was completed by progeny dispersal, adult mortality, and random survivorship during aging of progeny to cap the new generation at adult carrying capacity. We describe model modifications, including different carrying capacities N c ={50, 5000} and N p ={100, },or parapatric contact, in the Supporting Information. SELECTION As outlined in the Introduction, our goal with this study was to investigate two contrasting genetic architectures of epistatic hybrid incompatibilities. The two models, DMIs and pathways, were intended to simulate alternative genetic architectures resulting in unfit or fit F 1 and recombinant hybrids (corresponding to the wide range of hybrid fitnesses in empirical systems; Arnold and Hodges 1995; Barton 2001; Burke and Arnold 2001; Presgraves 2010). We focused on intrinsic postzygotic selection in a homogeneous environment, as this is a condition that can result in a largely stable barrier to gene flow (i.e., independent on changes in environment or mating preferences). Epistatic selection against incompatible two-locus interactions reduced the survival probability of progeny before dispersal (Tables 1 and 2), where survival probabilities of multiple two-locus epistatic interactions were combined multiplicatively (i.e., for more than two loci under selection; Tables S3 and S4). Because strong selection was required to result in observable effects, we chose selection strength s ={0.2, 0.9} for most runs. As our goal was to model the characteristics of DMIs and pathways and to compare them, we chose appropriate numbers of interacting loci and their recombination distance for our basic settings, and then investigated modifications of these architectures (Supporting Information). We modeled intergenomic incompatibilities (DMIs) following Dobzhansky (1937) and Muller (1942) strictly. In particular, we modeled incompatible interactions between dominant alleles that derived independently between species at two different loci (Table 1, Fig. 1A). Incompatibilities were thus manifested in interspecific heterozygotes (i.e., F 1 s were unfit). In our basic setting, we assumed that derived alleles within a species were selectively neutral (which is not explicit in the original description). We modeled interactions between two loci at recombination distance of 20 cm. We then extended this model and investigated complex multilocus interactions (up to 10 loci) and their genomic consequences (Supporting Information; Figs. S1 and S2). We modeled intragenomic interactions (pathways) following ideas from simulation studies on intrinsic coadaptation (Edmands and Timmerman 2003) or linear regulatory genetic pathways (Johnson and Porter 2000). We modeled p linearly interacting loci, and fitness varied according to epistatic interactions between all p 1 consecutive pairs of loci and their multiplicative effects. The model corresponds, for example, to a genetic pathway where a promoter at the first locus is necessary to regulate a gene at the second locus, whose product then interacts with the next locus, and so on. Interacting sites that diverged between species can lose compatibility, resulting in aberrant traits in recombinants (Fig. 1B). Fitness was reduced by s/2ors if one or two alleles at interacting loci were incompatible (Table 2). With this model, we explicitly allowed for full fitness of individuals with functional genetic pathways, independent on the origin of that pathway (i.e., pure species, F 1 hybrids, and the subset of recombinant hybrids with complete sets of interacting loci could be fully fit). Our basic setting involved four loci, each separated by 20 cm. We investigated additional genetic architectures involving two to 36 interacting loci (Supporting Information; Figs. S1 and S2). SIMULATION OUTPUT AND STATISTICS We ran 20 replicates of each of our settings for 100 generations, which appeared to be sufficient to investigate the most determining parameters of barrier efficacy and the genomic outcomes of selection. We included simulations where we set selection strength to zero to obtain base levels of admixture and introgression from purely demographic processes. For each generation, replicate, and deme, we recorded genome-wide admixture proportion (proportion of species 1 ancestry) and genome-wide intersource ancestry (interspecific heterozygosity), and locus-specific ancestry (0, 1, or 2 alleles from species 1) for all 500 adult individuals per deme. Locus-specific admixture proportion and locus-specific intersource ancestry were computed as averages from individual locus-specific ancestries within each deme or for the metapopulation. We calculated genome-wide and EVOLUTION AUGUST

6 D. LINDTKE AND C. A. BUERKLE locus-specific = (H T H S )/H T (Nei 1977) by obtaining the expected heterozygosity in the metapopulation (H T ) and the average expected heterozygosity within demes (H S ) directly from the genome-wide or locus-specific admixture proportions. We further calculated linkage disequilibrium from haplotype frequencies, adjusted by allele frequencies (D ; Lewontin 1964), from 50 randomly sampled individuals from the central deme. We simplified D to one dimension by reporting values only for locus pairs separated by 10 cm distance on the genetic map. Further details are in the Supporting Information. Results We modeled classic intergenomic incompatibilities (DMIs) and intragenomic interactions resulting from genetic pathways as two extreme forms of epistatic hybrid incompatibilities out of many other possible architectures, and investigated their potential efficacy in maintaining genetic differences between species in secondary contact. We describe the resulting genome-wide and locus-specific processes and patterns of admixture and introgression and how they were affected by the demographic and spatial context of hybridization and our model specifics. In particular, consistent with previous studies (Barton and Bengtsson 1986; Gavrilets 1997; Gompert et al. 2012), we show that DMIs typically broke down rapidly when interspecific recombination occurred, but constituted very efficient species barriers that protected whole genomes from introgression under some limited conditions. By contrast, barriers to gene flow resulting from genetic pathways were less affected by model specifics or temporal dynamics, but reduced introgression only for parts of the genome. THE GENETIC ARCHITECTURE OF HYBRID INCOMPATIBILITIES AND GENOME-WIDE PROCESSES OF ADMIXTURE Classic DMIs We illustrate the outcomes of genome-wide admixture after secondary contact with classic DMIs under different strengths of selection and different migration rates in Figure 2 (top row of A and B). As expected, the rate of reduction of over 100 generations was strongly affected by migration rate. Compared to purely demographic processes (s = 0.0, black lines in Fig. 2), only strongly selected DMIs (s = 0.9, orange lines) and low migration rates (m = 0.01) impeded the reduction of genome-wide considerably (Fig. 2A). Weaker selection (s = 0.2, green lines) or high migration rates (m = 0.2; Fig. 2B) resulted in disruption of DMIs because the formation and survival of F 1 hybrids was more likely, enabling interspecific recombination (see below). The consequent rapid breakdown of the species barrier resulted in converging levels of genome-wide among models with and without selection. With low migration rates, the timing of breakdown reflected chance variation (wide SDs among replicates in Fig. 2A for s = 0.9), but occurred after only few generations with high migration rates (Fig. 2B). Genetic pathways By contrast, with the pathway model, we explicitly allowed full fitness for F 1 hybrids, but selection acted against interspecific recombinants with unbalanced epistatic genotypes: species-specific alleles at epistatically interacting loci needed to segregate jointly for full fitness. As F 1 s were fully fit, genomic admixture was largely unrestricted and genome-wide was reduced immediately and at a rate inversely proportional to selection (Fig. 2, top row of C and D). The reduction of varied less stochastically among replicates and instead declined linearly with little variance among replicates, as parts of the genome unlinked to selected loci introgressed freely (see below). Genome-wide stabilized after few generations and was not subjected to further reduction even with high migration rates (Fig. 2D). THE GENETIC ARCHITECTURE OF HYBRID INCOMPATIBILITIES AND LOCUS-SPECIFIC CONSEQUENCES OF INTROGRESSION Classic DMIs We also monitored the genomic outcomes of hybridization at individual loci by calculating, admixture proportion, intersource ancestry, and D after several generations since contact (Fig. 2, second and third row of A and B; Figs. 3 and 4A, B). The observed dynamics were dependent on selection strength, migration rate, and time since contact. Within the first few generations, DMIs could remain intact and thus reduced gene flow genome-wide as hybrid fitness was low (Fig. 2, second and third row of A and B; thick, light-colored lines; Fig. S30A, B). However, DMIs were rapidly disrupted in the hybrid deme through interspecific recombination (Fig. 5). Consequently, incompatible derived alleles were quickly purged and compatible ancestral alleles increased in frequency (Figs. 3 and 4, top rows of A and B), whereas recombinant hybrids with compatible ancestral alleles recovered full fitness (Fig. S30A, B). This then allowed largely unrestricted introgression and reduction of across the genome (Fig. 2, second and third row of A and B; thin, dark-colored lines). Intact DMIs and derived variants from the infinite parental source populations continued to enter the hybrid deme, but did not cross it (Figs. 5 and S3). As unselected parts of the genome could introgress freely among demes after barrier breakdown, remained elevated relative to the remainder of the genome only at or closely linked to selected sites (Fig. 2A, B; these peaks disappeared through time when modeling finite source populations with high migration, Fig. S7B) EVOLUTION AUGUST 2015

7 THE GENETIC ARCHITECTURE OF HYBRID INCOMPATIBILITIES A DMIs, m= B DMIs, m=0.2 mean +/ SD s= g=10 g=20 g=40 g=60 g=80 C Pathways, m= D Pathways, m= , single replicate, single replicate, single replicate, single replicate Figure 2. Reduction of genome and locus-specific patterns of for two models of epistasis (DMIs and pathways) and different migration rates (m = 0.01 and 0.2). Depicted are results for the basic model (infinite source populations, three demes, each with capacity N c = 500). Top row shows reduction of genome over generations for different strengths of selection (black, no selection; green, s = 0.2; orange, s = 0.9). Thick and thin lines show mean and ± SD over 20 replicates. Other rows show locus-specific patterns of along the genome (x-axis, truncated within chromosome V; dotted vertical lines indicate chromosome boundaries). Shades of blue indicate generations since contact when samples were taken. Triangles show genome position of selected loci. The second and third row show mean values over 20 replicates for s = 0.2 and0.9, the fourth row shows results for a single replicate with s = 0.9. Genetic pathways With pathways, genome-wide gene flow was not delayed and instead started immediately after contact (Fig. 2C, D). However, gene flow at selected loci and linked sites remained low, resulting in elevated and even increasing with low migration rates over time (Fig. 2C; peaks did not disappear when modeling finite source populations, Fig. S7C, D). We did not observe an increase of particular species-specific alleles for pathway loci when averaging over replicates (Figs. 3 and 4, top row of C and D). However, the decrease in intersource ancestry with low migration (Fig. 4, middle row of C) suggests that one of the pathways became fixed in the hybrid deme, which we could confirm with our results for single replicates (Fig. S3, middle row of C). Fixation of one of the pathways will amplify population structure and probably caused the observed increase in. Allele combinations at pathway loci remained largely intact through time (Fig. 6), although population fitness was reduced by hybrids with unbalanced genotypes (Fig. S30C, D), consistent with outbreeding depression. With strong selection and high migration rates, alleles at pathway loci remained associated within large chromosomal blocks, as evidenced by highly elevated D across large parts of the selected chromosome (Fig. 4, bottom row of D; Fig. S32D). THE CONTEXT OF HYBRIDIZATION AFFECTS GENOMIC OUTCOMES Our results indicate that the demographic, spatial, and temporal context of hybridization strongly influenced outcomes, even for a single model of epistasis. Population demography, particularly migration rate, had a strong effect on the genomic outcomes of hybridization by altering the probability of interspecific recombination and by affecting the rate of genetic drift within a deme. For example, DMIs constituted strong barriers to gene flow but required very high selection coefficients and low migration rates (m s) to maintain species integrity over several tens of generations, but broke down otherwise (Fig. 7, orange symbols). EVOLUTION AUGUST

8 D. LINDTKE AND C. A. BUERKLE A Adm. prop. DMIs, m=0.01, B Adm. prop. DMIs, m=0.2, g=10 g=40 g=80 g=20 g=60 C Adm. prop. Pathways, m=0.01, D Adm. prop. Pathways, m=0.2, Inter source anc. Inter source anc. g=10 g=20 g=40 g=60 g=80 Inter source anc. Inter source anc. D' D' g=10 g=20 g=40 g=60 g=80 D' D' Figure 3. Locus-specific patterns of admixture for two models of epistasis (DMIs and pathways) and different migration rates (m = 0.01 and 0.2) with selection strength s = 0.2. Depicted are results for the hybrid deme (deme 2) for the basic model (infinite source populations, three demes, each with capacity N c = 500), mean over 20 replicates. Top, admixture proportion; middle, intersource ancestry; bottom, linkage disequilibrium (D ) between loci each 10 cm apart. Shades of color indicate generations since contact when samples were taken. Triangles show genome position of selected loci. Dotted vertical lines indicate chromosome boundaries, and the genome is truncated within chromosome V. A Adm. prop. DMIs, m=0.01, B Adm. prop. DMIs, m=0.2, g=10 g=20 g=40 g=60 g=80 C Adm. prop. Pathways, m=0.01, D Adm. prop. Pathways, m=0.2, Inter source anc. Inter source anc. g=10 g=20 g=40 g=60 g=80 Inter source anc. Inter source anc. D' D' g=10 g=20 g=40 g=60 g=80 D' D' Figure 4. As in Figure 3, but selection strength s = 0.9. In strong contrast, we did not observe this high dependency on selection strength and migration rate for pathways (Fig. 7, red symbols). Low migration rates intensified genetic drift for both architectures, for example, when one of the variants in pathways was lost from a deme (Fig. 4, middle row of C; Fig. S3C). The spatial context of hybridization, particularly the position of the sampled deme (e.g., middle or peripheral position) had a strong effect on the observable outcomes of hybridization for that deme (Figs. 5, 6, S3, S4, and S28 S31). The temporal context of hybridization (i.e., time since contact) substantially affected admixture and gene flow, particularly within the first generations (Figs. 2 6). Our findings thus illustrate that samples taken from different replicate hybrid zones can potentially show very different outcomes of hybridization even if the underlying genetic architecture of epistasis is identical. This context of hybridization also affected the genomic consequences of selection that we address below. HETEROGENEOUS GENOMIC OUTCOMES OF SELECTION Summary statistics at and near selected loci were often more extreme than statistics for the remainder of the genome. The extent of this heterogeneity in locus-specific statistics was in some cases affected by the specific genetic architecture of hybrid incompatibilities (Figs. 2 4). For example, was highly elevated across a large part of the selected chromosome with pathways, but was 1994 EVOLUTION AUGUST 2015

9 THE GENETIC ARCHITECTURE OF HYBRID INCOMPATIBILITIES Frequency Frequency Frequency mean q +/ SD for mean q species 1 DMI intact species 2 DMI intact total intact DMIs, m=0.01, DMIs, m=0.2, Deme 1 Deme 2 Deme 3 Figure 5. Disruption of DMIs over time (x-axis) and space (rows) for different migration rates (m = 0.01 and 0.2) and selection strength s = 0.9. Shown is the frequency of intact DMI loci with ancestry in species 1 (blue), species 2 (red), and the total frequency of intact complexes (white) in the hybrid deme (middle row; deme 2), and the peripheral demes (top and bottom row; deme 1 and deme 3). Boxplots summarize results for 20 replicates of the basic model. Thick and thin gray lines show mean deme admixture proportion q and ± SD of mean q over 20 replicates. Frequency Frequency Frequency mean q +/ SD for mean q species 1 pathway intact species 2 pathway intact total intact Pathways, m=0.01, Pathways, m=0.2, Deme 1 Deme 2 Deme 3 Figure 6. Disruption of pathways over time (x-axis) and space (rows). Shown is the frequency of fully intact pathways with ancestry in species 1 (blue), species 2 (red), and the total frequency of fully intact pathways (white). Otherwise as in Figure 5. less extremely elevated and this only closely linked to selected sites with DMIs (Fig. 2). In contrast, admixture proportions did not reveal any extreme statistics at or near selected loci for pathways when averaging replicates, but showed a very conspicuous outcome for DMIs (Figs. 3 and 4, top row). These extreme admixture proportions only emerged after DMIs broke down due to the rapid increase of compatible ancestral alleles. Averaging over replicates erased some consequences of admixture that are evident in individual simulations (i.e., random fixation of different pathway variants, Fig. 4C, top row, and Fig. S3C), but in most instances replicates were essential for detecting any genomic consequence of selection (particularly when drift obscured outcomes; Fig. 2, bottom row; Figs. S3 and S4). Low migration rates often, but not always, increased genomic heterogeneity, and several generations since contact typically amplified average statistical parameters for selected loci across replicates. EVOLUTION AUGUST

10 D. LINDTKE AND C. A. BUERKLE Figure 7. Reduction of genome for DMIs (orange), pathways (red), and purely demographic processes (black) dependent on migration rate (m, x-axis) for different generations since admixture (g, columns) and selection strengths (s, rows). Circles show mean over 20 replicates, vertical lines indicate ± SD. Simulations were run using the basic model. MODEL MODIFICATIONS We investigated how the above results were affected by several of the specifics of our basic model. In particular, we studied more complex genetic architectures of epistasis, which could have evolved after long allopatric divergence. We further investigated the effects of recombination distance between interacting loci, and of population demography and spatial arrangement of demes. Complexity of epistasis We constructed complex epistatic interactions for DMIs (involving two complexes, each with five interacting linked loci), and pathways (nine complexes, each with three linked loci and one unlinked locus; Supporting Information). Selection of strength s = 0.2 or 0.9 acted on each pairwise epistatic interaction and was combined multiplicatively for all interaction pairs, resulting 1996 EVOLUTION AUGUST 2015

11 THE GENETIC ARCHITECTURE OF HYBRID INCOMPATIBILITIES A Complex DMIs, m= B Complex DMIs, m=0.2 mean +/ SD s= C Complex pathways, m= D Complex pathways, m= g=10 g=20 g=40 g=60 g=80, single replicate, single replicate, single replicate, single replicate Figure 8. As in Figure 2, but for complex genetic architectures of epistasis. in very strong selection against F 1 s with DMIs and potentially very strong selection against recombinants with pathways. With these settings, species barriers became absolute for both DMIs and pathways even with high migration rates, and remained stable through time (Fig. 8). Nevertheless, some genetic introgression still occurred for s = 0.2 (Fig. 8, second row; Fig. S5), but not with s = 0.9 (Fig. 8, third row; Fig. S6). We describe additional modifications of the genetic architectures of epistasis including various numbers of interacting loci with different recombination distances, and selection against ancestral variants or recessive incompatibilities, in the Supporting Information. Recombination distance between interacting loci Modifying linkage between epistatic loci had differing effects for DMIs and pathways (Supporting Information; Figs. S11 and S12). Tightly linked loci (2 cm) delayed barrier breakdown for DMIs (Fig. S11A), but did not reduce gene flow for pathways (Fig. S11C). Correspondingly, unlinked loci decreased barrier strength for DMIs (Fig. S11B; both loci unlinked), but increased barrier strength for pathways (Fig. S11D; second interaction unlinked). These results should be expected because reduced recombination distance between DMIs will delay their breakdown (with complete linkage being equivalent to single-locus underdominance), whereas tight linkage within pathways will make them effectively behave like a single locus if their recombination is sufficiently rare. We note that the effects of linkage can differ for more complex architectures of DMIs, which is addressed in the Supporting Information. Population demography and spatial settings In our basic model, we simulated the hybrid zone as a metapopulation with three demes connected by bidirectional migration, but with unidirectional migration from infinite-sized unadmixed species sources, corresponding to secondary contact at range margins. We modeled sympatric contact and set the carrying capacity of each deme to N c = 500. In additional simulations, we modified these three assumptions and simulated finite species source populations that could receive admixed immigrants, parapatric contact, or carrying capacities N c ={50, 5000} (Supporting Information). Briefly, finite source populations affected the results significantly after several generations since contact for high migration rates (Fig. S7). Here, genome-wide decreased close to zero for both models of epistasis, as no new unadmixed individuals entered the metapopulation. DMIs inevitably broke down, and at selected sites remained high only for pathways. Parapatric contact increased the efficacy of DMIs as species barriers, particularly for low migration rates, but only had a minor effect on pathways (Fig. S8). High carrying capacities of N c = 5000 resulted in less EVOLUTION AUGUST

12 D. LINDTKE AND C. A. BUERKLE effective species barriers for DMIs (Fig. S9), whereas pathways remained largely unaffected (Fig. S10). In summary, species barriers and genomic patterns of admixture for pathways showed little dependency on model specifics. The effects for DMIs were more substantial, congruent with their breakdown depending on recombination opportunity, which increased with finite source populations, sympatric contact, and large hybrid zones. We address these dynamics and other model modifications (e.g., the effects resulting from ecologically based divergent selection on epistatic loci) in more detail in the Supporting Information. Discussion Various mechanisms for the evolution of hybrid incompatibilities have been suggested by theoretical studies, but little is known about their efficacy in maintaining species barriers in secondary contact and how empirical observations might be informative about different processes that shape genomic outcomes of hybridization. DMIs are sometimes treated as synonymous with epistatic hybrid incompatibilities, despite fundamental differences between the classic model of DMIs and other forms of epistatic interactions, including genetic pathways. Using simulations to study classic DMIs and genetic pathways and their efficacy as species barriers, we found that both models were unable to prevent extensive genomic introgression in secondary contact zones unless particular conditions were met. However, the models differed strongly in their consequences for species barriers and patterns of admixture. Although DMIs cause low F 1 hybrid fitness and thus could prevent whole genomes from introgression, even rare reproduction from F 1 hybrids and fit recombinant progeny resulted in barrier breakdown and subsequent unrestricted gene flow between species (consistent with previous work; Barton and Bengtsson 1986; Gavrilets 1997; Gompert et al. 2012). In contrast, with genetic pathways, F 1 hybrids were fully fit, and thus gene flow occurred immediately across the genome except at sometimes large chromosomal blocks linked to epistatic loci, where recombinant genotypes were selected against. Integrity of species-specific interactions could thus be maintained through time. Consequently, the two models of epistasis differ strongly in their genomic outcomes, temporal dynamics, and their efficacy as species barriers depending on the extent of migration in secondary contact. Our results indicate that heterogeneous patterns of introgression and substantial gene flow in empirical hybrid zones are unlikely to result from intact DMIs, but are consistent with the outcomes of intragenomic epistasis in genetic pathways. CLASSIC DMIS AND THE MAINTENANCE OF SPECIES BARRIERS IN SECONDARY CONTACT ZONES Our results show that barriers to gene flow are unlikely to be maintained by classic DMIs unless certain conditions are met. Specifically, DMIs only constituted effective barriers if recombination between interacting loci was prevented. This could be achieved by very strong and invariant selection against F 1 hybrids (s 1 or high complexity of epistatic interactions), very low migration rates between species so that the formation of F 1 hybrids was unlikely, or a combination of these factors (Fig. 2A, B; Figs. 7, 8A, B, and S8A). In these cases, parental allele combinations at DMIs remained associated and resulted in very strong species barriersasnof 1 hybrids reproduced and no fit interspecific recombinants could be formed, protecting the whole genome from introgression (panels A and B in Figs. 8, S6, S13, and S14). The efficacy of complex DMIs can be attributed to our model specifics where each pairwise epistatic interaction contributed multiplicatively to fitness. The probability of survival and reproduction of F 1 hybrids was thus near zero and even with rare recombination events, epistatic interactions among additional locus pairs reduced fitness of recombinants severely. Multiple pairwise DMIs versus complex multilocus interactions had very similar effects on the species barrier (Figs. S13 and S14; see Supporting Information for further discussion). Our results thus confirm that complex hybrid incompatibilities can constitute very strong species barriers (Orr 1995; Gavrilets 2003), and can be maintained even with neutral DMIs (i.e., without selection against ancestral variants). The latter finding differs from previous studies that showed instability of nearly neutral two-locus DMIs in two-deme setups (Nosil and Flaxman 2011; Bank et al. 2012), suggesting that our model specifics with multiple demes and complex genetic architectures under very strong selection were critical for barrier maintenance. Importantly, intact parental allele combinations at DMIs that prohibit hybrid survival provide genome-wide barriers to gene flow. Our finding that DMIs require very particular conditions to constitute persistent species barriers is congruent with analytical theory and previous simulation studies (Gavrilets 1997; Bank et al. 2012). The classic two-locus model of DMIs might thus be viewed as a simplified model of how complete speciation in allopatry could arise (see Porter and Johnson 2002). However, this model for the origin of intergenomic incompatibilities is limited in explaining the maintenance of species barriers that are incomplete and permeable to gene flow. Consistent with our results, few empirical studies have provided direct evidence for the significance of DMIs as species barriers (Brideau et al. 2006; Sweigart et al. 2006), although the suggested support has also been questioned by some authors and remains controversial (Rieseberg and Willis 2007; Presgraves 2010; Nei and Nozawa 2011). Experiments were rarely run for many generations and might thus have missed delayed barrier breakdown, whereas DMIs that fully preclude hybrid survival, or those that are located in nonrecombining chromosome inversions, will be particularly difficult to detect. By contrast, DMI-like epistatic polymorphisms exist within species (Corbett-Detig et al EVOLUTION AUGUST 2015