T testing whether a sample of genes at a single locus was consistent with

Transcription

1 (:op)right by the Genetics Society of America THE HOMOZYGOSITY TEST AFTER A CHANGE IN POPULATION SIZE G. A. WATTERSON Department of Mathematics, Monash University, Clayton, Victoria, 3168 Australia Manuscript received April 22, 1985 Revised copy accepted November 2 1, 1985 ABSTRACT The homozygosity of a population will be influenced by any recent change in the population size. The homozygosity test of the neutral mutation hypothesis might also be influenced by a population change. A computer simulation method is described that establishes the significance levels of observed homozygosities after a change in population size. Some numerical examples are given. WO earlier papers, (WATTERSON 1977, 1978) considered the problem of T testing whether a sample of genes at a single locus was consistent with the hypothesis that all mutants were selectively neutral. Under strict neutrality, and assuming statistical stationarity in the infinitely many alleles model, a random sample of genes should have allele frequencies consistent with EWENS (1972) probability distribution. If the alternative hypothesis is that heterozygotes have a selective advantage, the natural test-statistic to use to discriminate between the hypotheses is the sample heterozygosity-or, equivalently, the homozygosity-calculated from the sample allele frequencies. The same test statistic may also be used to test other alternate hypotheses; for instance, to detect the presence of deleterious alleles. The homozygosity test has been criticized on a variety of grounds. The criticism we shall study in this paper is that the test may break down if the stationarity assumption is false. Nonstationarity may be a result of a change in population size, and we shall consider either a sudden increase or a sudden decrease in the size. We shall show how the correct significance levels may be found, and in a few numerical cases we shall compare them with the values calculated assuming stationarity. Some of the other criticisms of the test will also be briefly mentioned. It is well known that a change in the size of a population can influence the homozygosity of the population; for example, see NEI, MARUYAMA and CHAK- RABORTY (1975). The possibility, therefore, that such a change could disrupt the homozygosity test was pointed out by PERLOW (1979) and GRIFFITHS (1979). A closely related study has been made by MARUYAMA and FUERST (1985), and this will be mentioned in more detail below. Genetics April, 1986.

2 900 G. A. WATTERSON SIMULATION METHOD As the distribution of the homozygosity is rather complicated, even in the stationary case of strictly neutral mutations, we shall investigate the effect of nonstationarity of the distribution by simulation methods rather than by mathematical analysis. The simulation generates the allelic types of a random sample of n genes. The number of different alleles in the sample will be denoted by k, and the numbers of genes of these various types by nl,, n2, -, nk. Suppose that before generation 0, i.e., for generations - - a, -5, -4, -3, -2, -1, the population was of constant size No diploids, but that from generation 0 onward the size was NI. We take our random sample from generation t, t 3 0. Suppose also that up to generation 0 the mutation rate was uo and that after generation 0 the mutation rate was u1. We assume that all mutants are new alleles and that each generation is produced by random sampling from the gamete pool produced by the previous generation, subject to mutation but no selective effects. We define the scaled mutation parameters 80 = 4 Nouo, 81 = 4 Nlul. In the applications we have in mind, the mutation rates u1 and U:, are equal, so that Bo and B1 are in the same ratio as the population sizes No and NI. However, there may be other cases when the mutation rates differ on either side of the population size change. In order to simulate the sample numbers nl, n2, e -., nk, we first simulate the number of lines of descent in the sample, which is defined as the number of distinct ancestors at generation 0 having nonmutant descendants in the sample. We here denote this number by L(t), following GRIFFITHS (1980), and its probability distribution has been found recently [see TAVAR~ 1984 (5.5)]. Although it is straightforward first to tabulate this distribution and then to use it to convert uniform random variables into variables having this distribution, it is easier still to use the fact that L(t) is a pure death stochastic process [see TAVAR~ 1984 (4.7)]. This means that the number of generations during which the sample genes had j nonmutant ancestors, Tj for instance, has (approximately) an exponential distribution with mean We then choose L(t) = 1, say, as the smallest value of 1 (I = 0, 1,. -, n), such that n 2 Tj 5 t, j=l+i where I = n if Tn > t. And the T, can be simulated easily by Tj = - ~ Nlog I (Uj)/[j(j I)] where the U, are independent and uniformly distributed on [0, 13. Having obtained the value, e.g., 1, for L(t), the next step is to determine the allelic types in the sample. This is done in succession. Arbitrarily label the type

3 THE HOMOZYGOSITY TEST 90 1 of the first sample gene as Al. The successive genes, 2, 3,- - -, n will have types chosen from A,, Ab As, determined by the following scheme. For gene j (2 < j s Z), the type will either be of a new allelic type (not previously allocated to genes, 1, 2, -.., j - 1) with probability eo/(& + j - l), or it will be the same type as a randomly chosen one of the already determined genes. This procedure is equivalent to choosing a sample of 1 genes from EWENS distribution (e.g., see HOPPE 1984). Similarly, for gene j (I j 5 n), the type will be either a new allelic type with probability &/(el +j - 1) or it will be of the same type as a randomly chosen one of the j - 1 previously determined genes. That this procedure is correct follows, for instance, from (2.4) in WAT- TERSON (1984). It is considerably simpler to simulate sample genes as above than to simulate population allele frequencies; see GRIFFITHS and LI (1983) for a method of doing the latter. Having determined the allelic types of the genes in the sample, it is easy to count the number, k, of different alleles and the numbers nl, nz, -., nk of genes of each type present. The sample homozygosity is then calculated from the allele counts by the formula k 6 = (nj/n)? j- 1 The notation used here indicates that 6 is an estimate of the homozygosity in the population, at generation t. RESULTS AND DISCUSSION The homozygosity test of an observed f v+e consists of comparing that value with percentiles of the distribution of F, for the given sample size, n, and conditional on the observed number of alleles in the sample, k. The point of using this conditional distribution is that under the neutral, infinitely many alleles model at stationarity, the distribution does not depend on the mutation parameter, 01, applicable at the time. Unfortunately, if stationarity is not the case, and in particular if there had been a different mutation parameter Bo applicable for all generations prior to the size change t generations before the sample was taken, the true distribution of F given K depends on k, n, 2, and 82, even under the null hypothesis of selective neutrali_ty. Using the stationary distribution percentiles to assess significance levels of F might lead to inaccuracies. A very similar, but complementary, study of the above difficulty has been made by MARUYAMA and FUERST (1985), who investigated the effect of a population size change the other way around; that is, by studying the number of alleles in the sample for various values of the homozygosity. They found that, if a population decreases in size (so that dl < eo), then for a short time afterward there is a marked deficiency in the number of alleles in the sample compared with the number expected for that level of sample homozygosity, assuming a stationary population. Interpreting their result in the opposite di-

4 902 ;t G. A. WATTERSON FIGURE. 1.--Expected homozygosity after a population reduction. Upper curve: E($); lower curve: E(Flk = 3). Sample size = n = 500 genes, pre- and post-reduction mutation parameters Bo = 0.1, 8, = 0.01, k = number of alleles. t = time since reduction (generations/pn~). rection, for a given number of alleles in the sample, we would expect that a decrease in population size would cause an apparent deficit in the homozygosity for a short period thereafter. This is exactly what we do find in this paper. It is somewhat paradoxical, nevertheless, as we now explain. Following a reduction of population size, the mean homozygosity of a sample increases monotonically over time until it reaches a new equilibrium value; if Bo and B1 are the two values of the mutation parameter involved, the equilibrium value before the reduction is 1/n + (1 - l/n)/(l + eo), and after equilibrium is reestablished, it is l/n + (1 - l/n)/(l + 6,) [see WATTERSON 1984 (3.14), (3.16)]. However, if we study the mean of the sample homozygosity after a population reduction, conditional on a given number of alleles, it tends first to decrease and then very gradually to increase again. The equilibrium value is identical at each end of the time axis and is given by E(&) = l/n + (1 - l/n) cs: /sp l= 1 (see WATTERSON 1977), where S:) is a Stirling number of the first kind. A numerical example is shown in Figure 1, where we have simulated samples of n = 500 genes, with BO = 0.1 and 81 = At each of the time points t = 0, 0.01, 0.025, 0.05, 0.1, 0.5, 1.0, 2.0 (now measured in 2N1 generation units), some 10,000 replicate sa_mples were formed as described in the previous section, and for each, the F value was calculated as in (1). The upper curve in Figure 1 shows the way the mean of these 10,000 F values increases over time (between the theoretical values of and ). The lower curve shows the behavior when the averages are calculated only from those replicate samples which happen to have k = 3 alleles present. The numbers of such k

5 THE HOMOZYGOSITY TEST t FIGURE 2.-Expected qeik = 3). n = 500, eo = 0.01, e, = 0.1. homozygosity after population increase. Upper curve: E@); lower curve: replicates varied between 1168 for t = 0 and 52 for t = 2.0, so that not all points have the same accuracy. The theoretical mean >f P when k = 3 and n = 500, at stationarity, is (see WATTERSON 1978, table 2). The simula- tions agree closely with this at t = 0, but have not recovered to this value by t = 2. Still keeping Bo > 81, similar curves (not shown) apply for other values of the parameters, except that, of course, if there is only one allele in a sample Iz = 1, = 1 constantly, whatever the time t. By contrast, if a smaller population expands to a larger one, so that Bo < B1, the unconditional mean homozygosity decreases after the size change, whereas the conditional mean (for k > 1) first increases extremely rapidly and then slowly falls back to the initial equilibrium value. For instance when n = 500, Bo = 0.01 and B1 = 0.1, the same theoretical figures as above imply that E(@) should fall from to , whereas E(Flk = 3) should start and end at These facts are illustrated by simulations in Figure 2. When k = 3, the mean rises from a simulated value of 0.71 at t = 0 to 0.88 (approximately) almost immediately after. It is clear from the above that E(@lk) varies after a population size change, but it is not clear whether the nonstationarity has a serious effect in the tails of the distribution, which is where significance levels are important. In the simulation progtam, note has been taken of the observed percentiles of the distribution of F, for given values of k. In Figure 3 we show the results in the case when n = 500, k = 5, 80 = 1.0 and 81 = These correspond to an 100-fold reduction in population size at the change. The top curve in Figure 3 is the (simulated) 97.5 percentile for F, the central curve is the mean and the lower curve is the 2.5 percentile, all being conditional on k = 5. The percentiles clearly drop as a result of the population reduction and take a long time to recover their stationary (t = 0) values. If the stationary values are used as critical values for a two-sided 5% test of neutrality, then for substantial

6 904 G. A. WATTERSON t FIGURE 3.--Mean and percentiles of homozygosity. Upper curve: upper 91.5 percentile of k, given k = 5; middle curve: E(Flk = 5); lower curve: lower 2.5 percentile of F, given k = 5. n = 500, e, = 1.0, 81 = periods of time, beginning shortly after the population reduction, there will be too many significant results in the left tail (falsely suggesting heterozygote advantage) and too few significant results in the right tail (ignoring possibly real selection effects, such as heterozygote disadvantage or deleterious alleles). Notice that the true left-hand percentile is at, or only slightly above, the minimum value that P can take when k = 5; namely Ilk = 0.2. If two-sided tests are undertaken, there is some compensation between the errors in the tails, so that the overall significance level may not be far wrong, but one-tail tests would be more seriously in jeopardy. In Figure 4 we show the results for the same case as in Figure 3, except that now the population has increased its size. Here, we have n = 500, k = 5, Bo = 0.01 and 8, = 1.0. The increase in size has now raised the homozygosity percentiles and the mean very rapidly, but these begin to fall back soon after. The conclusions are now opposite to those made earlier. It is now the upper percentile that is at, or near, the maximum possible value for fi when k = 5 and n = 500 (namely, ), and if the stationary percentiles were used after the bottleneck, then too few low values and too many high values of fi would be judged significant. One wonders if the growth of populations in the recent past has caused us falsely to conclude the presence of deleterious alleles, whether by the use of the homozygosity test or by other tests which assume stationarity. For a given value of k, it seems that the effects of (1) the presence of deleterious alleles and (2) an increase in population size are similar, in that they both tend to incrase fi (e.g., see WATTERSON 1978, p. 412). In this context, I thank M. NEI (personal communication) for the following observation. Suppose that the homozygosity test is used as a one-sided test to detect a deficiency of homozygosity, this being of more interest than detecting an excess due to deleterious alleles. If the stationary distribution of P (given

7 THE HOMOZYGOSITY TEST t FIGURE 4.-Mean and percentiles of homozygosity. Upper curve: upper 91.5 percentile of P, given k = 5; middle curve: E@C = 5); lower curve: lower 2.5 percentile of F, given k = 5. n = 500, 00 = 0.01, 81 = 1.o. K) is used for the test, but the population has really been expanding (e.g., the human population), then the test is conservative. It will help to guard the investigator against rejecting neutrality by mistake. It is easy to write computer code to make simulations as above and to check cases other than those presented here. A listing of one such program is available on request from the author. The program can be used to find the correct percentiles for the homozygosity test after a size change, but it does assume that Bo, 81 and t are known, as well as k and n. In these circumstances, one might consider doing the test unconditionally on k. In this paper, we have used the appropriate order-statistic of the simulated values only as a point estimate of the true percentile. Other order-statistics could be used to set confidence limits for such percentiles, e.g., see section 32.8 of KENDALL and STUART (1961). OTHER CRITICISMS The homozygosity test has met with other criticisms, as well as its possible inaccuracy due to nonstationarity. For instance, NEI (1975, p. 171) wrote of EWENS test (and the same criticism could be leveled at the homozygosity test if it were performed as a two-sided test): This method is, however, very sensitive to deleterious alleles. If any of these alleles are included in the sample, the test would generally indicate nonneutrality of genes, even if they constitute a minor component of genetic variability. This criticism is really directed against taking the null hypothesis to be strict neutrality. Authors who make this criticism (e.g., OHTA 1977; LI 1979; KIMURA 1983, pp. 212 and 273) wish to test a different null hypothesis; namely, that

8 906 G. A. WATTERSON the genetic variation or the heterozygosity of a population is mainly due to neutral or almost neutral mutations, LI (1979). Various methods have been proposed to test this generalized neutrality, but as it is a vaguely worded hypothesis, no precise comparisons can be made to identify which of these methods has the most power to detect departures from this hypothesis. It is not even possible to ascertain the significance levels of observed data using these methods; strict neutrality has sometimes been used as the null hypothesis for this purpose. Presumably, population size changes would also influence these tests. Another criticism is that the homozygosity test uses data from only one locus, and so it cannot have much power to detect moderate amounts of selection. Instead, it is argued that it would be better to pool data from all available loci. This may well be true subject to certain provisos: 1. In pooling data from different loci, we would be testing for neutrality at all those loci simultaneously. We would be losing the ability to discriminate between neutral and selected loci. If this loss is of no interest, then pooling may be justified. 2. Whatever pooled test is devised, we would wish to be able to calculate the significance level of the data. Thus, we would wish to have a precise form of the null hypothesis and a precise knowledge of linkage effects for the loci being pooled. A start in this direction has been made by HEDRICK and THOM- SON (1985). A related criticism of the homozygosity test is that, even under strict neutrality at one locus, there will be recombination occurring between nucleotide sites within that locus, and EWENS sampling distribution does not apply exactly. However, HUDSON (1983) has concluded, on the basis of allowing recombination between four blocks of sites at a locus, that the critical values of the statistic used in the homozygosity test are only weakly affected by recombination at least for small values of the mutation and recombination parameters. It is an understatement to say that it would be difficult to find a test of generalized neutrality that is not also subject to these criticisms! I thank the referees for several helpful comments. LITERATURE CITED EWENS, W. J., 1972 The sampling theory of selectively neutral alleles. Theor. Pop. Biol. 3: GRIFFITHS, R. C., 1979 A transition density expansion for a multi-allele diffusion model. Adv. Appl. Probab. 11: GRIFFITHS, R. C., 1980 Lines of descent in the diffusion approximation of neutral Wright-Fisher models. Theor. Pop1 Biol. 17: GRIFFITHS, R. C. and W.-H. LI, 1983 Simulating allele frequencies in a population and the genetic differentiation of populations under mutation pressure. Theor. Pop. Biol. 23: HEDRICK, P. W. and G. THOMSON, 1985 A two-locus neutrality test with applications. Genetics 112: HOPPE, F. M., 1984 Poiya-like urns and the Ewens sampling formula. J. Math. Biol. 20:

9 THE HOMOZYGOSITY TEST 907 HUDSON, R. R., 1983 Properties of the neutral allele model with intragenic recombination. Theor. Pop. Biol. 23: KENDALL, M. G. and A. STUART, 1961 New York. KIMURA, M., 1983 bridge. The Advanced Theory of Statistics, Vol. 2. Hafner Press, The Neutral Theory of Molecular Evolution. Cambridge University Press, Cam- LI, W.-H., 1979 Maintenance of genetic variability under the pressure of neutral and deleterious mutations in a finite population. Genetics 92: MARUYAMA, T. and P. FUERST, 1985 Population bottlenecks and nonequilibrium models in population genetics. 11. Number of alleles in a small population that was formed by a recent bottleneck. Genetics 111: NEI, M., 1975 Molecular Population Genetics and Evolution. North-Holland Publishing Company, Amsterdam. NEI, M., T. MARUYAMA and R. CHAKRABORTY, 1975 The bottleneck effect and genetic variability in populations. Evolution OHTA, T., 1977 Extension to the neutral mutation random drift hypothesis. pp In: Molecular Evolution and Polymorphism, Edited by M. KIMURA, National Institute of Genetics, Mishima. PERLOW, J., 1979 The transition density for multiple neutral alleles. Theor. Pop. Biol. 16: TAVARB, S., 1984 Line-of-descent and genealogical processes, and their applications in population genetics models. Theor. Pop. Biol WATTERSON, G., 1977 WATTERSON, G., 1978 WATTERSON, G., 1984 Heterosis or neutrality? Genetics 85: The homozygosity test of neutrality. Genetics 88: Allele frequencies after a bottleneck. Theor. Pop. Biol Communicating editor: M. NEI