INCREASING SURVEY RESPONSE RATES: COMBINING EXPERIMENTAL MANIPULATIONS. Linda S. Buyer, PhD Kathleen J. Miller

Transcription

1 INCREASING SURVEY RESPONSE RATES: COMBINING EXPERIMENTAL MANIPULATIONS Linda S. Buyer, PhD Kathleen J. Miller Governors State University Office of Institutional Research 1 University Parkway University Park, IL PH: FAX: l-buyer@govst.edu Running Head: Increasing Survey Response Rates

2 Increasing Survey Response Rates. ABSTRACT Three experiments exploring the impact of combining incentives known to individually increase survey response rates are reported here. For the first experiment, two levels each of (a) cover letter personalization, (b) pre-survey gift, and (c) advance "handwritten" thank-you note were factorially combined to create eight experimental conditions. Results indicated that the advance "handwritten" thankyou note significantly increased the alumni survey response rate, significantly decreased the response time and also significantly increased the return rate of an additional form asking for employer contact information. The only other effect was a significant increase in the alumni response rate for alumni who received both a personalized letter and a pre-survey gift. For the second experiment, two levels each of (a) deadline (present/absent), (b) response modality (online/paper), (c) number of reminders (2/3), and (d) timing of reminders (3 days/2 weeks after previous mailing) were factorially combined to create 16 experimental conditions. Results indicated that the number of reminders interacted with the deadline and, separately, with timing of reminders, in both cases to increase response rates. There was a strong main effect of more reminders and a marginal main effect of deadlines. In the third experiment two incentives were factorially combined. A binary online response option (yes/no) was combined with three levels of number of reminders (2, 3 or 4) to create six experimental conditions. Results indicated that the online option interacted with reminders such that the three groups that had the online option did not respond differentially to different numbers of reminders (response rates varied between 19% and 20%) while the alumni without the online option responded positively to greater numbers of reminders (three and four reminders generated approximately 5% more responses than two reminders). Key Words: Survey Response Rates, Combining Manipulations, Personalization, Pre-survey gift, Handwritten notes, Number of Reminders, Timing of Reminders, Deadlines, Online response option INTRODUCTION As the demands on universities to demonstrate the value of their contribution to their students satisfaction and success in life have been increasing, the likelihood that a student will respond to a survey has been declining (Porter, 2004). Clearly these two trends are at odds. Equally clearly, there isn t any indication that the emphasis on accountability is going to decrease any time soon. If anything, the final report (2006) of the Secretary of Education s Commission on the Future of Higher Education suggests that assessment and accountability are critically important to the nation s future agenda for higher education. Thus, it is imperative for institutions of higher education to find ways to increase the rates at which target audiences respond to surveys. One of the ways that Governors State University (GSU) determines the value of its contributions to students satisfaction and success is by surveying its alumni. Between 1990 and 2006, GSU s alumni survey response rates declined from XX% to 9.5%. In 2007, we decided to see what we could do to reverse the downward trend. Pragmatically, we started by trying to identify the factors that were most likely to produce large increases in survey response rates. Porter s (2004) summary chapter Raising Response Rates: What Works? was very helpful in answering this question. His review of the literature indicates that: Page 2 of 47

3 Increasing Survey Response Rates. o Multiple contacts are one of the most effective mechanisms for increasing survey response rates, raising response rates by as much as 29% (Yammarino, Skinner, & Childers, 1991). o Incentives are only effective if they are provided along with the survey (pre-survey incentives) and are not conditional upon the person returning the survey (post-survey incentives). o Survey salience is an important factor in response rates, with more salient surveys producing as much as a 14.9% greater response rate (Groves, Singer, & Corning, 2000). o University or academic organization sponsorship is one of the more effective factors for increasing response rates (9% to 14% increases have been identified using metaanalyses, Fox, Crask and Kim, 1988; Goyder, 1982; Hebberlein and Baumgartner, 1978). Simply combining all of the incentives shown to increase survey response rates might possibly have put our office to either unnecessary work or expense to maintain any positive result that was observed because there would be no way to know whether one or more of the incentives could be eliminated without reducing the response rate. Thus, we decided to conduct a series of factorial experiments to identify the incentives or combinations of incentives that most increased our alumni survey response rates. Because GSU had accurate addresses for 666 of the pool of 9-year-out undergraduate alumni, and because we expected overall response rates to be relatively low, it was determined that three was the maximum number of two-level factors that could be manipulated and still retain reasonable power for the significance tests. (A 2x2x2 factorial design yields 8 experimental conditions which means that there would be approximately 666/8 = 83 individuals in each experimental condition). A review of GSU s current survey practices indicated that we were already using multiple contacts (one pre-survey notification, a post-survey reminder, a second survey mailing and a second post-survey reminder) and thus were presumably already enjoying the benefits to be derived from multiple contacts. Thus, we decided to look to other factors to help us increase our response rates. The literature with regard to pre-survey incentives shows increases of more than 10% for very modest pre-survey incentives. For example, a $1 prepayment increased survey responses by approximately 24% (Heberlein and Baumgartner, 1978), while $0.50 increased rates by 18% (Yammarino, Skinner and Childers, 1991). Because the research shows good effects for pre-survey incentives, the authors decided to include a pre-survey gift among the factors we manipulated in the first experiment. Because GSU is a public institution (which makes giving money away extremely complicated), we decided to use a non-monetary pre-survey incentive in the form of a lapel pin with a GSU logo on it as the first factor in our experiment. Page 3 of 47

4 Increasing Survey Response Rates. Because we were planning on surveying the group of alumni (9-year-out undergraduates) mandated by our state regulatory body, the Illinois Board of Higher Education, there didn t seem to be much that the authors could do to effectively manipulate salience. Thus, we again decided to look further for factors to help us increase response rates. The last factor Porter (2004) points to as effective in increasing response rates is sponsorship by a university or academic organization. All GSU s alumni surveys are university sponsored, so we again are presumably already enjoying whatever benefits could be derived from such sponsorship. At this point, the authors decided to turn to Dillman s (2007) tailored design model of survey response rates, which has its basis in social exchange theory, in our effort to identify incentives likely to increase response rates. Social exchange theory posits that human behavior is motivated by the response the behavior is expected to elicit. Dillman proposes that survey response rates are the result of a combination factors (rewards, costs and trust) affecting the likelihood that an individual will fill out a survey. Rewards are the perceived gains to be obtained from filling out the survey, costs are the perceived losses (time, money, effort, etc.) associated with filling out the survey and trust is the expectation that in the long run the rewards of doing something will outweigh the costs. (Dillman, 2007). Among the ways that Dillman proposes that rewards can be increased is by showing positive regard. One way to he proposes doing so is to personalize contact materials instead of sending generic ones. Because the advent of newer technologies has made it easy to personalize contact materials, we decided to include personalization as the second factor to test with regard to increasing survey response rates. Another factor that Dillman points to as a way to provide rewards is to say thank you in advance. He recommends phrases such as we appreciate very much your help or many thanks in advance be added to correspondence. Because this is also a very easyto-effect manipulation, the authors decided to thank the potential survey participants in advance as our third manipulation. It was decided to do this in as personal a way as possible, which was to have the president of the university provide a hand-written thankyou note to be included with the survey cover letter. Because Dodd and Markwiese (1986) have shown that handwritten signatures and photocopied signatures produced the same response rates, the authors decided to have the president s thank-you note scanned and electronically transferred to the cover letters that were sent with the first survey. Dillman states that one way to reduce the perceived cost of filling out the questionnaire is to avoid subordinating language and inconvenience. GSU s existing protocol already included simple, straightforward language in the cover letter and sending postage-paid envelopes in which the surveys could be returned. Thus, given the three factors available to us to manipulate, we decided to focus our efforts on increasing the perceived rewards of filling out the survey and on increasing the trust that the survey recipients had that the effort involved in filling out the survey was justified and assume that we are already acting to reduce the perceived costs. Page 4 of 47

5 Increasing Survey Response Rates. Dillman points out that enclosing tangible rewards with the survey is not only a way to provide rewards but suggests that it is also a way of increasing the recipient s trust that filling out the survey is a worthwhile activity. Thus, our first manipulation (pre-survey gift of a lapel pin) should serve two of the functions Dillman s tailored design model predicts will positively affect response rates: it should both be perceived as a reward and it should increase trust. Because it serves a dual function in Dillman s model, while the other two factors are assumed to serve single functions, we predicted that the main effect of the pre-survey incentive would be greater than the main effects of either personalization or the pre-survey thank you alone. We didn t expect that combining the three two-level factors we had decided to manipulate (personalization, handwritten thank-you and pre-survey incentive) would produce effects that were simply additive. Rather, we expected to see interactions among the factors such that two factors in combination produced better survey return rates than either factor alone. Similarly, we expected the three factor combination to produce better results than either of the two factor combinations. We also expected to find that none of the factors alone would outperform the two- and three-factors combinations, and that all groups receiving an incentive would outperform the control group (who received materials identical to those we had used the preceding year). Page 5 of 47

6 Increasing Survey Response Rates. METHOD PARTICIPANTS All GSU alumni who graduated with a bachelor s degree in Spring/Summer 1996, Fall 1996 or Winter 1997 (class of 97) and for whom the Alumni Relations Office thought it had accurate addresses (N = 666) were included in the experiment. The 666 alumni were randomly assigned in approximately equal numbers to the eight experimental conditions. After mailing the pre-survey notification postcard, it was determined that 12 addresses were incorrect, yielding a total of 654 survey recipients. The survey recipients ranged in age from 30 to 77, with an average age of Two-thirds (67.7%) of them were female. The majority of the survey recipients were Caucasian (74.9%) or African- American (18.8%). No other ethnicity was represented by more than 3.5% of the survey recipients. EXPERIMENTAL CONDITIONS Eight experimental conditions were created by factorially crossing the three independent variables: personalization (generic v personalized cover letter), pre-survey gift (GSU lapel pin v no gift) and pre-survey thank-you ( handwritten note from the university president thanking alumni in advance for participating v no note). Personalization of the standard cover letter mailed with the survey included addressing the alumna/alumnus by name and included the name of the program the individual graduated from and the degree he/she earned. Personalization also included an acknowledgement of the individual s return to GSU (if he or she had registered at GSU after graduating) and an acknowledgement of the individual s contributions to GSU since graduating (if any). Too few surveys were returned to analyze the effects of more/less personalization within the personalization condition. Thus, degree of personalization will not be addressed further. Approximately fifty percent of the 654 alumni for whom we had correct addresses received personalized cover letters with their surveys (N = 328) and the remainder received the generic letter (N = 326). Similarly, GSU lapel pins were included with the cover letter of approximately 50% of the survey recipients (N = 331) while the remainder did not receive a lapel pin (or any mention of one; N = 323). The pre-survey thank-you note from the president was handwritten originally but was scanned and transferred to the cover letter electronically. We made no effort to make the handwritten note look handwritten on the letters (for example, the note was printed in the same black ink as the letter) because the literature suggests that the impact of electronic and handwritten additions to cover letters is the same (see, for example, McKenzie-McHarg, et al., 2005). Approximately 50% of the alumni had the handwritten thank-you note transferred to their cover letter (N = 330) while the remaining half did not (N = 324). A sample of the cover letter showing the fields that were used to personalize the letters is shown in Appendix A. Page 6 of 47

7 Increasing Survey Response Rates. The eight experimental conditions and the number of alumni assigned to each is shown in Table 1 below: Surveying Process ******************** Insert Table 1 about here ******************** Pre-survey Notification Postcard: All alumni were sent identical pre-survey notification postcards that informed them that an alumni survey would arrive within the next few weeks. The text of the pre-survey notification is shown in Appendix B. Of the 666 pre-survey notification postcards that were mailed, 12 were returned to the university with no forwarding address. No additional materials could be sent to these individuals, thus yielding a total sample size of 654. First Survey Mailing: Approximately three weeks after the pre-survey notification postcard was mailed, the initial surveys were mailed. This mailing included either a personalized or generic cover letter, a survey, a pre-printed postage-paid return envelope, an alumni contact form (see Appendix D) and, only for those in the pre-survey gift condition, a lapel pin embossed with the GSU logo. All of the generic letters were identical, e.g., the salutation was Dear 1997 Graduate. The degree of personalization for the personalized letters depended on the alumnus/a s post-graduation involvement with GSU. All of the personalized letters included: Alumni s full name, address, city, state, and zip code Salutation which included the graduate s first name Type of degree earned (e.g., B. A.) and name of the program which awarded it If relevant, the personalized letters might also have mentioned: Post-graduation re-enrollment at GSU Contributions made to GSU For alumni assigned to the pre-survey thank-you condition, the cover letter, whether personalized or generic, contained an electronically transferred copy of a thank-you note from the university president. For alumni assigned to the pre-survey gift condition, the letters included mention of the included GSU lapel pin. Both of these additions to the letters can be seen in Appendix A. The surveys were four double-sided pages long. A copy of the survey is shown in Appendix E. The surveys were comprised of twenty-seven questions clustered under the following five section headings: Section I: Opinions about Educational Outcomes Section II: Satisfaction with Governors State University Section III: Further Education Page 7 of 47

8 Increasing Survey Response Rates. Section IV: Employment Section V: Comments The survey included both closed- and open-ended questions. The closed-ended questions included multiple-choice, two-category forced choice, fill-in-the-blank and Likert-scale response options. The alumni contact information form requested the following information about the graduate on behalf of the Alumni Relations Office: Name Mailing address Numbers for work, home, and mobile phones address Name of employer Job title Length of employment Years in current position. To better enable us to contact employers of GSU alumni, we also requested contact information for the graduate s supervisor. This included: Supervisor s name Supervisor s job title Supervisor s mailing address, Supervisor s phone number(s) Reminder Postcard: Two weeks after initial survey mailing, we mailed a reminder postcard to those that had not returned the survey. The text of the reminder postcard is shown in Appendix E. Second Survey Mailing: Two weeks after the reminder postcard, second surveys were mailed to those who still had not returned the survey. All of the second survey mailing packets were identical regardless of the original condition assignments. The second mailing included a second, generic cover letter (see Appendix F), a second copy of the survey, another copy of the alumni contact form, and a postage paid return envelope. The only difference between the first and second surveys that were mailed out was the presence of a small x in the lower right hand corner of the back page of the survey. This x allowed the authors to determine whether a returned survey was from the first or second mailing. Variables Tracked: Dates that each of the postcards and surveys were mailed were recorded. As the alumni responded, we tracked whether the returned survey was from the first or second mailing and the date it was received. We also recorded whether the alumni contact form was returned and whether it came with a first or second survey. Page 8 of 47

9 Increasing Survey Response Rates. RESULTS Overall, the results of the experiment were quite promising. The 2007 return rate was 22.8% (149 surveys returned), which was more than double the 2006 return rate of 9.5%. Effects on Response Rate: A logistic regression was computed to determine whether some combination of the incentives could explain the high return rate. The model tested included survey returned/not returned as a binary dependent variable, along with the main effects of the three incentives, the three two-way interactions and the three-way interaction of the incentives as independent variables. The results indicated that all of the main effects were significant (smallest Wald = 14.99, SE =.25, df = 1, all p <.001). The three-way interaction was not statistically significant and is not presented below as a result. Interaction of Personalized Letter and Gift The main effects of cover letter personalization and the gift are superseded by their significant interaction (Wald = 10.31, SE =.45, df = 1, p.001). The data producing the interaction are shown in Figure 1 below: ********************* Insert Figure 1 about here ********************* Figure 1 makes the source of the interaction clear. Presented alone, the personalized letter and the lapel pin both produced a lower survey return rate (respectively, 21.1% and 18.3) than did presenting neither (24.1%). Please note that the no personalized letter/no lapel pin group should not be confused with the control group: some of these individuals received the handwritten thank-you while others did not. Given that personalization alone and the lapel pin alone both reduced the response rate relative to not having either of them, combining them should be expected to further reduce the response rate. However, contrary to expectation, when the personalized letter and lapel pin were presented together, the response rate was 3.4% higher than when neither was presented (27.5% % = 3.4%). Another way to examine this effect is to compare the proportions of respondents from the groups above to the proportion from the control group (who received no inducements at all). 21.5% of the control group returned their surveys. Z-tests comparing the proportion of respondents receiving only the lapel pin (.183) or only the Personalized letter (.211) to the proportion of respondents in the control group (.215) were not significant (both z 1). A one-tailed z-test comparing the proportion of respondents who received both (.275) to that of the control group (.215) produced a z-score of 1.89 (p <.05). This indicates that significantly more surveys were returned from those who received both a personalized letter and a lapel pin than from those in the control group. Again, the interaction is clearly due to the unexpected positive effect of combining personalization with the gift of a lapel pin. Page 9 of 47

10 Increasing Survey Response Rates. Main Effect of Thank-you Note: The remaining effect to be considered is the main effect of the handwritten thank-you note. The data on which the effect is based are shown in Figure 2. Clearly, those who received the note were more likely to return the survey than those who did not (24.8% % = 4.1%). ********************* Insert Figure 2 about here ********************* Again, it is worthwhile to compare these return rates to that of the control group using the z-test for proportions. The proportion of respondents who did not receive the note (.207) did not differ from that of the control group (.215; z < 1). The proportion of respondents who received the note (.248) was marginally greater than that of the control group (z = 1.46, one-tailed p =.07). Effects on Time to Return Survey: To further test the effects of the three incentives on the survey recipients, a stepwise regression analysis was conducted with days to return a survey from the date that the first surveys were mailed out as the dependent variable and the three dummy-coded incentives as independent variables. The only effect the program entered into the model during the analysis was the thank-you note (t =-3.39, df = 1,SE = 3.23, p.001). Surveys came back an average of 38.3 days after the first surveys were mailed from those that did not receive a note and an average of 11 days faster (27.3 days) from those who did receive a note. Figure 3 below shows the difference. ********************* Insert Figure 3 about here ********************* Further exploration of this result was accomplished by comparing these results to those from the control group. Days to return a survey for both groups were compared to the average for the control group (32.4) using t-tests. Both of these tests were significant (No note: t = 2.3, df = 66, p <.05; Note Included: t = -2.5, df = 81, p <.05). Thus, the two levels of handwritten thank-you note not only differed from one another but both also differed from the control group (with those who received no note responding significantly slower and those who received the note responding significantly faster than the control group). Effects on Returning the Alumni Contact Form: The third dependent variable included in the experiment to measure the effects of the three incentives was an alumni contact form that alumni were asked to complete and to submit to the Office of Alumni Relations Office so that they and/or their supervisors could be contacted by their degree program to distribute employer surveys. This form was included in the mailing with the first survey (see Appendix D). We did not receive any alumni contact forms from alumni who did not also return a survey. A logistic regression analysis, identical to the first except that the dependent variable was whether or not the alumni contact form was returned, was conducted. The only significant effect was the main effect of the handwritten thank- Page 10 of 47

11 Increasing Survey Response Rates. you note (Wald = 5.79, SE =.51,df = 1, p <.05). The data involved in this effect are shown in Figure 4 below. Survey respondents who received the thank-you note with their letter were more than twice as likely to return the alumni contact form than were respondents who didn t receive the thank-you note (31.7% v. 14.9%). ********************* Insert Figure 4 about here ********************* Once again, it is worth considering how these groups differ from the control group. Twenty-three-point-six percent (23.6%) of the control group s respondents returned the alumni contact form. The proportions returning the alumni contact form from the different levels of the note condition (.149 and.317) were compared to the proportion of the control group respondents returning the contact form (.236) using one-tailed z-tests. Both of the z tests were significant (smallest z = 1.68, p <.05). Thus, survey respondents who received the handwritten note were more likely to return their alumni contact form and respondents who didn t receive a note were less likely to return the alumni contact form than the control group. DISCUSSION Considering all of the results described above, the most effective incentive for this group of GSU alumni was the inclusion of the handwritten advance thank-you note. All three dependent measures showed a positive effect of the note. The handwritten thank-you not only significantly increased the percent of surveys that were returned, it also significantly increased the speed with which they were returned, and the likelihood that they would be accompanied by the information requested for the alumni contact information form. Had we simply adopted, wholesale, the likely effective incentives, we would not have been able to tease out this result. Our use of factorial exploration of incentives allowed us tease apart the effects. There was no three-way interaction, and we found that that the handwritten thank-you, alone, more effectively increases response rates than in combination with either more reminders or a pre-survey gift. Worth pointing out again here is that the comparisons involving the control group indicate that, collectively, groups receiving incentives other than the handwritten note, responded at a lower rate than the group that received a Dear 1997 graduate letter and no incentives. There was a positive effect of combining the personalized letter with the lapel pin on the percentage of surveys that were returned. This interaction did not impact returning the alumni contact form or affect the speed with which surveys were returned. In light of the fact that this effect did not impact all of the dependent measures and the expense of including the lapel pins, the interaction doesn t seem worth us pursuing it at this time. However, others may wish to investigate the combined effects of reward with other factors. It is possible that there is a positive result to be gained. Page 11 of 47

12 Increasing Survey Response Rates. EXPERIMENT 2 Based on the results from the 2007 experiment, the handwritten thank-you notes were included as part of the base condition for the 2008 experiment. Because it seemed ludicrous to do otherwise in an era when most mail is personalized, we also included personalization of the cover letters as part of the 2008 control condition even though there was only minimal evidence from the first experiment that this is an effective incentive (and perhaps its ubiquity is why). Between 2007 and 2008, Governors State University decided to survey all 1-year-out and 5-year-out alumni every year. From an experimental point of view, this was a very welcome change because it meant that we could increase the number of factors we combined in each experiment. Thus, in 2008 we decided that we would factorially combine four binary variables into 16 experimental conditions (2 x 2 x 2 x 2 = 16). Both because the new university president very much wanted us to provide an online response option, and because we thought that social response theory would predict that providing such an option would decrease the recipents subjective impressions of the cost of filling out the survey (no need to find a mail box, etc.) as well as increase their sense of our positive regard for their time and effort, we decided to create a binary online response option (present/absent) as one of the four incentives manipulated in the present experiment. As stated earlier, multiple contacts have been empirically shown to have a strong postive effect on response rates (Yammarino, Skinner, & Childers, 1991). Although GSU was already contacting the survey recipients as many as five (5) times (pre-survey postcard, survey 1, post-survey-1 reminder postcard, survey 2 [if necessary], and a post-survey-2 reminder postcard [again, only if necessary]), the authors decided to see if more contacts would the increase the response rate. We again created a binary manipulation, this time by adding a second post-survey-1 reminder postcard. Thus, half of the recipients would potentially be contacted by GSU six times in total, while the other half would only be contacted a maximum of five times. Also summarized earlier was the strong positive impact on response rates of increasing the salience of the survey to the recipients (Groves, Singer, & Corning, 2000). We hypothesized that, in addition to sending more post-survey reminders, sending the postsurvey reminders with a shorter delay than the two weeks we used in the first experiment might suggest to the alumni that GSU considers the survey very important and thus increase its salience to them.we decided to use a lapse of three days for the short reminder interval and two weeks, as in the previous experiment, for the long interval. Thus half the receipients received their first post-survey-1 reminder postcard almost immediately on the heels of receiving their first survey, while the remainder didn t receive theirs until two weeks later. Theoretically, deadlines should increase response rates because limited response opportunities should make the opportunity to respond to the survey seem more valuable. Page 12 of 47

13 Increasing Survey Response Rates. This should increase the perceived salience of the survey as well as decrease the perceived cost to the respondents. According to social exchange theory (Dillman 2007), perceptions of both increased salience and decreased cost should result in increased response rates. The literature contains mixed evidence regarding this hypothesis (see, for example, Porter s 2004 summary of the effects of deadlines on response rates, p. 6), with meta-anlyses generally showing no effect of deadlines (see for example, Fox, Crask, & Kim, 1988; Yammarino, Skinner, & Childers, 1991; Yu and Cooper, 1983). However, it may be that deadlines interact with other variables and it is possible that the effect of deadlines is being masked by the presence of other variables when they are not explicitly included in the analyses. Thus, we included a binary deadline/no deadline incentive in order to explicitly explore its interactions with the online response option, as well with the number and timing of reminder postcards. Half of the cover letters that were sent were modified to include a deadline for returning the survey and half were not. In summary, the incentives we decided to manipulate in 2008 were the presence/absence of a deadline for submission, the existence/absence of an online response option and the number (2 v 3) and timing (3 days v 2 weeks) of reminders to respond to the survey. METHOD PARTICIPANTS All GSU alumni from the Graduating Classes of 2002 (graduated in Spring/Summer 2001, Fall 2001, or Winter, 2002) and 2006 (graduated in Spring/Summer 2005, Fall 2005, or Winter 2006) for whom the Alumni Relations Office thought it had accurate addresses (N = 2,664) were included in the experiment. The 2,664 alumni were randomly assigned in approximately equal numbers to the 16 experimental conditions. After mailing the pre-survey notification postcard, it was determined that 38 addresses were incorrect, yielding a total of 2,626 survey recipients. The survey recipients ranged in age from 22 to 87, with an average age of Two-thirds (69.6%) of them were female. The majority of the survey recipients were Caucasian (59.5%) or African-American (29.1%). No other ethnicity was represented by more than 4% of the survey recipients. EXPERIMENTAL CONDITIONS Sixteen experimental conditions were created by factorially crossing the four independent variables: (a) deadline (present/absent), (b) response modality (online/paper), (c) number of reminders (2/3), and (d) timing of reminders (3 days/2 weeks after previous mailing). Personalization of the standard cover letter mailed with the survey included addressing the alumna/alumnus by name and included the name of the program the individual graduated from and the degree he/she earned. Page 13 of 47

14 Increasing Survey Response Rates. The 16 experimental conditions and the number of alumni assigned to each is shown in Table 2 below: ******************** Insert Table 2 about here ******************** Surveying Process Pre-survey Notification Postcard: All alumni were sent identical pre-survey notification postcards that informed them that an alumni survey would arrive within the next few weeks. First Survey Mailing: Approximately two weeks after the pre-survey notification postcard was mailed, the initial surveys were mailed. This mailing included a personalized cover letter either with or without a deadline and with or without the option to respond online, a survey, and a pre-printed postage-paid return envelope. Subsequent to GSU s decision to survey graduate alumni as well as undergraduate alumni, it also reviewed and modified the existing survey. The new surveys for 2008 were slightly different from the one used in 2007 (many of the survey s questions are mandated by the State of Illinois and cannot be altered). In 2008, they were comprised of either 36 (undergraduate alumni) or 37 (graduate alumni) questions that still clustered under the same five section headings: Section I: Opinions about Educational Outcomes Section II: Satisfaction with Governors State University Section III: Further Education Section IV: Employment Section V: Comments As before, the surveys included both closed- and open-ended questions, with closedended question types including multiple-choice, two-category forced choice, fill-in-theblank and Likert-scale response options. Examples of the revised surveys can be seen online at: Reminder Postcards: Either two weeks or three days after initial survey mailing, we mailed a reminder postcard to those that had not returned the survey. Alumni in the three-reminder condition received another postcard, either three days or two weeks after the first, again reminding them to please respond. (Alumni in the two-reminder condition received one reminder after they received each survey. Alumni in the three-reminder condition received two reminders after receiving their first survey and one reminder after receiving their second.) Second Survey Mailing: Two weeks after the second round of reminder postcards, second surveys were mailed to those who still had not returned the survey. All of the survey mailing packets were identical regardless of the original condition assignments. The second mailing included a Page 14 of 47

15 Increasing Survey Response Rates. second, personalized cover letter, a second copy of the survey, an alumni contact form, and a postage paid return envelope. Once again, the only difference between the first and second surveys that were mailed out was the presence of a small x in the lower right hand corner of the back page of the survey. Last Reminder Postcard: All non-respondents received yet another reminder postcard after receiving their second survey. Variables Tracked: Dates that each of the postcards and surveys were mailed were recorded. Postcards and surveys returned with address corrections or no forwarding address were also monitored. As the alumni responded, we tracked whether the returned survey was from the first or second mailing and the date it was received. We also recorded whether the alumni contact form was returned. RESULTS Overall, the results of the experiment were again quite promising. The overall return rate for 2008 was 27.1% (712 surveys returned), which was well above 2007 s overall return rate of 22.8%. Due to an error in setting up the capture of the completed online surveys, the data from surveys submitted online during the first two weeks after the first surveys were mailed were lost. The impact of that incentive won t be analyzed further in the current experiment. Thus, this experiment ended up with 3 factors rather than the intended 4 factors Effects on Response Rate: A logistic regression was computed to determine whether some combination of the incentives could explain the high return rate. The model tested included survey returned/not returned as a binary dependent variable, along with the main effect of the three manipulated variables (again, the online option was not analyzed because we lost some of the online responses from this condition due to a programming error) as well as the three two-way interactions and one three-way interaction as independent variables. The results indicated that the only main effect that was statistically significant was for number of reminders (3 Reminders = 29.1%, 2 Reminders = 25.2%; Wald = 12.83, SE =.12, df = 1, p <.001). Although there was no main effect of timing of reminders, it interacted with both number of reminders (Wald = 8.83, SE =.15, df = 1, p <.005) and the existence of deadline (Wald = 9.71, SE =.14, df = 1, p <.005). There was no indication of a three-way interaction nor of a two-way interaction between number of reminders and the existence of a deadline. The significant interactions are shown in Figures 5 and 6. Page 15 of 47

16 ****************************** Insert Figures 5 and 6 about here ****************************** Increasing Survey Response Rates. As can be seen from the Figures 5 and 6, the shorter reminder delay (3 days) was involved in the highest return rates for both interactions. When combined with three reminders, 3-day reminders produced a response rate of 32.3%. When combined with the deadline, they produced a response rate of 29.4%. Although the three-way interaction was not significant, the alumni who had a deadline and were sent three reminders each with a 3-day delay had the arithmetically highest return rate of any of the eight groups reported on in this experiment (35%). Effects on Time to Return Survey: To further test the effects of the three manipulations on the survey recipients, a regression analysis was conducted with days to return a survey from the date that the first surveys were mailed out as the dependent variable and the three dummy-coded incentives and all of their interactions as independent variables. There were significant main effects of timing of the reminders and deadline on the length of time it took to return a survey (respectively, F = 10.16, df = 1, MSe = , p.001; F = 8.42, df = 1, MSe = , p <.005). The average number of days to return a survey was 31.9 for those who were not given a deadline and 27.5 for those who were. Surveys were returned more quickly by those who received reminders after 3 days (mean = 27.2 days) than those who didn t receive their reminders until 2 weeks had elapsed (mean = 32.3). There was no main effect of the number of reminders. The only other significant result was for the interaction of deadline with number of reminders (F = 8.07, df = 1, MSe = , p.005). These results are shown in Figure 7 below: ********************* Insert Figure 7 about here ********************* Apparently, the impact of deadlines is limited to the alumni who were in the tworeminder condition. They responded an average of 9 days sooner if they were given a deadline than they did if they were not. Alumni in the three-reminder condition responded equally quickly regardless of whether they were given a deadline. (The reader should recall, here, that the three-reminder group received two reminders to complete the first survey while the two-reminder group only received one reminder to complete that survey and didn t receive their second reminder until after they received the second survey.) Effects on Returning the Alumni Contact Form: The third dependent variable included in the experiment to measure the effects of the three incentives was an alumni contact form that alumni were asked to complete and to submit to the Office of Alumni Relations Office so that they and/or their supervisors could be contacted by their degree program to distribute employer surveys. This form was only included in the mailing with the second survey. Only alumni who sent back a survey sent back the alumni contact form. A logistic regression analysis, identical to the first except that the dependent variable was Page 16 of 47

17 Increasing Survey Response Rates. whether or not the alumni contact form was returned, was conducted. There were no significant effects. DISCUSSION With the handwritten note and personalized contact materials serving as the basis for the additional incentives used in this experiment, most of the effects on response rate that were observed in this experiment resulted from interactions. Although no main effect on response rates of timing of reminders or of the deadline was found, timing of the reminders did exert an effect on response rate via an interaction with the presence/absence of the deadline as well in interaction with number of reminders. The short reminder delay (3 days) produced higher response rates when combined either with a deadline (29%) or with greater numbers of reminders (32%) than the long reminder delay did. The condition that produced the single highest response rate (35%) was the one in which alumni were given a deadline, were sent 3 reminders and received those reminders on the shorter (3 day) reminder schedule. The finding that number of reminders had a main effect on the survey return rate in addition to interacting with timing of the reminders, suggests that it is worth exploring to see whether continuing to increase the number of reminders will further improve response rates. Timing of reminders did have a main effect with regard to the speed with which surveys were returned, with the short reminder delay producing faster responses. Having a deadline also produced faster responses than the absence of the deadline did. Interestingly, increasing the number of reminders to three apparently negates the effect of the deadline. Both groups that received three reminders returned their surveys after approximately 29 days, whereas there was a nine-day difference for the two-reminder groups (deadline = 25.6 days, no deadline = 34.6 days). Page 17 of 47

18 Increasing Survey Response Rates. EXPERIMENT 3 The results of the 2008 experiment encouraged us to include 3-day reminders and deadlines for all recipients of the 2009 alumni surveys. Thus, the base condition for the 2009 surveys included personalization of contact materials, a handwritten note from the president, a deadline and reminders sent after three days. Because three reminders significantly increased response rates in 2008 over two reminders, we decided to explore further the shape of the relation between number of reminders and survey response rates. One third of the alumni each received two, three, or four reminders. Partly because the only alumni who would actually receive all four reminders were those who had successfully ignored all previous efforts to get them to respond, we expected to find an asymptotic curve indicating diminishing returns as we increased the number of reminders from three to four. The 2009 experiment also gave one half of the survey recipients the option to log on to a computer and respond to the survey online. The three levels of number of reminders combine with the binary online option (present/absent) created six experimental conditions. METHOD PARTICIPANTS All GSU alumni from the Graduating Classes of 2003 (graduated in Spring/Summer 2002, Fall 2002, or Winter, 2003) and 2007 (graduated in Spring/Summer 2006, Fall 2006, or Winter 2007) for whom the Alumni Relations Office thought it had accurate addresses (N = 2,921) were included in the experiment. The 2,921 alumni were randomly assigned in approximately equal numbers to the six experimental conditions. After mailing the pre-survey notification postcard, it was determined that 74 addresses were incorrect, yielding a total of 2,847 survey recipients. The survey recipients ranged in age from 22 to 77, with an average age of Two-thirds (71.0%) of them were female. The majority of the survey recipients were Caucasian (55.0%) or African-American (30.8%). No other ethnicity was represented by more than 3% of the survey recipients. EXPERIMENTAL CONDITIONS The 2,847 alumni who received mailings were assigned to the experimental conditions as follows: ******************** Insert Table 3 about here ******************** Surveying Process Pre-survey Notification Postcard: All alumni were sent identical pre-survey notification postcards that informed them that an alumni survey would arrive within the next few weeks. Page 18 of 47

19 Increasing Survey Response Rates. First Survey Mailing: Approximately two weeks after the pre-survey notification postcard was mailed, the initial surveys were mailed. This mailing included a personalized cover letter either with or without the option to respond online, a survey (the surveys used in this experiment were identical to those used in Experiment 2), an alumni contact form and a pre-printed postage-paid return envelope. Post-Survey-1 Reminder Postcards: Three days after the initial survey mailing, we mailed a reminder postcard to those that had not returned the survey. Alumni in the three- and four-reminder conditions received another postcard, three days after the first one, again reminding them to please respond if they had not yet done so. Second Survey Mailing: Two weeks after the second round of post-survey-1 reminder postcards, second surveys were mailed to those who still had not returned the survey. The survey mailing packets were identical, regardless of the original condition assignments. The second mailing included a second, personalized cover letter, a second copy of the survey, a second alumni contact form, and a postage paid return envelope. Once again, the only difference between the first and second surveys that were mailed out was the presence of a small x in the lower right hand corner of the back page of the survey. Post-survey-2 Reminder Postcards: Three days after the second survey mailing, we mailed a reminder postcard to all alumni that had not yet returned a survey. Only alumni in the four-reminder condition received another postcard, three days after the first post-survey-2 reminder postcard, once again asking them to please respond if they had not already done so. Variables Tracked: Dates that each of the postcards and surveys were mailed were recorded. As the alumni responded, we tracked whether the returned survey was from the first or second mailing and the date it was received. We also recorded whether the alumni contact form was returned. Results: As in Experiments 1 and 2, there were three dependent variables in this experiment: response rate, time to respond, and return of an alumni contact information form that was mailed with the surveys. Overall, the response rate was 20.4%, which is much lower than the 2008 return rate (27.8%). Effects on Response Rate: A logistic regression was computed to determine whether there were main effects or an interaction of the two experimental factors. The regression model tested included survey returned/not returned as a binary dependent variable, along with the main effects of the online option and the number of reminders as well as the Page 19 of 47

20 Increasing Survey Response Rates. two-way interaction of the independent variables. The results indicated that only the interaction was significant (Wald = 5.41, SE =.03, df = 1, p <.05). As Figure 8 shows, for alumni who had the online response option, there was no effect on response rates of more than two reminders to fill out the survey (return rates ranged between 18.6% and 20.0%), whereas the return rates for those who were not offered the online response option, increased from 18.1% to 23.7% (a 5.6% increase) as the number of reminders increased from 2 to 4. ********************* Insert Figure 8 about here ********************* Effects on Time to Return Survey: To further test the effects of the manipulations on the survey recipients, a regression analysis was conducted with days to return a survey (from the date that the first surveys were mailed out) as the dependent variable and number of reminders and online response option as the independent variables. There was a marginally significant main effect of number of reminders (F =2.58, df = 2, MSE = 309.8, p =.076). Surveys came back an average of 19.6 days after the first surveys were mailed from those that received two reminders, 19.9 days for those who received three reminders and an average of 23.5 days for those who received four reminders. This effect is superseded by the significant interaction of the number of reminders with the online option (F =3.045, df = 2, MSE = 309.8, p <.05). As is apparent in Figure 9, alumni in the online response condition, took the same number of days to respond regardless of how many reminders they received (mean calculated across number of reminders = 21.1) whereas the average number of days to respond increases (by an average of 8.2 days) as the number of reminders increases for the groups with no online option. ******************** Insert Figure 9 about here ******************** Effects on Returning Alumni Contact Form: The third dependent variable included in the experiment to measure the effects of the manipulations was the alumni contact form used in Experiments 1 and 2. This form was included in the mailing with both the first and second surveys. Very few alumni who did not send back a survey sent back the alumni contact form (N = 3). A logistic regression analysis, identical to the first except that the dependent variable was whether or not the alumni contact form was returned, was conducted. The only significant effect was again the interaction of the online option with the number of reminders (Wald = 4.38, SE =.04,df = 1, p <.05). The data involved in this effect are shown in Figure 10 below. Survey respondents who did not have an online option returned more contact forms when they received more reminders (Δ = 4.3%) while respondents who did have the online option were less likely to return the alumni contact form if they received more reminders (Δ = -2.5%). *********************** Insert Figure 10 about here *********************** Page 20 of 47

21 Increasing Survey Response Rates. DISCUSSION Clearly, there is no benefit to offering alumni the online response option. Although there was no significant difference in the response rates of those offered/not offered the online response option, the arithmetic difference indicated that the group not offered the online option, had a higher response rate (online option = 19.1%, no online option = 21.7%), which is certainly not what we expected. In addition, of the 263 surveys that were returned by those with the online response option, only 49 (18.6%) were online submissions, so the reduction in manual processing obtained by offering the online option was minimal. Not only is the online option not having a positive effect on any of the three dependent variables, in interaction with the number of reminders, it is negating the positive effects of increasing the number of reminders. Thus, part of the explanation for the overall reduction in response rate in this experiment as compared to last year s is that half of the alumni received an incentive that actually seems to have suppressed the positive effect of sending reminders. Why the online response option is negating the effect of number of reminders is difficult to explain. This will require further exploration. Within the no online response option condition, more reminders produces better results. With regard to response rates, three and four reminders produce approximately equal response rates, and both produce more responses than two reminders. Thus, our hypothesis that the response rate curve would become asymptotic was borne out. However, there was a linear relation between increasing numbers of reminders and time to respond as well as returning the alumni contact forms. Overall, the combination of four reminders with no online response option produced the best outcomes. This group returned the most surveys and alumni contact forms, as well as sending their surveys back the fastest. GENERAL DISCUSSION Overall, the results suggest that survey response rates are very amenable to manipulation. However, the results aren t always what we expected. We were very surprised to find no positive results from offering an online response option. We were even more surprised by its suppression of the positive effect of increases in number of reminders. We were also surprised to find that a handwritten thank-you note from the president of the university was a more effective, incentive than was personalizing the contact materials or than sending a pre-survey gift both of which seem to the authors to show greater regard for the recipient. Not surprisingly, across experiments, the results indicate that (a) more reminders to complete a survey are better, (b) reminders sent after a shorter lapse are more effective, and (c) as we hypothesized at the outset, the effects of deadlines are moderated by other variables. Page 21 of 47

22 Increasing Survey Response Rates. The decline in response rates between 2008 and 2009 deserves further analysis. The first step in doing so will be to include all of the five incentives we think can be combined effectively (handwritten note, personalization, deadline, timing of reminders, and number of reminders) into a single factorial experiment. This will allow us to determine if one or more incentives are having an unsuspected suppressor effect on one or more of the others. A second step is to follow up with non-respondents and see if a factor like survey exhaustion is a possible cause. Because so many of Governor State s undergraduate alumni return to GSU for a graduate degree [approximately 40% in the last five fiscal years], students might be receiving very similar one-year-out undergraduate and graduate alumni surveys within a very short period of time, thus reducing response rates. Page 22 of 47

23 Increasing Survey Response Rates. REFERENCES (2006). A Test of Leadership: Charting the Future of U.S. Higher Education. Jessup, MD: ED Pubs, Education Publications Center, U.S. Department of Education. Carini, R.M., Hayek, J.C., Kuh, G.D., Kennedy, J.M., & Ouimet, J.A. (2003). College student responses to web and paper surveys: Does mode matter? Research in Higher Education, 44, Dillman, D. A. (2007). Mail and Internet Surveys: The Tailored Design Method, (Second Edition). Hoboken, NJ: John Wiley & Sons. Dodd, D.K. and Markwiese, B.J. (1986). Survey response rate as a function of personalized signature on cover letter. Journal of Social Psychology, 127, Fox, R.J., Crask, M.R., and Kim, J. (1988). Mail survey response rate: a meta-analysis of selected techniques for inducing response. Public Opinion Quarterly, 52, Goyder, J.C. (1982). Further evidence on factors affecting response rates to mailed surveys. American Sociological Review, 47, Groves, R.M., Singer, E., & Corning, A.D. (2000). Leverage-saliency theory of survey participation. Public Opinion Quarterly, 64, Heberlein, T.A. & Baumgartner, R. (1978). Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature. American Sociological Review, 43, McKenzie-McHarg, K., Tully, L., Gates, S., Ayers, S., and Brocklehurst, P. (2005). Effect of survey response rate of hand written versus printed signature on a covering letter: randomized controlled trial. BMC Health Services Research, 5, p.52 Mehta, R. & Sivadas, E. (1995). Comparing response rates and response content in mail versus electronic surveys. Journal of the Market Research Society, 37, Mowen, J.C. and Cialdini, R.B. (1980). On implementing door-in-the-face compliance technique in a business context. Journal of Marketing Research, 17, Porter, S. R. (2004). Overcoming Survey Research Problems. New Directions for Institutional Research, Number 121, pp Shannon, D.M. & Bradshaw, C.C. (2002). A comparison of response rate, response time, and costs of mail and electronic surveys. Journal of Experimental Education, 70, Page 23 of 47

24 Increasing Survey Response Rates. Suskie, L. A. (1996). Questionnaire Survey Research: What Works. Tallahassee, FL: Association for Institutional Research. Yammarino, F.J., Skinner, S.J., & Childers, T.L. (1991). Understanding mail survey behavior: a meta-analysis. Public Opinion Quarterly, 55, Yu, J. & Cooper, H. (1983). A quantitative review of research design effects on response rates to questionnaires. Journal of Marketing Research, 20, Page 24 of 47

25 APPENDICES

26 APPENDIX A <Insert Date Here> «First_Name» «Middle_NameInitial» «Last_Name» «Address_1_Line1» «City», «State» «Zip» Dear «First_Name», It s been nine years since you graduated from GSU with a «Deg_Earn_Txt» from our «Major_Txt» program <If true, insert and we re pleased that you returned to GSU to continue your education. >. <If true, insert As a token of our appreciation, enclosed is a lapel pin that we hope you will wear proudly. > <If true, insert We appreciate the contributions you have made to GSU in the past. There is no better indication of your continuing interest in your alma mater. We would like to ask for your assistance in another way. > To help us meet the needs of our students, we are currently studying the impact a GSU education has on our graduates career paths. By completing the enclosed survey, you can help us ensure that our programs provide students with outstanding career preparation. It won t take more than 10 minutes to complete the survey. Because the «Major_Txt» program would like to survey employers of their graduates to see how satisfied they are, we also ask that you enter the contact information for your employer and your job title, and mail that back in the smaller of the enclosed envelopes. Your responses will be anonymous. At no time will any responses be identified with individual graduates. Only information that allows us to meaningfully group the responses we receive is contained in the survey. For example, your survey is marked to indicate that you graduated from the «Major_Txt» program with a «Deg_Earn_Txt». Only summary results will be made available to the «Major_Txt» program and to Career Services. The data that we collect from our graduates is also sent to the Illinois Board of Higher Education, as is data from all of the other public institutions, so that our graduates responses can be compared to those of graduates from other institutions. (We d really like to have the highest response rate in the state.) If you have any questions, please call either Kathy Miller ( ) or Linda Buyer ( ) in the Office of Institutional Research; or you can questions to k-miller@govst.edu or l- buyer@govst.edu. Thanks so much for your help with our effort to continuously improve the quality of the education that we offer. Your responses will have a real and direct impact on the future development of the «Major_Txt» program and on the services offered by GSU. Sincerely, Stuart I. Fagan President Page 26 of 47

27 APPENDIX B Sample Pre-notification postcard: Page 27 of 47

28 Survey of Bachelor s Degree Recipients in 1997 APPENDIX C GOVERNORS STATE UNIVERSITY Office of the President Please correct the information on the label below if necessary. SECTION I: OPINIONS ON EDUCATIONAL OUTCOMES: For each statement below, please check the box that best describes your thoughts regarding your experience at GSU. 1. While a student at Governors State University, I... a. Acquired knowledge and skills applicable to my present career b. Improved my ability to write clearly and effectively c. Developed effective speaking skills d. Improved my ability to comprehend written material e. Became (more) effective using computers and technology f. Developed an appreciation of art, music, and/or drama g. Enhanced my ability to accept and understand people with different beliefs and values h. Developed the skills to learn independently i. Gained insight into my own values and ethical standards j. Realized that learning is a lifelong process k. Made use of student services (career counseling, tutoring, fitness center, student life programs) l. Utilized alternative learning technologies (telecourses, correspondence study, computerassisted instruction) m. Acquired a greater interest in international issues and events n. Became more flexible in accepting change o. Received a well-rounded education p. Acquired a strong understanding of my major field of study q. Learned how to work well in groups r. Improved my quantitative skills (e.g., math, statistics) Strongly Agree Agree Neutral Disagree Strongly Disagree Does Not Apply Please continue

29 APPENDIX C 2. Listed below are some abilities often mentioned as important for personal and professional success in the 21st century. Please check one box for each ability listed. In your view, how helpful were your Governors State University experiences in.... a. helping you to develop better your critical thinking ability? b. helping you to develop better your sense of ethics? c. contributing to a greater understanding of people with different backgrounds, habits, values, appearances, and abilities? d. helping you to become a more active citizen? e. improving the quality of your life aside from financial benefits? Extremely Helpful Very Helpful Moderately Helpful Slightly Helpful Not Helpful SECTION II: SATISFACTION WITH GOVERNORS STATE UNIVERSITY. Please circle the letter of the best response to the following questions. 3. What is your present attitude toward Governors State University? a. Strongly positive d. Somewhat negative b. Positive e. Negative c. Somewhat positive f. Strongly negative 4. If you could start your education over again, would you have attended Governors State? a. Yes, definitely c. Probably not b. Yes, probably d. Definitely not 5. What is your present attitude toward your undergraduate degree major? a. Strongly positive d. Somewhat negative b. Positive e. Negative c. Somewhat positive f. Strongly negative 6. If you could start your education over again, would you have taken the same degree major? a. Yes, definitely c. Probably not b. Yes, probably d. Definitely not 7. Would you recommend GSU to another student? Yes, definitely Yes, probably Probably not Definitely not Why or why not? 8. For each program or characteristic listed below, please check the box that matches your perception of the quality of the program / characteristic at GSU. Program / Characteristic Excellent Good Fair Poor a. Program Advising (including Student Study Plans) b. Financial Aid Counseling c. Availability of courses I needed for my program of study d. Class size e. Scheduling of classes (days / times) f. Library Services g. Student Development (Tutoring) h. Accessibility for students with disabilities i. Student Life (extracurricular programs, fitness center, pool, student clubs) j. Equipment needed for program (computers, lab equipment, technology for use in the classroom) k. Quality of Teaching l. Career Counseling m. Alumni Relations Not Applicable Please continue

30 APPENDIX C 9. The evaluation items below on educational practices at GSU will provide feedback to help the university improve undergraduate education. For each item, please check the box that matches your evaluation of your program as a whole. In my major degree program, I felt that a. full-time professors were accessible outside of class ( via office visits, telephone, , video-conferencing, etc.) b. adjunct faculty were accessible outside of class ( via office visits, telephone, , video-conferencing, etc.). c. I was expected or required to work cooperatively with other students on projects, homework, and assignments. d. instructors encouraged me to challenge my own ideas, the ideas of other students, and those presented in readings and other course materials. e. faculty used appropriate teaching activities to help me learn. f. faculty expectations for the quality of student work were high. g. instructors provided me with timely feedback on my performance. h. I was expected to spend the necessary time and effort to come to class prepared. Very Often Often Sometimes Infrequently Never SECTION III: FURTHER EDUCATION: Opportunities for lifelong learning can take many avenues. Please tell us what avenues you have pursued after earning your degree from Governors State. Check ( ) the appropriate boxes in the table below. AFTER completing my baccalaureate degree at Governors State in 1994, my pursuit of lifelong learning has taken the following avenues: 10. I am currently pursuing the following:... ( check all that apply) a. Second Bachelor s degree Full-time Part-time b. Academic Master s degree (M.A., M.S., M.Ed., etc.) Full-time Part-time c. Professional Master s degree (M.B.A., M.S.W, M.F.A, M.H.A., M.P.A., etc.), or Educational Specialist degree Full-time Part-time d. Degree in Medicine (M.D., D.O.) Full-time Part-time e. Health Professional degree (dentistry, pharmacy, podiatry, optometry, etc.) Full-time Part-time f. Theology / Divinity degree Full-time Part-time g. Law degree (L.L.B., J.D.) Full-time Part-time h. Doctoral degree (Ph.D., Ed.D., D.A., D.B.A., etc.) Full-time Part-time i. Other degree (please specify ) Full-time Part-time j. Additional college courses in my field Yes No k. Additional college courses for personal interests Yes No l. Training/educational opportunities at work Yes No m. Continuing Education program (CEU s) Yes No n. Community education opportunities Yes No o. Museum/Art seminars/workshops Yes No p. Other (please specify ) Yes No 13. In general, how well did your bachelor s degree prepare you for your additional degree program? 11. In the past five years, I have pursued: ( check all that apply) 12. After my bachelor s, I have earned: ( check all that apply) a. Very well b. Well c. Adequately d. Inadequately e. Poorly f. Very poorly 14. Please list the colleges or universities from which you have earned a degree or which you have attended since earning your bachelor s degree. Institution Year(s) Attended Degree Please continue

31 SECTION IV: EMPLOYMENT: Please circle the letter of the best response to the following questions: 15. Are you currently employed (including self-employed)? a. Yes, full time c. No, but am seeking employment b. Yes, part time d. No, am not seeking employment APPENDIX C 19. How closely related is your current job to your bachelor s degree major? a. Closely related c. Unrelated by my choice b. Related d. Unrelated not by my choice IF YOU CIRCLED C OR D ABOVE, PLEASE SKIP TO SECTION V. 16. In general, how well did your bachelor s degree prepare you for your present job? a. Very well d. Inadequately b. Well e. Poorly c. Adequately f. Very poorly 17. How would you classify your primary employer? a. Self-employed or private practice b. Business (Industrial, commercial or service) c. Professional firm (e.g., engineering, law) d. College or university e. Elementary or secondary school (including cooperatives) f. Health agency (e.g., hospital, clinic) g. Federal, state, or local government h. Armed Services i. Human service agency (non-profit) j. Human service agency (proprietary) k. Non-profit (non-government) l. Other (please specify) 20. For how many different employers (companies, firms, agencies, schools, etc.--not individual bosses ) have you worked since you earned your bachelor s degree? a. One c. Three to six b. Two d. More than six 21. Describe your occupation in the space below: 22. What is your job title? 23. Please enter your annual earned income before taxes on your current job (nearest thousand). $, What is the name and address of your employer? 18. How satisfied are you with your current job? a. Very satisfied d. Somewhat dissatisfied b. Satisfied e. Dissatisfied c. Somewhat satisfied f. Very dissatisfied 25. Please provide the ZIP code of the place where you work: SECTION V: COMMENTS 26. We would appreciate information about promotions, awards, academic scholarships, or other honors and recognition you may have received since you graduated from GSU with your bachelor s degree 27. Please use the space below for any comments you might have regarding positive or negative aspects of your undergraduate experience at Governors State University. What advice would you offer to current students majoring in the field of your bachelor s degree? If you need more space, please attach a separate page. Please return this completed survey in the enclosed postage paid envelope. Surveys\Alumni\Alum94-5\ba94_5sur3: THANK YOU FOR SHARING YOUR VIEWS!

32 APPENDIX D Governors State University Alumni Contact Information (Please Print or Type) Graduate Information: Name Address City State Zip Phone Numbers: Home Work Mobile Employer Length of Employment (years) Job Title Years in Current Position Supervisor s Information: Supervisor Name: Supervisor Title: Supervisor Work Address: Supervisor Work Phone Supervisor City State Zip The Office of Alumni Relations wishes to thank you for filling out this form. Please include it in the post-paid envelope provided.

33 Sample Follow-up postcard: APPENDIX E

34 APPENDIX F April 06, 2007 Dear 1997 GSU Graduate: About four weeks ago, we mailed you a survey asking about your experiences at GSU. We have not yet heard from you and would really like to know what you have to tell us. The tapestry being woven about 1997 graduates is missing some very important threads: yours. You are part of the fabric that makes up GSU. You may think that as only one graduate your responses and opinions won't mean much, but that is not true. Your responses represent some of the threads in the tapestry that emerges from weaving together the many responses from different individuals. Without your responses the tapestry is incomplete and threadbare in spots. Only you can provide your thoughts that will give us a more complete tapestry of what works well at GSU and what needs improving. For your convenience, we've enclosed another copy of the survey and a self-addressed, postage-paid envelope in which to return it. Won't you take a few minutes to fill out the survey? The input you give will not only help us better serve current and future students of Governors State University but will also help us satisfy a requirement of the Illinois Board of Higher Education. Thank you again for your time. Sincerely, Stuart I. Fagan President SF/kjm

35 Tables and Figures TABLE 1: Number of Surveys Delivered to and Returned by Alumni (by Condition) Personalized Letter Lapel Pin Thank-you Note Number Surveys Delivered Number (Percent) Surveys Returned 17 (21.5%) No No No 79 (Control Group) No No Yes (26.5%) No Yes No (15.7%) No Yes Yes (21.0%) Yes No No (18.3%) Yes No Yes (24.1%) Yes Yes No (27.5%) Yes Yes Yes (27.6%) TOTAL (22.8%)

36 FIGURE 1: Effect of Interaction between Personalization of Cover Letter and Pre-survey Gift (Lapel Pin) on Survey Response Rate (Data Collapsed across Note Conditions)

37 FIGURE 2: Response Rates for the two levels of Handwritten Thank-you Note (Data Collapsed across Levels of Personalization and Pre-survey Gift)

38 FIGURE 3: Average Number of Days to Return Survey (First or Second) from Date First Survey Mailed for Levels of Thank-you Note (Data Collapsed across Levels of Personalization and Pre-survey Gift)