Results from the Ride the Road with TxDOT Customer Satisfaction Pilot Program

Transcription

1 Results from the Ride the Road with TxDOT Customer Satisfaction Pilot Program By Tina Geiselbrecht Assistant Research Scientist Texas Transportation Institute Doug Oxley Postdoctoral Research Associate Bush School of Gov t & Public Service Kristine Miller Associate Transportation Researcher Texas Transportation Institute Tim Lomax Research Engineer Texas Transportation Institute Eric Lindquist Associate Research Scientist Bush School of Gov t & Public Service Curtis Beaty Associate Research Engineer Texas Transportation Institute December 2010

2 Table of Contents Page List of Figures... ii List of Tables... iii Introduction... 1 Route and Survey Overview... 1 Recruitment... 3 Methods of Analysis... 3 Descriptive and Inferential Statistics... 3 Statistical Software... 4 Indices... 4 Charts and Graphs... 4 Results... 5 Demographic Characteristics... 5 Customer Satisfaction... 7 Knowledge of TxDOT... 9 Values, Priorities, and Importance of TxDOT Services to Customers Meeting Expectations while Riding the Road with TxDOT Comparison of Ride the Road with TxDOT results and the Texas Maintenance Assessment Program Recommendations Regarding Ride the Road with TxDOT Next Steps Appendix i

3 List of Figures Page Figure 1 Distribution of Participants by Race Figure 2 Distribution of Participants by Party Figure 3 Customer Satisfaction by Knowledge of TxDOT Figure 4 Level of Knowledge by Education Level Figure 5 Participants Views on How TxDOT s Effort Should Be Allocated Figure 6 Participants Choices when Faced with Dilemmas between Two Opposing Types of Effort Figure 7 Roadway Sections Figure 8 How Well Expectations Are Being Met for All Road Categories Figure 9 Average Expectations Regarding Pavement Conditions Figure 10 Average Expectations Regarding Signs Figure 11 Average Expectations Regarding Stripes Figure 12 Average Expectations Regarding Design Figure 13 Average Expectations Regarding Appearance Figure 14 Average Expectations Regarding Overall Condition ii

4 List of Tables Page Table 1 Customer Satisfaction by Gender Table 2 Customer Satisfaction by Race Table 3 Customer Satisfaction by Age Table 4 Customer Satisfaction by Education Table 5 Customer Satisfaction by Party Table 6 Importance Rankings Table 7 Customer Priority Rankings Table 8. Rating Scores from TxMAP and Focus Group Participants Table 9. Rating Places from TxMAP and Focus Group Participants iii

5

6 Introduction The Texas Department of Transportation (TxDOT) has a number of internal programs and tools to measure and track the scientific and engineered aspects of many elements of transportation. The results from these programs and tools aid TxDOT in determining current and future projects to address deficiencies, improve safety, and extend asset lifecycles. More recently, TxDOT developed a series of performance measures across a number of functional areas that go beyond the scientific and engineered items. Aligned with TxDOT s strategic plan, these performance measures provide a reporting mechanism for TxDOT to hold themselves accountable to the citizens of Texas. One aspect that is missing from the internal programs and performance measures is an assessment of the public s perception of how TxDOT is doing. In other words, TxDOT wants to know if the public s perception of Texas transportation system matches the technical measures the agency uses in the various functional areas that TxDOT is responsible for providing and maintaining. This assessment of customer satisfaction is essential to TxDOT, and the best way to find out whether TxDOT s customers the citizens of Texas are satisfied is to ask them. The Texas Transportation Institute (TTI) developed a Customer Satisfaction Program that will bring the public s priorities and expectations on certain issues into the decision making process of the department. A guidebook, Tell TxDOT, outlines the various techniques to collect public input, and the contents and logistics required for the effective use of each information gathering method. As part of the Customer Satisfaction Program, TTI developed the Ride the Road with TxDOT Survey Program. It is a demonstration survey developed to study customer expectations about specific roadway elements and services. In the Ride the Road with TxDOT survey road trip, TTI escorted participants over a pre defined route where individuals were asked to grade various elements of the roadway infrastructure. The survey included elements such as pavement/road surface, signage, pavement markers, and other elements. The initial Ride the Road with TxDOT survey was used to the best approach for this type of data collection method and how best to conduct these surveys in a broader scope of an on going customer satisfaction program. TTI will use the experiences from the pilot Ride the Road with TxDOT survey to refine questions and procedures that will improve the process for future road trips the department may wish to conduct at other locations across the state. Route and Survey Overview Ride the Road with TxDOT is a type of demonstration survey that uses personal experience in a relatively controlled environment to gauge levels of customer satisfaction about various roadway features. This pilot program provides TxDOT designers, planners, and maintenance personnel with a unique opportunity to understand the opinions of travelers who use TxDOT roadways. While TxDOT regularly conducts peer reviews with other state departments of transportation, this was the first undertaking of this kind by TxDOT with members of the general public. 1

7 TTI researchers worked with the TxDOT Austin District Director of Maintenance and the Maintenance Administrator to determine a route that would allow participants to experience several types of roadway features and several condition levels. The route segments were in eastern Travis County. The Director of Maintenance felt that these roadway segments would offer a variety of roadway features and differed enough for general public participants to recognize the difference without technical expertise. Additionally, most of these segments were included on a route developed for a state department of transportation peer review conducted one month prior. Some of these segments are also rated using TxDOT s Texas Maintenance Assessment Program (TxMAP). Using segments that are also scored with other technical grading components will facilitate a comparison between technical measures and the general public s perception. With guidance from an internal TxDOT Task Force advising the development of the TxDOT Customer Satisfaction Program, the researchers developed a survey instrument to use on the Ride the Road with TxDOT trips. The survey gathered input about participants expectations of the roadway features and also gauged the participants knowledge of TxDOT. Finally, the survey was designed to ask participants what priorities they feel are important for the department. To accomplish these tasks, the survey was divided into three general sections. 1. Part one of the Ride the Road with TxDOT survey examined participants knowledge of the department. This was determined by asking about various services and asking about whether or not it was provided by TxDOT. Participants were questioned about which entities were responsible for building and maintaining different types of roadways including city streets and interstates. Participants knowledge of transportation funding was also gauged by asking if they knew how much the state gas tax is, whether it is a flat tax or a sales tax, and how it is calculated. Participants were also asked to identify, from a list, what other funding mechanisms are currently available to TxDOT. Several questions asked the participants to indicate the level of importance and priority for various TxDOT actions. 2. Part two of the survey involved driving participants in mini vans over the predetermined route and asking them to indicate whether or not various elements met their expectations as a driver on the roadway. There were eight roadway sections that varied in length from 1.2 to 4.7 miles. The total route took approximately 75 minutes to complete. Participants were allowed time to complete the survey questions on one section before beginning the next. Similar questions were asked in each section although some sections contained unique elements that the participants were asked about (such as the smoothness of a railroad crossing). 3. Finally, after returning from the road survey, the participants were asked to complete a third section of the survey that asked demographic questions. The survey is shown in its entirety in the Appendix. 2

8 It is difficult to infer statistical validity from the small sample size in this pilot Ride the Road with TxDOT exercise. As more Ride the Road with TxDOT surveys are conducted, the overall survey sample size will grow. This will make the sampling technique more representative of Texas demographics. In addition, during the participant recruiting process for this pilot effort, researchers strived to achieve a broad cross section of the population. Recruitment For this pilot Ride the Road with TxDOT effort, researchers wanted 12 individuals that were willing to ride in a vehicle with unfamiliar people for 90 minutes and be able to read and write while the vehicle was in motion. Potential focus group participants were asked questions to identify their ability to participate in the survey. One potential participant was excluded on the basis of the answers provided to these questions. Participants were recruited by ing past focus group participants and by placing an advertisement in the Event Gigs category on the Austin, Texas, Craigslist website. TTI also sent notices to people who had previously participated in focus groups for other research projects; however no responses were received from past participants before the recruiting time window closed. The Craigslist ad resulted in a potential participant pool of 45 individuals. Individuals were instructed to phone the TTI Austin office to indicate their availability and interest in participating. When the person contacted TTI, they were asked a series of interview questions as part of a screening criterion. Researchers were then able to select a broad representation of individuals to participate. The demographic make up of the participants is described later in this document. Methods of Analysis This section describes some of the analytical methods utilized in the production of the results from the Ride the Road with TxDOT survey. Descriptive and Inferential Statistics Statistics can be classified into two types. Descriptive statistics provide information about a group s characteristics based upon observations made about each individual in the group. Inferential statistics use descriptive statistics and additional techniques to infer about the characteristics of a large population based upon the characteristics of a smaller random sample of that population. Descriptive statistics include measures of central tendency (means, medians, and modes) and measures of dispersion (ranges and standard deviations) regarding a group. Measures of central tendency provide information on the characteristics of the typical individual in the group. Sometimes they are referred to as the average characteristics (mean, median, and mode are all considered to be averages). Measures of dispersion provide information on how widely spread the individuals in the group are on a particular measure. Inferential statistics use a randomly chosen group to infer the characteristics of the population. We can know the probability that a particular random group s characteristics accurately reflect the population characteristics. At the time of this writing, we are unable to confidently use inferential 3

9 statistics on the Ride the Road with TxDOT program data because of the small number of participants that have participated. When we have a larger sample group, we can use tests that compare the means of sub groups to other sub groups within a sample, or we can learn about which factors cause the overall level of satisfaction to be as it is for particular individuals. Statistical Software This analysis has used Stata 11.1 SE software produced by StataCorp LP as the primary tool for calculation and graphing. In addition, Microsoft Excel has been utilized for some of the data management tasks. Indices In some circumstances, indices were created based upon the answers to a range of related questions. For example, in order to calculate an overall assessment of the score on Road Appearance, we calculated the mean of all eight questions related to appearance for each of the road sections (i.e., each road section contributed 1/8th part of the index that was reported). Another example of the use of an index was the value calculated to grade knowledge about TxDOT. Some of the questions from the group with correct/incorrect answers were valued more than others. The questions below were given a weight of 2 (rather than 1) in the index to differentiate the score of participants with correct knowledge about important aspects of TxDOT s operations (for example, it is more important to know road maintenance responsibilities than to correctly identify the agency responsible for removing debris from the road). Is providing bus service a TxDOT service? Who is responsible for building Interstates? Who is responsible for building county roads? Who is responsible for building city streets? Who is responsible for maintaining Interstates? Who is responsible for maintaining US or State Highways? Who is responsible for maintaining county roads? Who is responsible for maintaining city streets? The state gas tax is: What other funding is used for state transportation projects? (Given additional weight if vehicle registration fees were checked). Charts and Graphs Three types of charts have been used in the results section. A scatterplot was used to represent the data showing customer satisfaction by their overall knowledge of TxDOT. A scatterplot shows each observation on two different variables. Each observation receives its own dot. Bar charts were used to compare the average values of a variety of groups. Bar charts display the average value for some category. A special type of bar chart is known as a histogram. A histogram is a bar chart that shows the number of items of data at each level of a particular variable. The distribution of participants by party (Figure 2) is an example of a histogram. 4

10 Box plots provide information about the distribution of data around the median value. The middle line inside the box represents the median, and the edges of the box represent the 25th and 75th percentiles. The ends of the whiskers attached to the box indicate the upper and lower statistically valid values. These ends are a maximum distance of 1.5 times the size of the box (e.g., longer boxes mean longer whiskers). Any dots that might be shown outside of the whiskers represent potential statistical outliers from the data. Results The Ride the Road with TxDOT program investigated the attitudes and beliefs that customers hold regarding the responsibilities of TxDOT. The survey studied two components of participant reactions: (1) their knowledge of and attitudes toward TxDOT and its services, and (2) their expectations and opinions while driving across eight defined sections of roadway maintained by TxDOT. The demographic characteristics were also gathered to assist the analysis. Demographic Characteristics The focus group recruitment process, described in an earlier section, developed a group of participants with a range of age, race, education, and ideological/partisan perspectives. Five men and five women participated while two participants did not disclose their gender. Four were high school graduates, three had some college credit, and five had a bachelor s degree. The participant s age at the time of the survey spanned from 18 to more than 65. Figures 1 and 2 demonstrate the distribution of participants by race and party identification, respectively. 5

11 Figure 1 - Distribution of Participants by Race. Figure 2 - Distribution of Participants by Party. 6

12 Customer Satisfaction The participants were asked, Overall, how satisfied are you with the services TxDOT provides? The scale for response included: Very Dissatisfied (1). Somewhat Dissatisfied (2). Somewhat Satisfied (3). Very Satisfied (4). The results are described below in a few of the many possible ways to analyze focus group responses. A broader analysis with a greater ability to explore relationships between beliefs, demographics, and expectations will be possible with more participants and a greater understanding of the audience interests. Table 1 demonstrates the overall customer satisfaction by gender. Participants of both genders responded equally with an average value of 3.0, which reflects Somewhat Satisfied as the overall level of satisfaction, although the variation in responses was slightly higher for females (as shown by the higher standard deviation value). Table 1 - Customer Satisfaction by Gender. Gender Average Standard Deviation Frequency Male Female Total Some differences were found for participants of different races. These results appear in Table 2. In particular, those whose race is Black were closer to being Very Satisfied overall (average = 3.7) with TxDOT, while those who are Hispanic were Somewhat Dissatisfied overall (average = 2.0). Whites (non Hispanic) were close to being Somewhat Satisfied (average = 3.22). Table 2 - Customer Satisfaction by Race. RACE Average Standard Deviation Frequency Black or African American Hispanic White (non Hispanic) Other Total

13 There was a variation in satisfaction by age, but no clear pattern emerges from the 11 responses. Participants of different levels of education seem to be equally satisfied with TxDOT. These results appear in Tables 3 and 4. Table 3 - Customer Satisfaction by Age. AGE Average Standard Deviation Frequency 18 to to to to to to to and older Total Table 4 - Customer Satisfaction by Education. EDUCATION Average Standard Deviation Frequency High School Some College Bachelor s Degree Total

14 Participants who were Democrats, Republicans, and Others had similar views on the overall level of satisfaction with TxDOT, while one of the two Independents was more dissatisfied with TxDOT as shown by the lower average value and the high standard deviation in Table 5 below. Table 5 - Customer Satisfaction by Party. PARTY Average Standard Deviation Frequency Democratic Independent Republican Other Total Knowledge of TxDOT The specific services that TxDOT provides may not be well known by the general public. The study included questions designed to measure the overall level of knowledge of the types of services that TxDOT provides (and does not provide) and questions about how TxDOT is funded. All of these questions have correct answers. An index of the knowledge of TxDOT was constructed to reflect the number of times an answer was given correctly. The questions below were given a weight of 2 (rather than 1) in the index to identify those participants with correct knowledge about important aspects of TxDOT s operations. Is providing bus service a TxDOT service? Who is responsible for building Interstates? Who is responsible for building county roads? Who is responsible for building city streets? Who is responsible for maintaining Interstates? Who is responsible for maintaining US or State Highways? Who is responsible for maintaining county roads? Who is responsible for maintaining city streets? The state gas tax is: What other funding is used for state transportation projects? (Given additional weight if vehicle registration fees were checked). A maximum score of 37 was possible on the knowledge index. The range of scores was from 6 to 34 with an average of 22 and a standard deviation of 6.9. The customer satisfaction and knowledge of TxDOT scores are graphed in Figure 3. 9

15 All of the participants who scored below the average of 22 on the knowledge index also rated TxDOT s service as either Somewhat Satisfied or Very Satisfied. Those with better than average knowledge of TxDOT s services were more varied in their responses with some high knowledge participants reporting dissatisfaction with TxDOT. If confirmed by further focus groups, this would point to the disturbing finding that the more people know about TxDOT, the less satisfied they are with the services. Figure 3 - Customer Satisfaction by Knowledge of TxDOT. 10

16 Figure 4 suggests an inverse relationship between knowledge of TxDOT and education level. Those who are generally better educated know less about TxDOT as indicated in the box plot diagram. The box indicates the range of responses between the 25th percentile and the 75th percentile, with the median value noted by the horizontal line inside the box. The lines extending above and below the boxes show the maximum and minimum values for survey responses. This type of plot provides readers with a powerful evaluative tool. The median value shows the midpoint of responses and the size of the box shows how much the opinion varies. The whiskers (lines above and below the boxes) extend out to the values deemed statistically reliable. If there are any outlier responses they are noted with dots. Figure 4 also shows that while the groups responded with about the same average value, the high school graduate group had much greater variation in their knowledge. Figure 4 - Level of Knowledge by Education Level. Note: See the analysis methodology for more discussion about box plots. 11

17 Values, Priorities, and Importance of TxDOT Services to Customers We measured TxDOT customers attitudes about the importance of different services that TxDOT provides and we also measured the importance of particular priorities to the participants. The Importance ranking of services is presented in Table 6, and the Priority rankings are presented in Table 7. The scale for the Importance rankings included: 1 Not at all Important. 2 Somewhat Important. 3 Important. 4 Very Important. For the top ranked item, Keeping Roadway Surfaces in Good Condition, only one participant ranked this item as Important with all of the other participants ranking it as Very Important. In fact, all but two of the items were ranked between Important and Very Important, and those were ranked between Somewhat Important and Important. This means that the participants seem to want it all. There were some relative differences in importance, but the participants considered most of the items on the Importance ranking to be reasonably important. This is documented in Table 6. 12

18 Table 6 - Importance Rankings. Rank Item Average Standard Deviation 1 Keeping roadway surfaces in good condition Removing debris Providing highway striping that is visible at night Providing highway striping that is visible during wet weather Keeping bridges in good condition Ensuring signs are easy to see at night Water drainage Traffic flow from traffic signals Keeping guardrails/barriers in good condition Congestion free commute Public transportation services Keeping highway shoulders in good condition Removing snow and ice Road work/driving condition information Picking up trash Ensuring signs are easy to see during the day Urban area lighting Mowing to increase visibility Providing highway striping that is visible during the day Rural area lighting Keeping rest areas clean Reliable travel time Options for alternative modes

19 The Priority rankings use a similar measurement concept, but with a difference designed to address the they want it all phenomenon. The response scale went from 1 to 7 with: 1 Not at all Important. 3 Somewhat Important. 5 Important. 7 Extremely Important. In addition, participants were instructed to only mark two items with a 6, and two items with a 7. This forced more variation in the responses than in Table 6. The items may have produced some variation in attitude response regardless of the restriction for items on the high end, but the choice limitation aspect is a closer model of the real world situation. Table 7 - Customer Priority Rankings. Rank Item Average Standard Deviation 1 Maintaining existing highways Adding truck only lanes Adding lanes to major highways Adding high speed rail Adding bus and carpool lanes Providing separation between oncoming traffic Adding lanes to state routes Adding transit service Adding shoulders to state routes Maintaining existing traveler rest areas Adding turn lanes Installing traveler message boards Adding traveler rest areas Adding bike lanes

20 Bike lanes, turn lanes, rest areas, and message boards were rated relatively lower on this measure. In particular, adding bike lanes was particularly low in priority for the participants with an average score of 2.7 (slightly lower than Somewhat Important). Most items were either close to Somewhat Important or to Important. The item in which the average exceeded 5 Important was maintaining existing highways, which had an average of 5.3. This means that, on average, this item received enough of the reserved 6 or 7 scores to push its value above the 5.0 level. Two other measures were included in the survey to address customers beliefs and values regarding what TxDOT s priorities should be. The first was a series of four questions about how TxDOT should allocate its effort in percentage terms to Aesthetics, Operations, Safety, or Pavement. On this measure, Operations and Safety were more important to the participants with Pavement slightly more valued than Aesthetics. These results appear in Figure 5. Figure 5 - Participants Views on How TxDOT s Effort Should Be Allocated. 15

21 Immediately following the allocation of effort to the four categories, participants were asked to choose their priority between a series of two item pairs on a slider scale. Each item reflected one of the four categories (Aesthetics, Operations, Safety, or Pavement). Each category was paired with each other category twice. These individual priority evaluations were then aggregated into an index that reflected the votes for each of the categories. The results of the slider scale voting are presented in Figure 6. In most cases, the slider scale voting reflected the same priorities as the effort allocation presented in Figure 5. However, Pavement is the one category that is substantially different; participants rated Pavement as the third priority when asked directly about it. However, when rating the Pavement outcomes ( Smooth Roads or Road with No Potholes ), the participants valued these more than the other items. This is an interesting result, which probably reflects a misunderstanding of the meaning of the term Pavement. In other words, participants probably did not recognize the Pavement effort allocation category would include things like smoothness of the road or the absence of potholes. Therefore, it is likely that the slider scale voting is a better measure of customer values than the effort allocation measure, and that different wording is needed in the effort allocation question. Regardless of the effectiveness of the specific measures, Operations and Safety are valued in both exercises, and Aesthetics is less valued in both exercises. Figure 6 - Participants Choices when Faced with Dilemmas between Two Opposing Types of Effort. 16

22 Meeting Expectations while Riding the Road with TxDOT Following the completion of the first section of the survey, participants were driven in mini vans through a series of eight road sections (illustrated in Figure 7) to evaluate whether particular elements of each roadway section met their expectations. Participants were asked to rate various elements in each section on a scale that included the following: 1 Fails to Meet Expectations. 2 Almost Meets Expectations. 3 Meets Basic Expectations. 4 Exceeds Expectations. 5 Greatly Exceeds Expectations. Figure 7 - Roadway Sections. 17

23 Each of the specific items being rated was categorized into one of eight types including: Design. Stripes. Pavement Condition. Aesthetics. Other Condition. Signs. Appearance. Overall Condition. Figure 8 presents the results for each of the main categories spanning all of the eight road sections in a box plot diagram. The best description of the participants views while on the road is that all eight elements are rated between the Meets Basic Expectations level and the Almost Meets Expectations level. The dots above and below the whiskers indicate there were one or two potential outlier values for almost all categories. The Aesthetic category has the most response variation, indicating a wide range of aesthetic variation among the eight sections. Figure 8 - How Well Expectations Are Being Met for All Road Categories. Note: See the analysis methodology for more discussion about box plots. 18

24 Figure 9 demonstrates the degree to which expectations were met for each road section regarding pavement conditions. Most road sections were rated between 2 Almost Meets Expectations and 3 Meets Basic Expectations. However, Airport North had all ratings at 3 Meets Basic Expectations or above. FM 3177 and FM 969 were interesting because there was a substantial variation in the ratings of the participants. Some participants rated the pavement conditions highly for these roads and other participants did not and the range of responses was so wide that no statistical outliers were detected. Figure 9 - Average Expectations Regarding Pavement Conditions. 19

25 Figures 10 and 11 illustrate the variation in participants evaluations of Signs and Stripes for each road section. The Parmer Road section had most people evaluating the signs as meeting basic expectations (Figure 10). However, the variation in responses (indicated by the number of dots) suggests that individuals may not know what they expect regarding signs, or they may have very different expectations. In addition, this result could be due to random chance. Figure 10 - Average Expectations Regarding Signs. Figure 11 - Average Expectations Regarding Stripes. 20

26 Figure 12 provides the expectation evaluations for the Design elements of each road section. These results show that Airport North and Airport South were evaluated higher on Design than Manor Loop, which had a very low overall design rating. Other road sections design ratings were mostly centered on the 3 Meets Basic Expectations level. Figure 12 - Average Expectations Regarding Design. 21

27 Figures 13 and 14 demonstrate the Appearance and Overall Condition ratings of each of the road sections. The two ratings agree with each other for all sections except Airport South where Condition is rated half a point higher than Appearance. It will be interesting to track this relationship and identify the type of sections where the ratings diverge. Figure 13 - Average Expectations Regarding Appearance. Figure 14 - Average Expectations Regarding Overall Condition. 22

28 Comparison of Ride the Road with TxDOT results and the Texas Maintenance Assessment Program Because one of the goals of the Customer Satisfaction Program is to relate the public s perception of roadway attributes with the technical measures used by TxDOT to score them, the research team compared answers from the pilot Ride the Road with TxDOT results with scores from the Texas Maintenance Assessment Program (TxMAP). TxMAP is TxDOT s statewide system whereby it evaluates a statistical sample of all its roads for the pavement condition, traffic operations, and roadside features. Ratings are based on a 1 5 scale with 5 being the best and 1 being the worst. The scoring component elements are broken down as: Pavement 50% of score, Traffic operations 25% of score, and Roadside 25% of score. It is difficult to compare the focus group scoring with the values from the TxMAP dataset for at least the following reasons: Sections are different The TxMAP ratings cover a much longer section of road than the focus groups was able to grade. The section of US 183 that the group rated, in particular, has some of the worst pavement quality on US 183. Categories are not an exact match The pavement quality and overall categories appear to be a reasonable match, but the traffic operations category used by TxMAP scores was not replicated in the questions about signs and stripes. Table 8 illustrates that in every case TxMAP scored the sections with a higher score (out of a possible 5 points) than the focus groups. There was an average of approximately one rating point difference between TxMAP and the focus group in each of the four comparison categories. 23

29 Table 8. Rating Scores from TxMAP and Focus Group Participants Road Section Scores Rating Source Rating Category Airport Blvd (LP 111) US 183 FM 3177 FM 973 FM 969 TxMAP Pavement Focus Group Pavement Point Difference TxMAP Traffic Operations Focus Group Signs Average Signs Point Difference Focus Group Stripes Average Stripes Point Difference TxMAP Overall Score Focus Group Overall Score Point Difference There was reasonably good agreement between TxMAP scores and focus group ratings if one analyzes the which is worst and which is best ratings. Table 9 illustrates the place of the ranking scores for each category. In 11 of 20 comparisons the rankings agreed or were one place different. While not entirely surprising given that only five sections were directly comparable, this does show a method of comparing ratings that may be useful with larger datasets. Table 9. Rating Places from TxMAP and Focus Group Participants Road Section Placing (1 to 5) Rating Source Rating Category Airport Blvd (LP 111) US 183 FM 3177 FM 973 FM 969 TxMAP Pavement Focus Group Pavement TxMAP Traffic Operations Focus Group Signs Average Focus Group Stripes Average TxMAP Overall Score Focus Group Overall Score

30 Recommendations Regarding Ride the Road with TxDOT As noted earlier, this was the first in a series of planned pilot applications. As such, it is important to point out improvements that can be made in subsequent experiments. When working with the general public it is difficult to anticipate events that may cause participants difficulty. This could be anything from an unexpected illness that prevents a participant from attending to the actual ability to sit in a vehicle for an extended length of time. In this effort participants were compensated $ for their time. This is a reasonable amount considering the length of the entire undertaking (approximately three hours). We recommend this practice continue. However, it may be useful to pay some participants a lesser amount to act as alternates in the event that a previously selected participant cannot attend. Logistically, Ride the Road with TxDOT requires considerable planning. A meeting location that is convenient to the designated route must be secured. The location must be able to accommodate the participants comfortably while they complete Parts One and Three of the survey. With input from the Task Force, the research team decided that individuals would be more comfortable participating in the road portion in a vehicle that would be more similar to a typical passenger vehicle. Mini vans were chosen over 15 passenger vans for this reason. The vans accommodated four participants, a moderator and a driver. However, this required more vehicles and staffing than originally anticipated. One way to reduce the number of vehicles and staff required would be to schedule participants for various time slots, perhaps three or four sessions in one day. This study was conducted on a Saturday morning because researchers felt this day and time would yield a greater pool of possible participants but it would be worthwhile to test an appointed time slot approach. Other suggestions from the research team and participants include: Shorter, more homogenous sections some sections were almost 5 miles in length. This created too much variation in the elements for participants to accurately score. Choose the route with TxMAP scoring sections in mind. It would be easier to begin with TxMAP date and find route segments that coincide with TxMAP control sections. Include a break most participants felt the amount of time in the vehicle was reasonable but some suggested they could score more sections if there was a break. More clearly define survey instrument elements some sections included a Find the Street feature where participants were asked to locate a cross street in the section. In some cases participants forgot to do this. It would be helpful to shade this in future versions. Similarly some sections contained specific features, such as smoothness of a railroad crossing that participants were asked to score; it would be helpful to call out these elements more prominently by moving them to the top of the list of elements. 25

31 Add a practice section many participants noted that they better understood the questions after driving through a few sections. Include nighttime experiment some participants felt this would be especially useful for questions regarding signage and striping. Consolidate some questions such as striping to shorten the survey. Survey in rainy weather some participants thought their ratings would change in wet weather conditions. Next Steps The customer satisfaction program, in general, and the Ride the Road with TxDOT program in particular can be a very useful connection between TxDOT s designers and planners who carry out functions that respond to the public s interests and the public, who often are not as active in the project development process as TxDOT needs them to be. This connection can take the form of calibrating the technical measures that are used to rate road, sign, striping, and other features. In the interest of getting as much service out of the available funds, there may be ways to economize on maintenance activities that the public finds less important or when they cannot distinguish between good and bad quality levels. TxDOT can then use the information collected in a comprehensive manner that relates to relevant performance measures and ultimately to TxDOT s strategic plan. It will be important to conduct the Ride the Road with TxDOT surveys in a programmatic so that results remain current and relative. It is unlikely that data from five years in the past, for example, will be persuasive to either TxDOT personnel or to the variety of public, legislator, or decision maker audiences. As the specific elements of the customer satisfaction program are conducted, it is very likely that other audiences will be identified and additional uses of the results will be discovered. While this puts increased emphasis and importance on the public input mechanisms, it also extends the reach of the information collection and provides additional opportunities for interested parties to have discussions about TxDOT priorities, practices, and funding. In the broader context, the customer satisfaction information may prove useful in developing strategic investment plans and service quality expectations. An extensive set of market research information (as might be available in two or three years) could be used to adjust pavement quality targets; develop cost effective replacement programs for stripes, signs, and other infrastructure elements that meet customer expectations; identify the effects of budget limitations over the recent time period and assist public information officials in understanding the public s knowledge about key aspects of TxDOT. Moreover, this information will identify gaps in knowledge and misunderstanding of various population groups. The survey results will allow the department to develop and deliver targeted messages that increase understanding and demonstrate transparency and accountability. 26

32 All of these uses and others can be important to TxDOT s Division and District offices, but only if the program is conducted. Certainly there will be changes in the methods and extent of the surveys over the years; the program is at the start and the end will probably look different. The results must continue to be meaningful. 27

33 Appendix 28

34 29

35 30

36 31

37 32

38 33

39 34

40 35

41 36

42 37

43 38

44 39

45 40

46 41

47 42

48 43

49 44

50 45