Assessing Customer Satisfaction with Vocational Rehabilitation Programs

Transcription

1 Assessing Customer Satisfaction with Vocational Rehabilitation Programs Dr. Mary Stutzman, Director FSU Survey Research Laboratory Presentation at the Quarterly Meeting West Palm Beach, September Florida State University Survey Research Laboratory College of Social Sciences

2 Why Should the FRC Survey Customers? Provides an outside look of customer service. Customers are sometimes uncomfortable to give feedback directly to the agency giving them services. Gives customers an opportunity to speak to someone outside the Department. Open-ended comments 800 number calls and referrals to FRC Provides longitudinal data base to assess trends. Data can be packaged for to address specific concerns. 2

3 Evolution of the Survey Effort Survey of Customers with Cases Closed Launched 2001 Questions developed by FRC Evaluation Committee and DVR staff Designed survey instrument one-page legal front and back Sample Design all cases closed previous month Customers in Program after 6 months Launched 2002 Questions mirrored Closed Cases Survey Sample Design random sample of 700 per month Customers in program after 6 months 3

4 Customer Surveys Overview Gather information from customers: Program Outcomes Services Staff CLOSED CASES--Surveyed Cases Closed Successful (Status 26) Unsuccessful (Status 80 97, 89) ACTIVE CASES Surveyed Customers in Program after 6 months Random sample of 700 per month; 800 beginning July

5 Survey Approach Closed Cases Survey all cases closed at the end of month Letter Instrument 12 Closed items 2 Open-ended Active Cases Survey sample of 800 cases still active (opened 6 months earlier) Letter Instrument 12 Closed items 2 Open-ended 5

6 Staffing the Survey Effort FSU Survey Research Laboratory All staff paid Staffed with trained interviewers 7 days a week can handle PR issues on 800 line and conduct callin interviews Mail within 1-week of sample receipt Data and open-ended responses entered Analysis of data Provide reports and presentations on a quarterly and ad hoc basis 6

7 Design Benefits Sampling Strategy Power of numbers to analyze by various segments (Areas, Disabilities, etc). Monthly helps eliminate seasonal bias Response Categories Allow ordinary language responses for respondents Helps make reports understandable Operations Mode choice mail, Internet, phone 800 number and open comments public relations for Council Reporting Sufficient numbers for Quarterly Trends Sufficient numbers Annual Trends and sub-analyses 7

8 Respondents are King/Queen Make convenient 2-pages legal Keep short 12 Questions Accommodate Respondent Preferences Web access 800 number for phone response TTY line 8

9 Privacy Issues: Called Project 9

10 Closed Cases: Survey Instrument 10

11 Closed Cass: Survey Instrument 11

12 Survey Areas Program Outcomes Program Choice Services Staff 12

13 Program Outcomes Program Outcomes Closed Cases Active Cases Overall Satisfaction with Services X X Services Made Life Better X X Would Recommend to A Friend X X Employment Outcomes and Satisfaction X 13

14 Program Choice Program Choice Informed of Alternative Service Providers and Vocational Goals Able to Choose Service Providers and Vocational Goals Informed of Alternative Service Providers Able to Choose Service Providers Informed of Alternative Vocational Goals Able to Choose Alternative Vocational Goals Closed Cases X X Active Cases X X X X 14

15 Program Services Program Services Closed Cases Active Cases Accessibility to VR Offices X X Appropriateness of Services Considering Rehabilitation Goals X Promptness of Services Provided X X Need Other Services but Did Not Receive X Open-Ended Closed Cases Active Cases Especially helpful VR associates or services X X What VR can do to improve services X X Why did not get a job X 15

16 Program Staff Program Staff Closed Cases Active Cases Understand Needs and Feelings X X Treated with Courtesy and Respect X X Explained Responsibilities as Client Do What Say X X 16

17 CLOSED CASES: APPROACH AND RESPONSE RATE CUSTOMERS WITH CASES CLOSED: Surveys mailed monthly Surveys mailed to all customers whose cases were closed in the prior month October 2007 to September 2008 mailings 18,100 surveys mailed Two Mailings Analysis based on 5,176 returns Response Rate = 29% 17

18 CURRENT CASES: APPROACH AND RESPONSE RATE CUSTOMERS WITH CASES ACTIVE CASES: Surveys mailed monthly Surveys mailed to random sample of customers whose case is active 6 months after plan 8,700 surveys mailed Two Mailings Analysis based on 2,441 returns Response Rate = 28% 18

19 Adaptation and Change Refined and revisited over time Instrument already have a job Letters explanation of council and repeated surveys Reports content and format 19

20 FFY PROGRAM OUTCOMES Overall Satisfaction --Very or Mostly Satisfied Closed Cases (Q9) -- 85% Active Cases (Q11) 79% Services Made Life Better Yes, Definitely Helped or Somewhat Closed Cases (Q8) -- 86% Active Cases (Q10) -- 81% Recommend to a Friend Yes, Absolutely or Probably Closed Cases (Q10) -- 90% Active Cases (Q12) -- 86% 20

21 FFY PROGRAM OUTCOMES [Closed Cases - Status 26] Job Status of Customers (Q12) Nearly one-third of the customers (31%) obtained a job since becoming a client Nearly one-half (60%) already had jobs Customers (Status 26) who Obtained New Jobs were Satisfied (Q12A) 83% were Very or Mostly Satisfied with their jobs 21

22 FFY PROGRAM SERVICES Accessibility Very or Mostly Satisfied Closed Cases (Q1) 91% Active Cases (Q1) 88% Appropriate Services Highly Appropriate or Appropriate Closed Cases (Q4) -- 88% Active Cases (Q8) -- 86% Need Other Services Yes--Definitely or Think So Closed Cases (Q5)-- 29% 22

23 FFY PROGRAM STAFF Understand Needs & Feelings Very Clearly or Clearly Closed Cases (Q3) 87% Active Cases (Q3) 81% Promptness Very Promptly or Promptly Closed Cases (Q7) 76% Active Cases (Q9) 67% Courtesy & Respect Yes (Definitely or Think So) Closed Cases (Q2) 95% Active Cases (Q2) 94% 23

24 FFY PROGRAM STAFF Do what They Say Will Do Yes, definitely-yes, I think so Active Cases (Q4) 84% Explained responsibilities as a client Yes, definitely-yes, I think so Active Cases (Q5) 93% 24

25 Reporting the Results 25

26 Packaging Results Design allows for detailed analyses Evolves and changes Different products to address different needs Quarterly Statewide Comparisons District analyses Open-ended summaries Special Reports 26

27 Quarterly Comparison 27

28 District Comparison 28

29 Quarterly Comparison over Time 29

30 Open-Ended Summary 30

31 Special Report: Open-Ended Detail by Category and Satisfaction 31

32 Special Report: Results by Primary Disability by Fiscal Year (2003 through 2005) 32

33 Major Challenges for Customer Survey Efforts Participation in Surveys Balancing informational needs of FRC and Agency 33

34 Response Rates are Declining in the New Millennium 70%

35 Respondent Behavior Then 35

36 Respondent Behavior Now 36

37 COPING WITH REFUSALS What part of NO don t you understand? 37

38 Balancing Informational Needs Monitoring data Operational information Data Packaging detail vs. overview 38