Goodbye Sampling, Hello Universe?

Transcription

1 Goodbye Sampling, Hello Universe? FAST Presentation at 2015 ALGA Annual Conference Corrie Stokes, Acting City Auditor Austin, Texas Terminology From dictionary.com From Corrie (.com) universe all records/transactions that relate to a particular question or problem sample a statistically valid selection of individual records/transactions designed to be representative of the population 2

2 Why do auditors use statistical sampling? 1) We like random numbers 2) We think complex formulas are cool 3 Why do auditors use statistical sampling? 3) Math theory supports it! 4) It s efficient! You can review a small number of items and extrapolate your results to the population. 4

3 My hypothesis: Auditors over-rely on sampling 5 We have a lot of electronic data! Recommendation: Improve reliability of data collection efforts by installing mechanical or electronic counters at library sites. Recommendation: The court computer system should be modified to provide additional status codes Recommendation: Program the management information system to include information on vehicle performance and reliability Recommendation: The department should implement an integrated management information system 6

4 Why not analyze it? 7 Reasons to consider analyzing the universe 1) It can save time and trouble in the long run Do not have to spend time designing your sample Do not have to spend time extrapolating your sample results Do not have to defend your sampling methodologies and assumptions 8

5 Reasons to consider analyzing the universe 1) It can save time and trouble in the long run Do not have to spend time designing your sample Do not have to spend time extrapolating your sample results 9 Reasons to consider analyzing the universe 1) It can save time and trouble in the long run Do not have to defend your sampling methodologies and assumptions Management does not have to re-do the analysis on the universe 10

6 Reasons to consider analyzing the universe 1) It can save time and trouble in the long run Example: utility late payment fees Sampled 270 bills 87 bills were paid late 38 late payments (44% of those paid late) not assessed a late fee Projecting the exceptions found in our sample to the total bills issued in our scope period, we estimate the City may have forgone as much as $13.6M in late payment penalties. 11 Reasons to consider analyzing the universe 1) It can save time and trouble in the long run Example: utility late payment fees Sampled 270 bills 87 bills were paid late 38 late payments (44% of those paid late) not assessed a late fee Projecting the exceptions found in our sample to the total bills issued in our scope period, we estimate the City may have forgone as much as $13.6M in late payment penalties. 12

7 Reasons to consider analyzing the universe 2) The universe is more convincing! Sample Results Favorable Unfavorable Typical Response Sample is effective! Sample is flawed! 13 Reasons to consider analyzing the universe 3) Even if you can t analyze the whole universe, you are going to analyze a large percent (sample!) of the universe Example: animal control response times Analyzed all citizen calls to animal control for a year and found that: 20% did not have data to determine time to respond Remaining 80% indicated long response times for active emergencies 14

8 Reasons to consider analyzing the universe 3) Even if you can t analyze the whole universe, you are going to analyze a large percent (sample!) of the universe Which sounds better? a) Based on a sample of 269 calls (.02%), Animal Control took an average of 16 hours to respond to active emergencies. b) Based on data for 24,000 calls (80%), Animal Control took an average of 16 hours to respond to active emergencies. 15 Reasons to consider analyzing the universe 4) You can tell management exactly what to fix Example 1: Customers who were not assessed late fees (all of them, not just the 38 in our sample) Example 2: Animal control records that do not have a response time recorded 16

9 Some cautions about analyzing the universe instead of a sample: 1) Analyzing the universe will generally not work if: You need to review hard copy documents to determine the criteria or condition You need to use auditor judgment to determine whether your criteria is met 17 Some cautions about analyzing the universe instead of a sample: You will need to review the data you cannot analyze and explore potential data limitations that might impact your conclusions Analyzed all citizen calls to animal control for a year and found that: 20% did not have data to determine time to respond Remaining 80% indicated that Animal Services took an average of 16 hours to respond to calls coded as active emergencies What do the 20% have in common? 18

10 Things to consider when deciding whether to analyze a sample or the universe: Do you have the data electronically? Can you answer your audit questions with the data you have? Will it save a significant amount of time to analyze a sample instead of the universe? 19 Hello, universe! 20