Business Survey Planning

Transcription

1 Designing and Conducting Business Surveys for Official Statistics, Tbilisi, Georgia, November 2016 Business Survey Planning Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

2 Remember this: Process map: Business Survey Life Cycle 1. Specifying Target population Information needs Required output 2. Designing Building, Testing survey components Sample frame (obs) Sample Constraints: org.&bus. Data collection mode Survey communication Indicators/q stions Questionnaire Testing 3. Collection Reporting unit(s) Fieldwork Response 4. Processing Data cleaning 5. Analysis, dissemination, archiving Analysis & estimation Dissemination/Archive 6. Evaluation Evaluation (PDCA) Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

3 Remember Question: this: Process map: Business Survey Life Cycle 1. Specifying Target population Information needs Required output 2. Designing Sample frame (obs) Constraints: org.&bus. Indicators/q stions Building, Suppose, within your office, you are Testing survey Sample Data collection mode Questionnaire components commissioned to do a business survey. Data are Survey communication Testing needed on a new topic, and those data are not 3. available Collection in Reporting secondary unit(s) sources Fieldwork (like registers), Response so 4. you Processing need to design and conduct Data cleaning a business survey: Analysis & estimation What is the actual first thing you will do? 5. Analysis, dissemination, archiving Dissemination/Archive 6. Evaluation Evaluation (PDCA) Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

4 Key Statement: survey planning A Survey Project Plan is one of the survey components! The survey data are not achieved by itself All planning activities are aimed at producing a pre-agreed level of quality, at minimum costs and response burden Plan for quality within constraints: Data quality is not achieved by itself Quality management Identify risks: what can go wrong? Risk management Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

5 Overview & Take aways Planning a survey basics: - What is a project? - Project management basics - Source of project failure Planning a business surveys within constraints: - Constraints to consider, specific for business surveys - the planning process An example Planning and progress tool Take aways: Issues to take into account when planning a business survey Detailed planning issues will be Haraldsen discussed & Snijkers, EFTA with course, the Tbilisi, various Nov

6 What is a project? A project is a goal-oriented set of activities aimed at delivering a pre-agreed, concrete and clearly specified result Which has to be realized within constraints: money, time, Clear beginning and ending resources Which is produced by a temporary organization unit, lead by a project manager While facing high levels of uncertainty A project has a clear stakeholder/client In contrast to an ongoing process, but also applies to recurring surveys (tile structure of linked projects) Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

7 Examples of business survey projects (Re)designing and conducting a business survey (Re)designing a questionnaire Introducing a new mode of data collection (Re)designing a business survey communication strategy (Re)designing a new data collection system Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

8 Project management basics ( 4.2) Relevant for planning business surveys 1. Start with the planning the survey as a project, then comes the doing Plan people, resources, activities, and money in time, to make sure that all survey components are ready in time, for the desired levels of data quality (e.g. response rates) Conduct risk assessment: an important part of the production process is outside the control of a project manager, and put in place mitigating plans and actions Deal with the project environment: balance between specific needs, established standards and production constraints Make expectations clear from the beginning Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

9 Project management basics ( 4.2) 2. Manage the project according to the plan: after the planning comes the doing 3. Control and monitor the survey project 4. Communicate project progress 5. Improve the quality of processes: apply the PDCA-cycle: Plan Do Evaluate Adjust Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

10 The Project Manager role ( 4.2) Make things happen: plan, start, and manage the project Control and monitor the survey project Focus on management tasks This should be a separate role! Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

11 Three sources of project failure: The project management principles are aimed at minimizing project failure, caused by: 1. Organizational factors - Determine internal constraints - Conduct risk assessment and put in place mitigating plans and actions 2. Poorly identified customer needs and inadequately specified project requirements - Steps in the planning process 3. Poor planning and control - SMART planning - Using paradata (process data) to monitor progress and achieved levels of quality (using indicators) Quality is established through processes, systems and procedures (Biemer & Lyberg, 2003) Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

12 So, what does planning a survey involve? 1. Specifying Project plan Target population Information needs Required output 2. Designing Building, Testing survey components Sample frame (obs) Sample Constraints: org.&bus. Data collection mode Survey communication Indicators/q stions Questionnaire Testing 3. Collection Reporting unit(s) Fieldwork Response 4. Processing Data cleaning 5. Analysis, dissemination, archiving Analysis & estimation Dissemination/Archive 6. Evaluation Evaluation (PDCA) Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

13 Planning the survey, includes: 1. Specifying the survey 2. Planning the survey process in detail 3. Specifying required resources: money, people, materials 4. Specifying the timetable Structure of 5. Project management issues project plan 6. Risk management issues Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

14 Step 1: Specifying the survey ( 4.3.1) Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

15 Step 1: Specifying the survey 1. Specifying Target population Information needs Required output 2. Designing Building, Testing survey components Sample frame (obs) Sample Constraints: org.&bus. Data collection mode Survey communication Indicators/q stions Questionnaire Testing 3. Collection Reporting unit(s) Fieldwork Response 4. Processing Data cleaning 5. Analysis, dissemination, archiving Analysis & estimation Dissemination/Archive 6. Evaluation Evaluation (PDCA) Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

16 Step 1: Specifying the survey ( 4.3.1) Consult with stakeholder, determine: - Survey objectives, target population and information needs - Deliverables and desired levels of quality (e.g. response rates) - Budget, timeframe, and response burden - Survey specifications Signed tender Consult with survey organization: - Determine constraints Establish business case: pre-plan go/no-go? Appoint project manager Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

17 Constraints to take into account Survey deliverables: the quality of the results, survey specifications, delivery date Check if this can be achieved People, material resources, money, time Apply project management principles Quality considerations: effects of methodologies on data quality Have the quality framework in mind The business context: response process and response burden Study the process that is outside of control and reduce risks upfront Survey production environment Check and consider Especially relevant when planning business surveys upfront Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

18 Study the business context What do want to know upfront about the businesses you are going to study? Input for: What data are available Questionnaire What terminology and definitions are used Where are the data (one/many departments) Format of the data (spread sheet, other format) & Mode Whom to contact Communication strategy Arguments for getting survey participation Other surveys they get (multi-surveyed businesses) When to inform businesses about the survey, so they have enough time to get prepared When are the data available Fielding survey & Frequency Remember: business survey response Haraldsen & Snijkers, process EFTA course, Tbilisi, Nov

19 Study the business context This has to be planned in and carried out at an early stage, in order to reduce risks. Otherwise risks will be pushed downstream. Example: Redesign of Survey on Financial Position of businesses : new data model was developed, based on data requirements End of Q1 in 2017: Fielding of new survey At the end of 2014: Planning of survey was discussed - What is needed to reach this deadline? - Including a feasibility study (business context) Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

20 The survey production environment ( ) If working in a survey or statistical organization, you will need to consider the survey production environment that you are working in. Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

21 21 Haraldsen & Snijkers, EFTA course, Tbilisi, Nov. 2016

22 Constraints Elements of a business survey production environment: These constraints have to be checked upfront They are input for (planning) the survey process See pp Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

23 23 Haraldsen & Snijkers, EFTA course, Tbilisi, Nov. 2016

24 24 Haraldsen & Snijkers, EFTA course, Tbilisi, Nov. 2016

25 This is the picture for one survey! However: for business surveys, the designs are not independent of each other, but related! 25 Haraldsen & Snijkers, EFTA course, Tbilisi, Nov. 2016

26 The survey production environment Take into account that businesses may be multi-surveyed! Investigate what other surveys your businesses get and how these designs are related: Data asked for (overlap in data requests) Timing of surveys 26 Haraldsen & Snijkers, EFTA course, Tbilisi, Nov. 2016

27 27 Haraldsen & Snijkers, EFTA course, Tbilisi, Nov. 2016

28 28 Haraldsen & Snijkers, EFTA course, Tbilisi, Nov. 2016

29 Plan for quality within constraints Identify: Optimal survey design Trade-offs between quality and constraints (costs and response burden): - How do constraints affect the survey design? - The consequences of these trade-offs regarding quality Decisions that need to be taken, and identify the risks for the various options, document these decisions Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

30 Step 2: Plan the survey in detail ( 4.3.2) Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

31 Step 2: Plan the survey in detail Plan all stages of the survey: Plan the entire survey process up to and including survey deliverables and survey evaluation, including how it will be monitored and managed. Write the project plan Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

32 Step 2: Project plan (Fig. 4.3) Results of step 2: project plan, with the following sections: 1. Statement of the survey - Objectives, deliverables, specifications, constraints 2. The survey in detail - Work breakdown structure of activities 3. Required resources - needed human and material resources, money based on work breakdown structure 4. The timetable - who is doing what and when core of the plan 5. Project management - how the survey will be planned, managed, controlled and monitored 6. Risk management Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

33 Step 2: Plan the survey in detail (Fig. 4.2) What the business survey project plan needs to cover Planned in step 2 (discussed in next sessions) Desired quality Business context Response burden Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

34 Step 2: planning the survey 1. Specifying Project plan Target population Information needs Required output 2. Designing Building, Testing survey components Sample frame (obs) Sample Constraints: org.&bus. Data collection mode Survey communication Indicators/q stions Questionnaire Testing 3. Collection Reporting unit(s) Fieldwork Response 4. Processing Data cleaning 5. Analysis, dissemination, archiving Analysis & estimation Dissemination/Archive 6. Evaluation Evaluation (PDCA) Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

35 Step 2: Planning the survey in detail Combine the planning for all survey components and stages: 1. List and number all activities in a work breakdown structure table with activities start with last activity: delivery of survey results, work backward 2. Determine critical path: no delays afforded questionnaire design: start ASAP! 3. Prepare a timetable with milestones: final delivery date of survey results start and end of fieldwork start and end of development of survey components 4. Plan the survey on a real-life calendar 5. Communicate the plan to those involved and revise, get sign-off Remember: the plan is likely to change! Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

36 Three rules of thumb Do things right from the start, otherwise you are pushing problems and costs downstream! Do not underestimate the time needed for DBT activities: multiply estimated time by 1.5 or 2 Plan for quality within constraints: quality is not achieved by itself use the total survey error framework Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

37 Survey planning and Total Survey Framework Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

38 To conclude: planning a survey is a complex process: planning and developments efforts rise with survey complexity not a linear process: it is an iterative process, going back and forth, making amendments to the plan. The project plan is very likely going to change It requires communication skills, ingenuity, and creativity Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

39 After the planning comes the doing Finalizing the project plan: Discuss with everybody involved Get commitment and approval Getting started: Kick-off meeting The project is running Everybody is expected to do their jobs according to the plan Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

40 Planning and progress tool (Fig.4.7) To summarize: A tool to assist the planning process and monitor progress Progress Status Overview: Work breakdown structure for a business survey: - are all activities planned? Overview of milestones: - to monitor the survey progress: to be ticked off during the various survey stages Haraldsen & Snijkers, EFTA course, Tbilisi, Nov

41 Planning and progress tool (Fig.4.7) Haraldsen & Snijkers, EFTA course, Tbilisi, Nov