Staff Guidelines for Conducting Impact Assessment

Transcription

1 Staff Guidelines for Conducting Impact Assessment Executive Summary: Much of the impact of the T-G PEC is not captured and documented in a systematic way. This Guideline is intended for Project, Sector and Intervention Managers to help them design better and more impactful interventions as well as helping them to document this impact in a consistent and credible way. The T-G PEC has been commissioned by the Thai and German Governments to design and implement a number of interventions that improve the competitiveness of SMEs in selected agroindustry sub-sectors. (see Angebot or Offer to BMZ) By following the 5 steps listed below, Project, Sector and Intervention Managers should be able to perform an impact assessment on all interventions in the most cost effective way. 1. Decide if what you are doing is a task or an intervention. Tasks do not need to be assessed for target group impact. 2. Identify the target SME group, their relevant problem and determine what measurable impact the intervention should have on them. 3. Identify the solution in terms of increased access of the target group of SMEs to a service or changed framework condition. 4. Based on the description of the four types of interventions, choose the type which is most appropriate to this intervention. 5. Follow the recommendations of this impact assessment guide to measure medium and long term changes on the target group and service market or framework conditions which are attributable to the intervention. While the design of interventions is completely an in-house activity of the Intervention, Sector and Project Managers, it is suggested to outsource the impact assessment for interventions to third parties such as universities, consultants and research institutes. In such cases these guidelines may constitute part of the terms of reference for these contracts. At the minimum the terms of reference for such consultants should include a specific section on their methodology which included information on: their sampling methods, training of interviewers, pre-testing of the questionnaire, levels of confidence and means of cleaning the data. Another section should include the presentation of the data in a way that is consistent with the indicators specified in the Offer to BMZ for Phase Introduction: T-G PEC measures impact in order to: Improve the design of interventions by anticipating what should be measured, when it should be measured and how it should be measured; Show partners that the programme has impact, so that this may lead to some change in the way that they operate; Use resulting evidence of impact to persuade target groups that T-G PEC interventions have impact on their bottom lines; Enable Programme Management to monitor where T-G PEC is having impact and where it makes sense to further invest; Show the Programme client, BMZ, in a transparent way, that T-G PEC has impact; and Page 1

2 Enable Programme Management to develop BMZ progress reports in an easy, simple and consistent way. 2. Impact Logic: The overall impact logic of the PEC is based on a means-end relationship. There are 3 facets to this impact logic; the one that forms the core of the Offer to BMZ, this is explained in the section below (goal level and component level); the one that exists for the sectors in which the PEC is operation, this impact logic is stated in the Sector Strategy Papers that must be updated on an annual basis, in these papers the impact logic is that if we correctly identify the most important constraints or opportunities for these sectors and then address these problems we should be able to claim some attribution to impact; the thirds aspect of impact logic is expressed at the intervention where the logic of the intervention combines the sector orientation with the overall means end relationship expressed in the Offer to BMZ. The diagram below illustrates how this impact logic applies to interventions, sector strategies as well as the whole T-G programme. Managing for Development Results orientation joint responsibility activity output use of outcomes output (objective) impact highliy aggregated impact (MDGs) 3. Impact as determined by the BMZ Offer: Overall, the T-G PEC measures impact at two levels: a) on the target group, or SMEs and b) on the service market or framework conditions. At the level of the target group (goal level), T-G PEC aims to improve enterprise competitiveness. 1 T-G PEC measures enterprise competitiveness using the following indicators: Productivity Business Performance Promising Innovations Sustainable eco-efficient products and processes T-G PEC supports the development of services or the improvement of framework conditions which enhance enterprise competitiveness. In the case of services, T-G PEC develops sustainable markets and systems for their delivery. For services offered through commercial transactions (either fee based or bundled with products) T-G PEC measures: Demand for services from our target group Supply improvements delivered by our intermediaries through commercial transactions 1 See the attached set of Phase 2 indicators for the goal and components of the T-G PEC Page 2

3 In the case of improving framework conditions or the business environment related to enterprise competitiveness, T-G PEC focuses on measuring changes in government institutions, policies, or laws and regulations aimed at improving competitiveness. The measurement of any of these indicators has to be convincing or plausible; T- G PEC has to show that changes have been substantially caused by something that the programme has supported. Therefore, T-G PEC includes a strategy for estimating the amount of observed change that is due to the programme in all goal level information gathering. At the level of service markets and framework conditions, T-G PEC gathers information to establish a clear and convincing link between programme activities and observed changes. 4. Measuring Impact and not Activities: T-G PEC undertakes many activities. These can be divided into two main groups. The first group consists of those activities that help initiate change, build relations with partners and lay the groundwork for other, more substantial activities, but are not intended, in themselves, to produce impact on either the target group, the service market or framework conditions. These activities are called tasks. Impact from these tasks does not need to be measured but substantial tasks should be concluded with a short task report. Examples of these tasks include: workshops, trips not connected to an intervention, PR events, observation trips and the like. Note that tasks are not part of interventions, but rather activities which lay the ground work for interventions. More substantial activities, or groups of activities, that are intended to result in impact on a service market or framework condition and the target group through a partner or intermediary, are called interventions. An intervention: Promotes a service or framework condition change Supports one or more service providers Expects to produce a change in behavior Has a competitiveness impact objective on the target group Has potential for sustainable replication Note that there may be a variety of activities within one intervention. 2 T-G PEC measures the impact resulting from each intervention. Note that T-G PEC measures the impact of the whole intervention which may include a variety of activities, not each activity individually. For each intervention, T-G PEC prepares a report showing impact on the service market and target group that can be plausibly linked to programme activities (attribution). Since T-G PEC does not deliver interventions to the target group directly, but rather through intermediaries, the focus of impact assessment is on changed conditions in both the target group and the intermediary. There are a number of ways to measure impact: Surveys before and after an intervention, sometimes with a control group; An opinion survey after the intervention; or 2 The T-G PEC defines an intervention as: Any significant allocation of programme resources (manpower or money) invested in a well-defined sub-project of a partner which is aimed at enhancing the competitiveness of SMEs through either: stimulating the demand for services from SMEs; improving the ability of the partner to provide better services (public or private) to SMEs; or improving the policy, legal or regulatory framework conditions for SMEs. Page 3

4 Commonly accepted and public sources of data if they are reliable. For goal level information gathering, a method to establish the amount of change that is attributable to the programme is always included. (See section 9 on impact assessment methodology.) The choice of which research and attribution methods to use is determined by the size, significance and nature of the intervention. Smaller interventions generally use simpler methodologies; larger interventions use more substantial methodologies in order to be convincing. 5. Goal Level Indicators: At the goal level, the Programme has a menu of indicators (see indicator list), from which to choose when designing an intervention; ideally each intervention will have measurable impact on as many indicators as possible. It is important that the impact is measurable and it is important that the indicators are chosen and a plan is made for measuring impact prior to each intervention. The intervention designer should determine, before the intervention starts, what can be measured, how it can be measured and when it should be measured. In terms of the target group (SMEs), T-G PEC has two types of impact that may occur: firstly, a positive behavior change in the target group (interim indicators) and secondly, a positive effect of this change on the target group s businesses. In the short term, it may be more realistic to measure the behavior change as it may take up to 18 months for the changed behavior to have an impact on the target group s businesses, depending on the nature of their businesses. Before and after comparisons that include participants opinions on the reasons for behavior changes are likely to be the best to use in these cases. 6. Component or Service Market and Framework Condition Indicators: The T-G PEC has been commissioned by the Thai and German Governments to design and implement a number of interventions to improve the competitiveness of SMEs in selected agro-industry sub-sectors. Some of these interventions should be aimed at improving the framework conditions for the target group while others are aimed at service markets. T-G PEC may support services which are a) fee-based, b) embedded within other commercial transactions c) paid for by business membership organizations (BMOs) membership dues or d) measures to improve framework conditions such as institutional, policy, administrative or regulatory issues. T-G PEC measures the following: Changes in Framework Conditions. Despite the challenges, T-G PEC is committed to measuring progress towards policy changes as well as the impact of policy changes on the target group in four ways: Actual changes in policies or regulations as a result (at least partially) of programme activities; Documented changes that will modify how a policy or regulation, aimed at the target group, is implemented by a public agency (institutional) By assessing the target group s opinions concerning how the change has impacted on their businesses. As an intermediate indicator, T-G PEC may also assess the stage at which the government is in implementing the stated policy, institutional or regulatory changes. Changes in the demand for services that T-G PEC supports. To measure change effectively, T-G PEC must clearly define the service product (benchmarking, business linkages, information systems, profitable environmental management, a new loan product, farm management etc.) and then outline how Page 4

5 demand for this service can be measured. This can be done for example, by measuring the target group s awareness of the service and the benefits it can deliver. Measuring changes in demand for services is particularly important for services like MIS that might only have direct impact on the target group in the long run. If T-G PEC can show that programme activities have influenced demand, even if demand has not yet reached a level that is commercially sustainable, then some impact has been achieved. Changes in the supply of services that T-G PEC supports. T-G PEC may partner with service providers that are private, semi- private (i.e. universities and institutes), BMOs or government. To assess changes in the supply of services, T- G PEC measures an increase in the quantity and the quality of the service to the target group. Quantity can be measured by the amount of output of service (number of clients served, volume of business, increased range of products offered). Quality is subjective and thus T-G PEC measures it through an opinion survey which assesses the target group s opinions of the service. As an intermediate indicator, T-G PEC also assesses service providers capacity to sustainably deliver the service. 7. Projecting Expected Impact Because it takes time for T-G PEC activities to have an impact on service markets, framework conditions and enterprises, T-G PEC makes upfront projections about expected impacts when starting activities. These projections give staff targets to aim for, and provide staff with feedback on the extent to which an intervention is on track. The projections also allow managers to forecast expected impact per sector and overall for a future point in time, and to assess progress towards these projections as data is collected and estimates of impact are made. Projections of impact are updated at least annually to reflect new data collected on indicators of change. Projections are made as an intervention is initiated, generally after baseline data has been collected. Projections are made for key indicators at both the service market / framework conditions level and the target group (goal) level. Projections are based on reasoned opinions of managers about what changes the programme activities will cause using data from the sector study and baseline. Projections are made for each year up to two years after an intervention is expected to be finished. As data is gathered during and after the intervention, the projections are revised to reflect the findings. 8. Defining Types of Services and Framework Condition Interventions T-G PEC is committed to delivering several interventions which, for the purposes of impact assessment, can be divided into four categories Fee-based or stand alone services These are services that can be delivered to the target group for a fee from service providers who are normally outside of the target groups buyer or supplier relationships. Examples of this within the Programme are: Benchmarking, where the supplier is DEDE and the palm oil association (POCA); Management Information Systems, where the supplier is currently DIW; Small Business Finance, where the suppliers are the 3 SFIs, and recently, the commercial banks: Longan Farm Management, where the supplier is Maejo University; PREMA, where the supplier is ECDA; Page 5

6 Technology Advisory Services, where the provider is TISTR; Food Safety Certification (including organic); where the providers are private certifying bodies. T-G PEC may partner with service providers that are either public organizations committed to supplying the service to the target group consistently and to a high standard or with private providers, again, if their commitment is long term. At the goal level, T-G PEC measures one or more of the competitiveness indicators. For example, for benchmarking or PREMA, T-G PEC might measure productivity improvements in the target group that participated. To find out, convincingly, if changes at the level of the target group are due to programme activities, T-G PEC will use a before and after methodology usually with a control group for this type of service. At the service market level, T-G PEC will aim to generate an increase in the demand and/or supply for this service in order that service delivery moves towards sustainability. A demand increase could be measured, for example, by an increase in the target group s awareness of the service, understanding of the service s relevance to their business performance or willingness of the target group to pay for the service. These changes can be measured through an opinion survey conducted at the beginning of the intervention and then after the intervention is completed. An increase in supply could be measured by an increase in the amount of the service T- G PEC s partner providers are delivering to the target group and by an increase in the value the target group places on the service (customer satisfaction) as a proxy for service quality. Because T-G PEC usually promotes new or substantially changed services where there is a clear link between programme activities and changes in the service market, measuring changes in the demand and supply of services is usually sufficient; a method to estimate attribution of changes to T-G PEC is usually not needed as attribution is assumed to be 100% Embedded Services (knowledge and advice embedded in business-tobusiness or B2B transactions) These are services that are delivered to the target group within, and as part of, existing or potential buyer- seller relationships. Examples of this in the Programme are: Palm oil plantation management training or advice where the provider is a buyer crushing mill; Advice to tangerine orchards on the optimum use of agricultural inputs, where the provider is the retailer of inputs; Small orchard productivity improvements, where the provider is a buyer tangerine producer. Fruit and vegetable exporters providing certification compliance standards to their supplier small farmer groups. Providers of these services are almost exclusively private companies. At the goal level, most of these services aim to improve the target group s productivity and/or sales. As with stand alone services, T-G PEC will use a before and after methodology with a control group for embedded services to learn and to show convincingly that the changes are due to Programme activities. Measuring changes in demand and supply of embedded services is more challenging than stand alone services because there is no discrete transaction for embedded services. To measure changes in demand, T-G PEC will use a survey of service recipients focusing on: a) Whether they are satisfied with the service or whether they feel that the service has had a significant impact on their business; Page 6

7 b) Whether they changed their business practices as a result of the service; and c) Whether they will continue to maintain their relationship with the supplier as a result of the service. Assessment of service supply will focus on measuring the service providers opinions of the service, for example whether they feel that this service will be continued or if their sales have improved as a result of this added service and, more objectively, the volume or percentages of the target group to whom they are providing the service Business Membership Organization based services These are services that are typically provided by a business chamber or association. They often fall into four broad categories: business linkages (trade fairs, study tours, delegations); advocacy to elicit some form of change in a policy or regulation; area or sector promotion (market research, sector strategies); and, information, usually in the form of newsletters, presentations, workshops, websites, membership directories etc. Research shows that these are the four main reasons why businesses join BMOs. The measurement of impact from BMO based services is difficult because: Impact is often very long term (information or sector promotion). Even the short term impact of a new policy may be difficult to measure. The services that usually show the most immediate impact are trade fairs as they usually involve some form of contact between buyers and sellers. However, many businesses do not wish to reveal their new contacts and how well they did as a result of this contact. Most of these services are of a public nature and are therefore open to everyone and not just members of the association or chamber. Given these challenges, T-G PEC will focus assessment of these types of services on the BMO itself (as a proxy for the target group) rather than the target group directly. Measurement will examine: Satisfaction of the BMO membership base with the service that the Programme supported (i.e. what did they expect, were expectations met, was it relevant to their businesses, will they do anything different as a result of using the service?) and, How sustainable is the service within the BMO (did members pay for most of it, do members want it be provided on a regular basis, do they have the staff and resources to do it on their own, did it enhance their image, have membership numbers or fees gone up for the BMO?). This will help T-G PEC learn if the service is useful to the membership and if the BMO has the capacity to deliver the service sustainably. These findings will help direct T-G PEC s future work with the BMO Framework Conditions These are the services or deliverables which the target group expects to come from Government. Often the government feels that these services are exclusively within their domain. They are usually related to information or public awareness of a need to adopt a new practice or to the enactment or enforcement of a new law, regulation or policy. Sometimes these services extend to infrastructure such as industrial estates, public market places, road access, water or electricity. All of these can have a profound impact on the competitiveness of our target groups. T-G PEC s challenge is to clearly define each of the public services or framework conditions it supports and to link each service to the competitiveness of the target group. For public services, T-G PEC will assess impact by measuring the target groups opinions about the service including: do they know about it, were they involved in the change process, Page 7

8 do they understand the impact it will have on their businesses or on the public well being (for example, in the case of pollution control), do they expect the change will benefit their businesses. To assess supply and the service provider usually a specific government agency T-G PEC will appraise the extent to which the service can be sustainably delivered to meet the demand (quantity and quality) from the target group. For example, T-G PEC will examine if the agency has sufficient staff to implement the service, if staff are sufficiently well trained, if the agency has the resources to implement the service, and what is their long term commitment to the changed condition (sustainability). In addition to measuring this direct impact, the T-G PEC will also be engaged in broader institutional reform processes that have impact on the thinking of Government. This can take the form of round table discussions on the role of the state in environmental management and enterprise or any other topic that is somehow related, in a Macro sense, to the goals of the programme. In such situations the incremental changes may be hard to evaluate as they depend on opinions and conducting opinion surveys by the programme in sensitive areas like this may be harmful to a successful outcome. In such cases anecdotal evidence will be adequate, as interim indicators, to show that progress is being made. Such anecdotes must however be more than mere rumors and some documentation must exist to support the claim of progress. Note that some interventions may fit into more than one category. From an intervention perspective, this is not a problem and involving more stakeholders may, in fact, improve service delivery. However, for the purposes of impact assessment, it is useful to classify interventions into one of the four categories. If the intervention seems to fit into more than one category, consider the sustainability strategy. Which organization is expected to sustain service delivery to the target group after the T-G PEC intervention ends? The sustainability strategy should help make it more clear which the intervention type it fits most closely, and therefore which impact assessment methodology should be chosen. 9. Impact Assessment Methodology Changes, particularly at the goal level, are a result of many factors. T-G PEC does not only need to know what changes have taken place over time, but also what changes are a result of the programme activities as opposed to the many other factors affecting the markets in which T-G PEC works. (See Figure 1.) Page 8

9 Growth Impact Assessment for T-G PEC June 8 th, 2008 Figure 1: Isolating T-G PEC Impact with intervention Impact without intervention Total change Intervention Start Intervention End Time For substantial interventions related to stand-alone and embedded service, T-G PEC will measure impact using a before and after survey of SMEs with control groups. Measuring SME behavior and competitiveness before and after the intervention will enable T-G PEC to pinpoint how the target enterprises have changed during the period of the intervention. Comparing changes in participating enterprises (i.e. those that access the service) with non-participating enterprises (i.e. those that do not access the service) will enable T-G PEC to isolate the changes that resulted from the intervention, as opposed to other factors such as: changes in the economy, the weather, or other programs, etc. The non-participating enterprises are called the control group. Using control groups will enable T-G PEC both to understand and to show the impact that each intervention is having on target enterprises. In order to keep impact assessment manageable, T-G PEC will generally measure enterprise performance and behavior before and after an intervention in approximately 20 participating or client enterprises and approximately 10 non-participating or control group enterprises. For BMO-based and public services, T-G PEC will generally measure impact using an ex-post survey of target enterprises. For BMO-based services, the target enterprises will be members of the BMO which T-G PEC is assisting. For public services, the target enterprises will be those that T-G PEC would expect to benefit from the change in policy or regulations. In these surveys, T-G PEC will assess what enterprises expected from the service, what they feel they got, how they think they changed their behavior as a result and what benefits they feel they got. This method uses the opinions of enterprises to understand the impact of the intervention. Page 9

10 These more formal methods are always supported by measurement of intermediate indicators at the service market / framework conditions level and at the goal level. These measurements enable T-G PEC to show that changes in service providers or government institutions led to changes in service markets or framework conditions, which led to changes in the behavior of enterprises, which led to changes in the competitiveness of enterprises and the sector. In some cases, other methods may also be used to support the more formal methods above. Managers may assess trends before the intervention and after the intervention to show that the trend has changed. Case studies of particular enterprises may be used to describe changes in qualitative terms, linking services or framework conditions to behavior changes to enterprise performance and competitiveness. For all intervention types (stand alone or embedded services, BMO services or framework changes), T-G PEC will measure the volume of service that the provider is delivering to the target group (SMEs) and provider capacity based on staff s assessment of each provider s abilities to sustainably deliver the service. Capacity assessment will be focused on critical success factors in each provider. Staff will identify these critical success factors for each provider early in an intervention based on the following menu of capacity areas: Attitude and motivation towards commercial service provision Leadership Marketing Product development Personnel Product delivery Finance and technology If appropriate, T-G PEC staff may choose other critical success factors. For example, different factors might be appropriate for government agencies. For BMOs and government, T-G PEC will also measure the opinions of BMO or government agency leadership on the usefulness of the service or policy/law/regulation change to the institution itself. For some interventions, T-G PEC partners with several institutions which are involved in service delivery. For example, a BMO might mobilize demand for a service provided by a private sector provider in a stand alone fashion. Or a government agency might provide technical assistance to a private sector company providing embedded services. In these cases, T-G PEC will focus measurement on the actual service provider who is delivering services to the target group (SMEs) and include in critical success factors, if appropriate, that service provider s ability to maintain the relationship with other involved institutions. T-G PEC may choose to also assess the capacity of other partner institutions where this is critical to the sustainability of service provision. To support T-G PEC s conclusions about impact, T-G PEC maintains files on all research conducted and on intervention implementation. This supporting documentation includes: Research reports Raw data for calculation of indicators and actual calculation methods TORs for consultants who conducted the research Supporting government data when appropriate Primary or secondary data on the target group Each version of the sector strategy / sector progress report Implementation Arrangements and Implementation Agreements Page 10

11 10. Impact Assessment during Different Project/Intervention Phases Major interventions in T-G PEC will go through two main phases: The pilot phase during which T-G PEC is developing and testing a service with only one or a few service providers and with limited outreach. During this phase, T-G PEC aims to learn if the service has the desired impact on target enterprises, if it is accepted by all relevant stakeholders and if it can be delivered sustainably. T-G PEC is also working with partner providers to refine the service and its delivery. Examples of this would be Benchmarking, PREMA and Longan Farm Management. The scale-up or market development phase during which T-G PEC aims to substantially increase the outreach of the service either by encouraging more providers to offer the service or by helping existing partner providers to substantially scale up their provision. During this phase, T-G PEC is focused on expanding the market for the service and ensuring it is sustainable. There are different requirements for impact assessment during each phase. During the pilot phase, T-G PEC needs to gain a fairly in-depth understanding of the delivery of the service and its impact on target enterprises. During the scale-up phase, T-G PEC will focus impact assessment on the scale of service provision, using the data from the pilot phase to estimate impact on target enterprises. If scale is likely to be high, a small survey may be conducted during the scale up phase to check that impact data from the pilot phase is still realistic. Thus, impact assessment is necessarily more intensive in the pilot phase than the scale up phase. T-G PEC has not yet fully designed its impact assessment methodology for the scale-up phase, although ideas have been generated. The description below provides a brief overview of impact assessment during each intervention stage Service Type: Stand Alone or Fee based Service Pilot Testing of Intervention s Business Model Phase Before and after personal surveys with ~ 30 clients and ~ 10 control group enterprises to measure perception of the service, use of the service and enterprise performance and some evidence of likely impact Provider(s) assessment focused on attitude, capacity and sustainability before and after intervention Scaling up Phase Change in transactions in the market; changes in demand and supply; capacity of supply to respond to changes in demand Service Type: Embedded Services Pilot Testing of Intervention s Business Model Phase Before and after personal surveys with ~ 30 clients and ~ 10 control group enterprises to measure perception of the practice(s) that the service addresses, actual practice and enterprise performance Provider(s) assessment focused on attitude, capacity and sustainability before and after intervention Scaling up Phase Changes in reach of the service, changes in capacity of more suppliers to deliver service so that it is beneficial to all concerned enterprises Service Type: BMO Based Services Pilot Testing of Intervention s Business Model Phase Ex-post personal or written surveys with a sample of clients focused on demand for service, expectations and satisfaction with service Page 11

12 Provider(s) assessment focused on attitude, capacity and sustainability before, interim and after intervention with some depth; interview with BMO on their opinions of the service. Scaling up Phase Changes in BMO membership, Amount of services delivered by the BMO, Changes in BMO capacity Service Type: Framework Condition Services Pilot Testing of Intervention s Business Model Phase Ex-post personal or written surveys with a sample of stakeholders focused on their experience of, satisfaction with and change in business performance as a result of the policy or regulatory change Provider assessments focused on attitude, capacity, change process etc. before, interim and after intervention with some depth; interview with relevant govt. officials on their opinions of the change process, policy or regulatory changes, etc Scaling up Phase Assessment of the momentum and sustainability of the policy and regulatory reform process, including stakeholder involvement and government capacity 9. Impact Assessment Guides This section is divided into 4 parts, one for each of the previously mentioned service types. In each part the guidelines suggest, for each service type,: what needs to be known and indicators who to interview when to interview the interview format, and the data presentation format In each one of the parts below sample questions are suggested that will help to reveal the data that will answer the questions Are we having an impact on the target group (SMEs)? and Are we having any impact on the sustainability of these services? In the cases of stand alone and embedded services, some form of before and after data collection is recommended and when the intervention is substantial a control group is also recommended. In the most extensive cases data is collected from three sources: the target group of SMEs that participated in the service delivery; the target group of SMEs that did not participate (for comparison purposes) and thirdly the service providers. In some, more simple, cases data is only needed from the immediate target group participating in the intervention and the intermediary or service provider of the service. Note that in all interventions, the target group is always considered to be SMEs, regardless of which other people, organizations or institutions the intervention is aiming to influence or change the behavior of. This is because, even though interventions work through other organizations and companies to reach SMEs, the ultimate goal of all interventions is to impact the competitiveness of SMEs. 9.1 Stand Alone Services In the initial phase of the intervention for stand alone services before and after surveys are suggested for a) the participating target group members, b) a control group and c) for the service provider. However, if the only goal level indicator is innovation, then a control group is not suggested as they would not provide a good comparison with the target group. Page 12

13 Participating Target Group Data: To assess impact on indicators at the Component Level Need to know Indicators Sample Questions Perception of the service Awareness of the service and its benefits Have you ever heard of service? What do you think the benefits of service can be for your business? Experience of Satisfaction with the Compared to your expectations, how satisfied are you the service Change in practice service Purchase of the service with service? Have you ever purchased service? If so, when was the last time? From whom? To assess impact on indicators at the Goal Level Need to know Indicators Sample Questions Competitiveness Perception of the service s contribution to competitiveness Did this service make a difference to the competitiveness (see definition) of your business? Productivity (inputs/outputs) Sales Quality (i.e. grades) (if applicable) Innovation (if applicable) Eco-efficiency (if applicable) For productivity, a number of questions will be asked related the volume and price of outputs for a given period of time and the costs of the three most important inputs required to produce that particular output. What amount/percent of your products/produce was A grade? Innovation questions will be specific to the expected types of innovation. Eco-efficiency questions will be specific to the expected types of eco-efficiency products or processes. Control Group Data: To assess impact on indicators at the Component Level Need to know Indicators Sample Questions Perception of the service Awareness of the service and its Have you ever heard of service? What do you think the benefits of Experience of the service benefits Purchase of service from any source service can be for your business? Have you ever purchased service? If so, when was the last time? From whom? To assess impact on indicators at the Goal Level Need to know Indicators Sample Questions Competitiveness Perception of the service s contribution to competitiveness (if applicable) Did this service make a difference to the competitiveness (see definition) of your business? Productivity (inputs/outputs) Sales Quality (i.e. grades) (if applicable) Innovation (if applicable) For productivity, a number of questions will be asked related the volume and price of outputs for a given period of time and the costs of the three most important inputs required to produce that particular output. What amount/percent of your products/produce was A grade? Page 13

14 Eco-efficiency (if applicable) Innovation questions will be specific to the expected types of innovation. Eco-efficiency questions will be specific to the expected types of eco-efficiency products or processes. Who to interview? Approximately 30 clients of the service can be more if there are many clients. Choose the sample to represent client group. Not more than 10 and not less than 5 control group enterprises o As similar as possible to the client group o Close enough (geographically) to the client group that they face similar conditions o Far enough away (geographically) from client group that the spill over from clients to control group is minimized When to interview? There will be 2 measurements: one before the clients obtain their services (baseline) and one when clients are expected to show changes in their competitiveness (final). Baseline: Clients when they have signed up to get the service but before they get it Control group within the first month of when clients start getting interviewed Final: When clients can be expected to show changes in terms of business performance according to their business or agricultural cycle Control group at the same time as clients T-G PEC may need to interview enterprises several times, for instance during the agricultural cycle or harvest period to gather accurate information on productivity (inputs and outputs), sales and quality (i.e. grades). Up to 4 times is acceptable. Only one final interview needs to gather information on the service. All others should gather information only on business competitiveness. Interview format: Interviews will be one-on-one and in person. It is recommended that 2 interviewers conduct each interview together. The interviewers will work off the questionnaire but may probe further to get accurate answers. Therefore, interviewers must understand the questionnaire and be skilled. For any new interviewer it is strongly recommended to pilot test the questionnaire, in at least three cases with a more experienced interviewer before attempting this on their own. Framework for data presentation: Baseline: For both client and control groups: Component Level Awareness of the service: % aware; % not aware; qualitative comments Understanding of benefits of the service: % understand; % partially understand; % not understand; qualitative comments Page 14

15 Purchase of the service from any source: % received service before; % not received. Of received: % paid; % received for free; source for service; qualitative comments Goal Level Competitiveness: Quantity and cost of 3 most important inputs Sales, including export sales, in a given period (i.e. palm oil sales for 2 weeks 4x a year; for longan, sales once a month for 4 months of pre and harvest period) Quality: as applicable (i.e. % of total produce in AA and A category for given sales period and/or % of total produce that is off-season) Innovation: as applicable Eco-efficient products/processes as applicable Qualitative comments Final: Client group: Component Level: Assessment of the service: % very satisfied; % a little satisfied; % a little dissatisfied; % very dissatisfied; qualitative comments (may also include client opinion on applicability of service) Goal Level Competitiveness: Entrepreneur s assessment of the service s contribution to the enterprise s competitiveness: % think the service contributed a lot to competitiveness, % think the service contributed a little to competitiveness, % think the service did not contribute to competitiveness, % think the service harmed competitiveness; qualitative comments Quantity and cost of 3 most important inputs Sales in a given period (eg palm oil sales for 2 weeks 4x a year; for longan, sales once a month for 4 months of pre and harvest period) Quality: as applicable (eg % of total produce in AA and A category for given sales period and/or % of total produce that is off-season) Innovation: as applicable Eco-efficient products/processes as applicable Qualitative comments Control group: Component Level Purchase/receipt of the service from any source: % received service since last interview; % not received since last interview. Of received: % paid; % received for free; source of service; qualitative comments Goal level Competitiveness: Entrepreneur s assessment of the service s contribution to the enterprise s competitiveness (if applicable): % think the service contributed a lot to competitiveness, % think the service contributed a little to competitiveness, % think the service did not contribute to competitiveness, % think the service harmed competitiveness; qualitative comments Quantity and cost of 3 most important inputs Sales in a given period (eg palm oil sales for 2 weeks 4x a year; for longan, sales once a month for 4 months of pre and harvest period) Quality: as applicable (eg % of total produce in AA and A category for given sales period and/or % of total produce that is off-season) Page 15

16 Innovation: as applicable Eco-efficient products/processes as applicable Qualitative comments What needs to be known from providers? Measurement focuses on the service providers, which are the organizations or companies that are actually delivering services to SMEs for a fee. The assessment of other institutions involved in the intervention is optional. There are two types of information that need to be gathered from providers. The first is objective information about the amount of services being delivered to the target group. The second is a subjective assessment by T-G PEC managers on the capacity of the service providers and their ability to sustain and expand service delivery. Objective Information on Service Delivery: Number of enterprises to whom the service is sold (from provider records) Monetary value of transactions for service in specified time period (from provider records) Any data on growth in demand or consumption of the service This information should be gathered from provider records whenever possible. It may also be gathered through interviews with the provider. When possible, the information should be verified by using at least two sources. Subjective Assessment of T-G PEC managers: T-G PEC managers assessment of capacity of provider in critical success factors; chosen from the following: o Attitude and motivation towards commercial service provision o Leadership o Marketing o Product development o Personnel o Product delivery o Finance and technology This assessment should be made based on T-G PEC managers regular interaction with the service providers. As necessary, this can be supplemented by discussions with the service providers at regular intervals. When to assess and document: Baseline: When the provider has been chosen but before significant capacity building has taken place. Interim: Half way through the intervention Final: At the end of the intervention Note: T-G PEC staff will keep informal track of this information regularly throughout the intervention as part of implementation Embedded Services The following table details, what needs to be known from client and control group enterprises, indicators and sample questions for embedded services. Note that if the only goal level indicator is innovation, then a control group is not suggested as they would not provide a good comparison with the target group. Page 16

17 Participating Target Group Data To assess impact on indicators at the Component Level Need to know Indicators Example Questions Perception of the service Experience of the service Change in practice Awareness of better practice and its benefits (can be more than 1 practice if applicable) Satisfaction with the service Change in practice To assess impact on indicators at the Goal Level Have you ever heard of practice for your business? What do you think the benefits of practice can be for your business? Compared to your expectations, how satisfied are you with service from buyer/supplier. Are you implementing practice in your business? If so, how? Need to know Indicators Example Questions Competitiveness Perception of the service s contribution to competitiveness Did this service make a difference to the competitiveness (see definition) of your business? Control Group Data Productivity (inputs/outputs) Sales Quality (eg grades) (if applicable) Innovation (if applicable) Eco-efficiency (if applicable) To assess impact at the Component Level Need to know Indicators Example Questions Perception of the service Experience of the service Awareness of better practice and its benefits Receipt of the service from any source For productivity, a number of questions will be asked related the volume and price of outputs for a given period of time and the costs of the three most important inputs required to produce that particular output. What amount/percent of your products/produce was A grade? Innovation questions will be specific to the expected types of innovation. Eco-efficiency questions will be specific to the expected types of eco-efficiency products or processes. Have you ever heard of practice for your business? What do you think the benefits of practice can be for your business? Have you ever received service from a buyer/supplier, other business or government agency? If so, when was the last time? From whom? To assess impact on indicators at the Goal Level Need to know Indicators Example Questions Competitiveness Perception of the service s contribution to competitiveness (if applicable Did this service make a difference to the competitiveness (see definition) of your business? Productivity (inputs/outputs) Sales For productivity, a number of questions will be asked related the volume and price of outputs for a given period of time and the costs of the three Page 17

18 Quality (i.e. grades) (if applicable) Innovation (if applicable) Eco-efficiency (if applicable) most important inputs required to produce that particular output. What amount/percent of your products/produce was A grade? Innovation questions will be specific to the expected types of innovation. Eco-efficiency questions will be specific to the expected types of eco-efficiency products or processes. Who to interview? Approximately 30 clients of the service can be more if there are many clients. Choose the sample to represent client group. Not more than 10 and not less than 5 control group enterprises o As similar as possible to the client group o Close enough (geographically) to the client group that they face similar conditions o Far enough away (geographically) from client group that the spill over from clients to control group is minimized When to interview? There will be 2 measurements: one before the clients obtain their services (baseline) and one when clients are expected to show changes in business performance (final). Baseline: Clients when the supplier/buyer knows to whom the will deliver the service but before they deliver it Control group within the first month of when clients start getting interviewed Final: When clients can be expected to show changes in terms of business performance according to their business or agricultural cycle Control group at the same time as clients T-G PEC may need to interview enterprises several times during the agricultural cycle or harvest period to gather accurate information on productivity (inputs and outputs), sales and quality (i.e. grades). Up to 4 times is acceptable. Only one final interview needs to gather information on the service. All others should gather information only on business competitiveness. Interview format: Interviews will be one-on-one and in person. It is recommended that 2 interviewers conduct each interview together. The interviewers will work off the questionnaire but may probe further to get accurate answers. Therefore, interviewers must understand the questionnaire and be skilled. For any new interviewer it is strongly recommended to pilot test the questionnaire, in at least three cases with a more experienced interviewer before attempting this on their own. Framework for data presentation: Baseline: For both client and control groups: Component Level Awareness of the service: % aware; % not aware; qualitative comments Page 18