Basic information form for auditing

Transcription

1 Basic information form for statistical auditing v6.1 Statistical auditing forms part of internal quality control and development at Statistics Finland. The auditing examines the production process of an individual set of statistics step by step, analysing the employed procedures and imposed demands, and their realisation. A key framework for steering auditing and helping discussion with the target statistics is: Basic information form for auditing The form divides into the following sections: Administrative data, Organisation of work and quality assurance, Interaction with users of statistics, Competence, Information technology, Data acquisition, Data processing and analysis, Release and dissemination, Documentation and archiving, and Development. Further information is available from the steering group of auditing: Pasi Piela (Standards and Methods) Ville Koskinen (Standards and Methods) Ari Leppälahti (Business Statistics) Pekka Mäkelä (Business Statistics) Vesa Virtanen (Population and Social Statistics) Mari Ylä-Jarkko (Economic and Environmental Statistics) Welcome to take part in the survey. IMPORTANT! If you wish to interrupt the survey at any point and carry on with it later, use the Continue later button found at the bottom of the screen. NB! You can use the same button to direct the survey to another person knowledgeable about the statistics or its practices. When continuing responding, the previously given answers are already in the survey.

2 1. ADMINISTRATIVE DATA 1.1 Name of the statistics, choose from the submenu: 1.1 b Other statistics than those above, which? 1.2 Main accounting codes of the statistics: 1.3 Name of the person responsible for the statistics: 1.4 Obligation to provide data for the data collection of the statistics: [ ] Statutory [ ] Voluntary 1.5 Is the production of the statistics based on an EU regulation? ( ) Yes, on a gentlemen's agreement, ( ) Yes, on basic Regulation No

3 2. ORGANISATION OF WORK AND QUALITY ASSURANCE 2.1 What objectives are set for the production of the statistics in connection with annual planning apart from the publication schedule? 2.2 a How are the phases of the production process defined? ( ) Only the completion of the publication has a predetermined due date ( ) The data collection and completion of the publication have predetermined due dates ( ) Several work stages have predetermined due dates 2.2b Is the schedule of the production process defined in writing? ( ) Yes ( ) Yes, partly 2.2c Has there been problems in keeping to the schedule? ( ) Keeping to the schedule has often required overtime work ( ) Keeping to the schedule has at times required overtime work, longer working hours or additional resources ( ) We have kept to the schedule within the normal working hours 2.2 d Comments 2.3 a Has a person been named for all tasks of the production process? ( ) Yes, the division of responsibilities is clear and the distribution of work is documented ( ) Yes, but the distribution of work is not documented comprehensively 2.3 b Comments:

4 2.4 a Have deputies been named for all tasks of the production process? ( ) Yes ( ) Yes, for some 2.4b Is the arrangement of deputies documented? ( ) Yes, fully ( ) Yes, in some parts 2.4c How is the competence of deputies ensured? [ ] The tasks are rotated systematically [ ] The deputy acts independently in the task at least once a year [ ] The deputy takes regularly part in the task together with the actual responsibility person [ ] In some other way, how? 2.4 d Comments 2.5 How are the support services for regular statistics production arranged? ( ) The support services are contacted through an electronic system (e.g. Mantis). The functioning of the service is monitored regularly. ( ) The support services have named contact persons and their background teams with whom we are regularly in contact. ( ) There is a named contact person in the support services, but in practice no deputy. contact person has been named in the support services and we have to look for help separately for each problem. [ ] In some other way, how? 2.6 a At which stage is the quality of work assured? ( ) Systematically at the end of major work phases ( ) Systematically at the end of the production round ( ) Where needed, based on consideration 2.6b How is quality assurance organised? ( ) Quality objectives have been defined and their realisation is assured with standard procedures ( ) Quality objectives have been defined and each is responsible for their realisation as regards their own solutions ( ) Quality objectives have not been separately defined

5 2.6 c Comments: 2.7 Have the critical risks for the production process been identified? ( ) Yes 2.8 Which matters have caused errors or touch and go situations for statistics production? 2.9 In your opinion, which are your main strengths in organisation of work and quality assurance that could be applied to other statistics as well?

6 3. INTERACTION WITH USERS OF STATISTICS 3.1 How have the user groups of statistics been identified? ( ) User groups have been surveyed and documented ( ) Some user groups have been identified through communication and other channels ( ) Information about user groups has not been collected 3.2 a In what different ways do you communicate with users? [ ] Matters concerning the statistics are discussed in a co-operation group or similar [ ] Meetings (or courses or seminars on the topic) are arranged with users at least yearly [ ] Users are sent notifications about topical issues related to the statistics [ ] Communication is dependent on the users' activity 3.2 b Describe co-operation modes in more detail: 3.3 a About which matters have you discussed with users lately? [ ] Availability of statistical data [ ] The topic of the statistics [ ] Interpretation of the results of the statistics [ ] Quality of the statistics [ ] Development needs of the statistics 3.3b On which matters do users most put development pressures? [ ] Extending data content [ ] Improving accuracy of results [ ] Lowering the release lag [ ] Improving comparability [ ] Improving usability 3.3 c Describe other development needs communicated by users:

7 3.4 Are there any significant unfulfilled information needs of users in the topic of the statistics? ( ) Yes, several ( ) Yes, some ( ) Unfulfilled information needs are not known ( ) There are no unfulfilled information needs 3.4 b If yes, why? 3.5 a Have the delivery schedules of customers' information requests had to be lengthened from normal due to lack of time or resources? [ ] No [ ] We have had to lengthen the agreed schedule at times [ ] Sometimes we have had to agree a longer schedule than normal [ ] We have had to turn down information requests 3.5 c Comments related to information service: 3.6 In your opinion, which are your main strengths in management of user relationships that could be applied to other statistics as well?

8 4. COMPETENCE 4.1a Which of the following alternatives best describes the work experience of the persons involved in the production of the statistics? ( ) Most have quite a long work experience in the statistics ( ) There are both experienced and less experienced persons ( ) Many still have quite little experience in the statistics 4.1b Comments: 4.2a Assess the sufficiency of the available competence (incl. support services): Lack of competence sometimes makes it difficult to solve problem situations Competen ce is sufficient for regular activities Competence gives good grounds for developing the statistics Topic knowledge (e.g. concepts, international recommendations) Methodological competence (e.g. statistical methods, data analysis and processing rules) Publication and product competence (e.g. presentation of statistical data, release practices) 4.2b If there have been shortcomings in the sufficiency of competence, describe what they are like:

9 4.3a Is there enough time for competence development? ( ) There is enough time for participation in personnel training and self-directed maintenance of professional skills ( ) It is not possible to take part in all the necessary training ( ) It is difficult to find time for competence development 4.3 b Comments: 4.4 a Are the work instructions of the statistics such that a new person can resort to them to a sufficient degree? ( ) Yes, the instructions have been gone through together and found sufficient ( ) Yes, there are instructions for most work stages ( ) There are not work instructions for all work stages 4.4 b Comments: 4.5 How is it assured that the legislation, decrees, principles and Statistics Finland's internal guidelines steering statistics production are known well enough? [ ] They are always gone through during job familiarisation for a new employee [ ] They are discussed in departmental information events or at unit or team meetings [ ] They have been discussed in study circles or similar over the last two years [ ] We have not gone through them together, but there is a person working at the statistics who knows them in depth [ ] In some other way, how? 4.6 In your opinion, which are your main strengths in competence management and job familiarisation that could be applied to other statistics as well?

10 5. INFORMATION TECHNOLOGY 5.1 With which tools are the IT solutions used in different process stages mainly implemented? Data acquisition (e.g. electronic form, reception of administrative data) Data editing (e.g. implementation of editing and implementation methods, application with which microdata are processed manually) Analysis (e.g. calculation programs, reports used as support to quality assurance) Release and dissemination (e.g. production of appendix tables and figures, production of Statfin tables for the release) 5.2a Is the information system of the statistics made following Statistics Finland's SAS or.net architectural models? ( ) Yes, ( ) Yes, partly 5.2b Comments:

11 5.3b Is the information system documented? Requirement specification Yes No DNK Use case descriptions Architectural description: description of the application, information and technological architecture of the information system Data model: description of the concepts, files and databases used in the information system Information system functions: description of main functions Program code: source code and its comments Testing plan and report: description of the system testing 5.3b Comments:

12 5.4 How much are IT experts from support services needed in statistics production? Data acquisition Data editing Analysis Release and dissemination System maintenance [ ] Continuously [ ] Often [ ] Sometimes [ ] Rarely [ ] Continuously [ ] Often [ ] Sometimes [ ] Rarely [ ] Continuously [ ] Often [ ] Sometimes [ ] Rarely [ ] Continuously [ ] Often [ ] Sometimes [ ] Rarely [ ] Continuously [ ] Often [ ] Sometimes [ ] Rarely Comments 5.5a How well do the present IT solutions of the information system serve the needs of the statistics? Data acquisition Data editing Well, is sufficiently efficient; it has the most important features and it is reliable Satisfactorily, there are shortcomings but they do not significantly hinder work Analysis Release and dissemination Weakly, shortcomings slow down work or weaken the quality of work 5.5b Comments:

13 In your opinion, which are your main strengths in management of information technology that could be applied to other statistics as well? 6. DATA ACQUISITION 6.1 What is the target population of the statistics (i.e. the population to which the results are meant to be generalised)? 6.2 Does the used sampling frame or frame register correspond to the needs of the statistics? The statistical units in the frame correspond to the units to be included in the statistics Yes To some extent No The coverage of the frame is sufficient The frame is sufficiently up-to-date The frame has sufficient information about the units to be included in the statistics (e.g. classification, contact information)

14 6.2b Comments on the challenges of the frame:

15 6.3a Is direct data collection utilised in the statistics? ( ) Yes IF NOT, THEN JUMP TO QUESTION b Describe in brief the sampling method used in drawing the population: 6.4 To what extent have the following affected development of data collection (or optimisation of data collection)? Lowering costs Reducing the response burden Reducing nonresponse Improving accuracy Speeding up production [ ] Significantly To some [ ] extent Less than [ ] other objectives [ ] Not relevant [ ] Significantly To some [ ] extent Less than [ ] other objectives [ ] Not relevant [ ] Significantly To some [ ] extent Less than [ ] other objectives [ ] Not relevant [ ] Significantly To some [ ] extent Less than [ ] other objectives [ ] Not relevant [ ] Significantly To some [ ] extent Less than [ ] other objectives [ ] Not relevant Comments

16 6.5 How common is it that the concepts in the data collection form need to be clarified to data suppliers? ( ) Often ( ) At times ( ) Rarely or never 6.6 a How is the quality of direct data collection ensured? [ ] The sample is checked regularly [ ] The data collection form has been tested [ ] Data collection is monitored by means of quality indicators (e.g. paradata) [ ] The quality of data collection is assured with other methods (describe in the Comments field) [ ] Not relevant 6.6 b Comments: 6.7 Is non-response regularly monitored during data collecting? ( ) Yes, as produced by the system ( ) Yes, in some other way t relevant 6.8 How is the effect of non-response on the statistical data to be produced taken into account during data collection? 6.9 How do you try to prevent non-response?

17 6.10 If administrative data are utilised in the statistics, how is the quality assured? [ ] The suitability of the definitions of the target units and variables of the data has been assured [ ] The coverage of the data is followed continuously [ ] Technical validation is made to the data upon reception [ ] The data are compared with other data sources [ ] In other ways (describe in the Comments field) 6.10 b Comments: 6.11 Have the persons responsible for releasing and receiving the administrative data needed for the production of the statistics been named? ( ) Yes t relevant 6.12 If the same basic data are used in other statistics, is checking made in them separately and is overlapping work thus possible? ( ) Data checking is centralised ( ) The data are checked separately in the statistics and overlapping checking is possible ( ) We have no knowledge of checking made in other statistics t relevant ( ) In some other way, how? 6.13 In your opinion, which are your main strengths in data acquisition that could be applied to other statistics as well?

18 7. DATA PROCESSING 7.1 a Are the methods and definitions used in the statistics in line with international recommendations? ( ) Yes ( ) To a large part ( ) We depart from international recommendations international recommendations have been given 7.1 b Describe and give briefly the reasons for departing from the recommendations: 7.2 How is the Classification Database/warehouse utilised? ( ) The information system uses classifications directly from the Classification Database ( ) The publication tables use classifications directly from the Classification Database ( ) The Classification Database is utilised by copying classifications to the system t utilised 7.3 a How are errors and other problems connected with the data identified? [ ] Automatic logicality checks are made to the data [ ] The data are analysed by means of graphical analyses, distributions and key figures [ ] The data area examined on the micro level based on expert consideration [ ] In other ways (describe in the Comments field) 7.3 b Comments: 7.4 If deduction (e.g. coding of classification variables) is made based on the collected data, how is its quality assured?

19 7.5 Which of the following alternatives best describe data checking and editing? ( ) Mainly manual ( ) To some extent automated ( ) Largely automated 7.5 b Comments: 7.6 a Can the results of the statistics be viewed preliminarily during the processing of statistical data? ( ) Yes, all the figures to be published can be produced as needed and deductions can be made on the current state of the statistical data ( ) Only the most important key figures can be estimated ( ) Only for certain parts ( ) It is not possible 7.6b Is the effect of the corrections made in connection with data checking on the results assessed? ( ) Yes, as a built-in feature in the system ( ) Yes, but they are assessed only occasionally ( ) In principle possible, but not much assessed 7.6 c Comments: 7.7 What other information is used in the production of the statistics in addition to the actual data? [ ] News relating to the topic [ ] Results of other statistics [ ] External expert assessments [ ] Internal expert assessments [ ] Statistical forecasts [ ] Other information, what?

20 7.8 a How are unbiasedness and accuracy of the data assessed? [ ] With revision monitoring [ ] With statistical key figures (e.g. confidence interval estimates) [ ] By comparing with other data [ ] In other ways (describe in the Comments field) 7.8 b Comments on assessing accuracy of data: 7.9 Which are the main sources of error independent of sampling and how is their significance assessed? 7.10 How can the changes occurring at different stages in the data be traced? [ ] Original, wholly unprocessed microdata are stored permanently [ ] Final data on which published data are based are stored permanently [ ] Changes made to the data are flagged (marked in the data) [ ] Corrections made to the data are stored without deleting the original data to be corrected In your opinion, which are your main strengths in data processing that could be applied to other statistics as well?

21 8. RELEASE AND DISSEMINATION 8.1 a With which other data sources could the data published in the statistics be compared? 8.1b Is it regularly assured that differences to the above-mentioned data sources are identified? ( ) Yes ( ) Yes, but only now and again ( ) Differences to other data sources have not been examined 8.1 c Comments: 8.2 c In your opinion, are the quality description and other background material offered to users (user's handbook, etc.) comprehensive enough? ( ) Yes, the material is maintained constantly ( ) Yes, but the material is not quite up-to-date ( ) The material should be developed 8.2 b Comments: 8.3 How are the proofreading and other checking of the final statistical release arranged? 8.4 a Are the protection guidelines of the tables to be published available in writing? ( ) Yes need for protection, why not?

22 8.4 b Describe briefly the need specification of data protection for the tables to be published (e.g. threshold or dominance rule) and the used protection method (e.g. suppression), if relevant: 8.5 In what ways is the media coverage of the statistical publication followed in the statistics? 8.6 Do people participating in the production of the statistics write articles related to the topic? ( ) Yes, articles are published at least yearly ( ) Yes, sometimes 8.7 In your opinion, which are your main strengths in data dissemination that could be applied to other statistics as well?

23 9. DOCUMENTATION AND ARCHIVING 9.1 Are the concepts and classifications of the statistics and their descriptions up-to-date in the Classification and Concepts Databases? ( ) Yes, the descriptions are checked regularly ( ) Yes, partly 9.2 a Are the key changes to the data content of the statistics and their effects on the comparability of data documented? ( ) Yes ( ) Yes, partly 9.2 b Comments: 9.3 a Is there a clear and understandable process description of the production stages of the statistics for general use (e.g. for job familiarisation)? [ ] There is a text-format description and a process diagram of the process flow [ ] The process stages are described in connection with the schedule, work instructions or other such document [ ] The process description is scattered or non-existent 9.3b Is there an up-to-date and detailed process description (applicable to the statistics own use)? ( ) Yes 9.3 c Comments: 9.4 How is archiving managed? ( ) As an automated part of regular statistics production ( ) As a separate process t known

24 9.5 a Are checking and corrections documented so that the reasons for the solutions made could be clarified where needed? ( ) Yes, comprehensively ( ) Yes, in some parts 9.5 b Comments: 9.6 In your opinion, which are your main strengths in documentation that could be applied to other statistics as well?

25 10. DEVELOPMENT 10.1 Which performance capacity indicators are used in the statistics to aid continuous improvement? 10.2 How has the data content or product selection of the statistics been developed in recent years? 10.3 Do you participate actively in European or other international co-operation in the topic of the statistics? ( ) Yes t recently 10.4 Which actions have resulted from the assessment of the work process or feedback received from users or data suppliers in recent years? 10.5 a Has external auditing been performed on the statistics or has quality reporting been done for an outside party? ( ) Yes 10.5 b Briefly describe the auditing or quality reporting made: 10.6 Last, we ask about the most important areas which in your opinion require development measures in the near future (e.g. production process, personnel, information technology)?

26 11.. FEEDBACK ON THE SURVEY 11.1 Feedback on the basic information form for auditing: