Before we begin, let s talk briefly about why GIS data quality is important.

Transcription

1 1

2 2

3 Before we begin, let s talk briefly about why GIS data quality is important. Regardless of whether you are producing or consuming GIS data, it is important that the data you use meets some agreed upon standard for quality. Poor quality data impacts decision making and is often expensive to fix. For example, utilities need to maintain accurate positions of their assets because incorrect information can impact their completion of construction or repair projects. Organizations could easily lose their reputation by publishing bad data. Think of it this way, you wouldn t publish a document without first running spell checks on it. Equally, you shouldn t use GIS data without first ensuring its quality or fitness for use. 3

4 At Esri, we have built a tool that helps you overcome these difficulties. ArcGIS Data Reviewer is the data quality management application for ArcGIS. It provides workflows for rule based quality control, interactive as well as automated tools, and a way to track and report error information throughout the quality control process. Whether you are a small team or working in an enterprise environment, implementing Data Reviewer provides significant time and money savings by helping you find errors early in the process. Data Reviewer has been available as a standard extension to ArcGIS for Desktop since version 8.0. At 10.1, we are offering it as an extension to ArcGIS for Server as well. 4

5 Lets take a look at the differences between automated and visual QC. We would recommend that you run automated QC first because you can find all major standards Violations up front. Because these review processes are automated with little human interaction, this is the fastest and least expensive form of quality control. This means it is often possible to validate 100% of data. Then on a second pass you should conduct a visual QC to take advantage of what the human eye does well: find missing features or things that are out of place. There are pros and cons to both, but we ll start by talking about automated QC and then discuss the benefits of performing a visual review in the next section. 5

6 [Animated slide] Data Reviewer provides a framework for you to manage the qc life cycle. Typically there are three stages: [click] Review data for errors: Data Reviewer provides a number of tools for both automatically and visually identifying errors, and recording them. [click] Correct errors: Whether it is you, another team member, or a contractor that is correcting the errors, Data Reviewer aids by recording details of who, when and how these errors were resolved. [click] Optionally, you can verify if corrections made are acceptable and that no new issues have been introduced into the data by re-running the same tools used in the review stage. 6

7 In the last section, we discussed the idea of setting quality related requirements or standards for data. We can think of these requirements as a set of rules and, in many cases, automated processes can be developed to validate data against these rules. This is the idea of automated data review. Because these review processes are automated with little human interaction, this is the fastest and least expensive form of quality control. This means it is often possible to validate 100% of data. 7

8 With Data Reviewer, quality requirements can be implemented as rule based Data Reviewer checks. These are a set of configurable data checks that allow you to assess different aspects of your data. Over 40 different checks are available categorized into 11 categories. We will learn a bit more about these checks through a few examples on the next slides. Shown on this slide is the Data Reviewer checks poster. This is a great resource for learning about the available checks and understanding how you can use these checks to validate your own data. This poster may be downloaded from the product website. 8

9 Automated review is an efficient tool for checking data, but it is limited in some ways. For the most part, automated review validates data that already exists and it is only able to validate data against those rules that can be defined and have been implemented. Generally, automated review is not checking for data that doesn t exist (features that are missing from the dataset) and it s not always possible to identify features with incorrect locations or attributes unless those characteristics are in violation of one of the rules. Visual review is the process of manually reviewing the data to assess the data in the ways that the automated review cannot or was not set up to do. This may involve a general inspection of the data and sometimes includes a comparison to another data source. The individual performing the review is using some level of judgment and discretion to assess the data so they must have good knowledge of the data and the quality requirements as well as the ability to make decisions in a consistent manner. Data Reviewer provides a number of ways to simplify this process by providing ways to organize and structure the review along with some tools to write errors found during the review to the Reviewer table. 9

10 Data Reviewer also provides a number of built in reporting capabilities for project managers and QC leads to measure the overall quality of their data. Reports within Data Reviewer include the ability to summarize the number of features validated and the errors returned along with some measure of the overall accuracy of the data based on these results. Reports are generated in Microsoft Excel for organizations to customize and format as required. 10

11 At 10.1, the mechanisms for participating in the quality control process have been greatly expanded. Prior to 10.1, manual QC workflow were implemented using Data Reviewer for Desktop leveraging the extension s tools to facilitate these resource intensive processes. For many non GIS users, their participation in this form of data review has been either through hardcopy plots (or their digital equivalent) or through custom web applications. Data Reviewer for Server expands on participation in this manual process by enabling input from data stakeholders and other interested communities through simple to use map feedback tools. Developers can leverage these tools to implement data quality feedback as a new function of their existing web applications without the burden of implementing the entire QC workflow from scratch. For example, a simple Submit error workflow could be added to an organization s customer facing mapping application. When users of the system discover an error, they can easily submit an error note identifying the source of the problem, location and guidance on what should be done. The feedback enters Data Reviewer s error lifecycle management process where it is reviewed and either rejected or allowed to pass on to technicians for correction as would any other Data Reviewer identified error. The Reviewer workspace serves as a centralized place for managing errors, both those detected using automated checks, as well as, errors detected manually by data consumers. 11

12 12

13 To summarize, ArcGIS Data Reviewer is a standard extension that helps automate, simplify, and improve your data quality control process. It provides tools for rule based automated data validation as well as tools for visual and interactive quality control. Once your errors are found, Data Reviewer helps you track them through the correction and verification process allowing you to summarize and report their progress. It s a great tool to implement irrespective of your industry or the size of your organization, especially in today s economic climate where there is the dual challenge of increasing productivity while maintaining or reducing production costs. 13

14 Before we leave, here are some helpful resources. Information about Data Reviewer can be found on the web at esri.com/datareviewer which provides information on demos, user case studies, key features, requests for free trials, and so forth. For information about instructor-led and free training courses, please visit the Esri training site. We also recently launched a resource center for Data Reviewer at resources.arcgis.com where users can access tutorials, sample scripts, help files, blogs, and forums. If there are additional questions or comments, you are welcome to send us an at datareviewer@esri.com. 14

15 15