White Paper Addicam Sanjay Software Architect Shahzad Malik Software Architect Abhishek Ranjan Software Engineer Shweta Phadnis Software Engineer Metrics for Anonymous Video Analytics on Digital Signage August 2011 Intel Corporation 325967
Metrics for Anonymous Video Analytics on Digital Signage Executive Summary Anonymous Video Analytics (AVA) is a passive and automated audience measurement technology designed for digital signage networks that can be used to provide digital signage operators with quantitative viewership information. Although AVA promises to bring accountability into the digital signage industry, there are still no industry standard metrics that can be used to assess the accuracy of any particular AVA system. In this paper we will describe a methodology and present a set of accuracy and error metrics that we believe will help to characterize the various parameters in most AVA systems, such as viewer counts, dwell times, and demographic (gender/age) classification accuracy. The basic question this paper attempts to address is whether you can trust the anonymous video analytic data which you have collected with a particular system. By demonstrating how these metrics can be used on an actual AVA system, we hope to move the industry towards a standard set of metrics that may help advertisers and digital signage operators make a more informed decision when assessing the various video analytic solutions available on the market. Our metrics can also be used by AVA solution providers to determine the strengths and weaknesses of their systems in order to improve the quality of their measurement technology. The Intel Embedded Design Center provides qualified developers with web-based access to technical resources. Access Intel Confidential design materials, step-by step guidance, application reference solutions, training, Intel s tool loaner program, and connect with an e-help desk and the embedded community. Design Fast. Design Smart. Get started today. http://www.intel.com/p/en_us/embedded. 2
Contents Business Challenge... 4 Solution... 4 Measuring AVA Accuracy... 4 Proposed AVA Metrics... 7 Track Match Error (TME)... 7 Gender Match Error (GME)... 8 Age Match Error (AME)... 9 False Positive Error (FPE)... 9 False Negative Error (FNE)... 10 Impression Count Error (ICE)... 11 Dwell Time Error (DTE)... 11 Conclusion... 13 References... 13 3
Metrics for Anonymous Video Analytics on Digital Signage Business Challenge Digital signage is the term that is often used to describe the use of a display screen for news, advertisements, local announcements, and other multimedia content in public venues such as restaurants or malls [3]. Recent technologies such as Anonymous Video Analytics (AVA) are now available on the market to help digital signage operators capture audience impression data. This is accomplished using standard, low-cost optical sensors embedded into the display panel along with some sophisticated computer software to analyze the video feed of the audience in real-time [1]. The data gathered by such an AVA system includes the total number of viewers, average attention span, and even the gender and age of viewers as they look at a screen. While many existing AVA technologies claim accuracy levels above 90% in most cases, there are currently no standard approaches for a third-party to verify these claims and compare the various AVA solutions side-by-side. Inaccuracies may occur due to sensor limitations (such as resolution or image quality), processing limitations (such as the speed of the computer system), and even environmental conditions (such as lighting or shadows in the area around the digital screen). Solution In this paper, we will describe a methodology for measuring the accuracy of data collected by an AVA system. We present a variety of performance metrics that can be used to provide a quantitative assessment of how closely the data gathered by an AVA system (AVA Data) matches the actual data or ground truth (GT) data. The metrics proposed in the paper measure the accuracy of the following aspects of the data: viewer location, gender, age, viewer detection, viewer missed, impression counts, and dwell time. In the following sections, we will define an accuracy measurement metric for each aspect; describe the motivation behind it, how to compute it, and how to interpret the metric. Measuring AVA Accuracy Our approach to measuring AVA accuracy is to compare the output of an AVA system with the Ground Truth (GT) data that has been collected for a test video. Our methodology assumes that the AVA system uses a face detection and tracking algorithm to detect viewers, where viewers are relatively frontfacing towards an optical sensor whenever they are looking towards a digital sign [2]. Therefore, the test video should also be prepared so that it closely mimics the scene as viewed by an optical sensor installed directly above or below the sign. Depending on the various aspects of data being evaluated, the video should contain viewers with appropriately large representation of 4
different demographics as they move in front of a digital sign. Once such a test video is obtained, the computation of the various metrics commences which is a multi-step process (see Figure 1). Figure 1. Performance comparison methodology. The first step in this process is to obtain both the GT data and the AVA test data for the test video. The GT data can be collected by manually marking the viewers in each video frame and taking note of all the relevant information (such as assigning an identifier to a viewer, marking the position and size of a box around the viewer s face, and taking note of the viewer s gender and age group). This is typically a one-time process for each test video. The AVA test data can then be generated automatically by processing the same video through the AVA application and, for each frame, outputting marked AVA test data in a format similar to the GT data. 5
Metrics for Anonymous Video Analytics on Digital Signage Figure 2. An example of AVA data and GT data after matching shown pictorially AVA Output feature track length (A1-A5) GT Output feature track length (B1-B5) Image Y axis A1 B1 A2 B2 A3 B3 B4 A4 Image X axis The next step is to match viewers that were marked in the GT data with the corresponding viewer in the AVA test data for each frame (if such a correspondence exists). This can be accomplished for each video frame as follows: - For each viewer detected in the GT data, the radial distance to all viewers in the AVA test data is computed (see Figure 3). A viewer correspondence between the GT box and the closest AVA box is made if the radial distance is below a predefined threshold (measured in pixels). - If a correspondence cannot be made for a GT box, then it is assumed that the AVA system failed to detect a viewer for that particular frame. This is considered to be a false negative. - If a correspondence cannot be made for an AVA box, then it is assumed that the AVA system incorrectly detected a viewer for that particular frame. This is considered to be a false positive. Figure 2 shows a typical example of the two types of data represented pictorially after they have been matched. Once all matches have been made, the accuracy metrics can be computed. 6
Figure 3. Viewer matching between GT data and AVA output using radial distance Proposed AVA Metrics In this section we define the metrics and describe how they should be used. Each metric has a range of possible values that it can take depending on the AVA software. We also provide an interpretation of these values. Track Match Error (TME) The TME metric assesses the accuracy of viewer location measurement. It is the average Euclidean distance between the location of a viewer in the AVA data and the corresponding viewer in the GT data. The Euclidean distance is the root of square differences between coordinates of a pair of objects. The TME can be computed as follows: where N is the total number of frames being processed, On is the total number of objects matched in the n-th frame, and Dni is the Euclidean distance between the i-th matched object in the n-th frame. Lower values of the TME should be interpreted as more accurate object tracking in the AVA output. The values for the TME metric should be interpreted as follows: 7
Metrics for Anonymous Video Analytics on Digital Signage Value range. TME can take any value from 0 to D, where D is the diagonal dimension of the input video frame being processed. The value 0 is assumed when the AVA output tracks exactly match the GT output tracks. This is the perfect tracking scenario. The value D is assumed when AVA output tracks are separated from the GT output tracks by the maximum possible distance. Best value: 0. All AVA software should target to achieve a TME value close to 0. Gender Match Error (GME) The GME metric assesses how accurately the AVA system assigns gender to a viewer. It is assumed that the gender of a viewer is tracked across all the frames for both the AVA output and GT data. where N is the total number of frames being processed, On is the total number of objects matched in the n-th frame, Gni is 0 if gender match happened for the i-th object in the n-th frame and 1 otherwise. The values for the GME should be interpreted as follows: Value range. GME can take any value from 0 to 1. The value 0 is assumed when the AVA output gender exactly matches the GT output. This is the perfect gender detection scenario. The value 1 is assumed when for each face the AVA software assigns gender differently from the corresponding assignment in the GT data. Best value: 0. All AVA software should target to achieve a GME value close to 0. The further the GME value is from 0, the higher is the error in gender detection. 8
Age Match Error (AME) The AME metric assesses how accurately the AVA system assigns an age bracket to a viewer. The age of a viewer is tracked across all the frames for both the AVA output and GT data. Where, N is the total number of frames being processed, On is the total number of objects matched in the n-th frame, Ani is 0 if age match happened for the i-th object in the n-th frame and 1 otherwise. The values for the AME should be interpreted as follows: Value range. AME can take any value from 0 to 1. The value 0 is assumed when the AVA output age bracket exactly matches the GT output. This is the perfect age detection scenario. The value 1 is assumed when for each face the AVA software assigns an age bracket differently from the corresponding assignment in the GT data. Best value: 0. All AVA software should target to achieve an AME value close to 0. The further the AME value is from 0, the higher is the error in age detection. False Positive Error (FPE) The FPE metric assesses the rate of false viewers detected by the AVA system. If a viewer is detected by the AVA system but there is no corresponding viewer in the GT data, then it is called a False Positive error. The FPE metric can be formally defined as follows: where N is the total number of frames being processed and Pn is the number of false positive in the n-th frame. The values for the FPE should be interpreted as follows: 9
Metrics for Anonymous Video Analytics on Digital Signage Value range. FPE can take any value greater than or equal to 0. The value 0 is assumed when there are no false positives in the AVA output, which is the goal of any AVA system. Higher values correspond to increasing false positive rates. Best value: 0. All AVA software should target to achieve a FPE value close to 0. The further the FPE value is from 0, the higher is the chance of invalid detections. False Negative Error (FNE) The FNE metric assesses the rate at which the AVA system fails to detect viewers. If a viewer is not detected by the AVA system but it is present in GT data, then it is called a False Negative error. The FNE can also be defined as the average number of faces missed by the AVA system per frame. where N is the total number of frames being processed and Fn is the number of false negatives in the n-th frame. The values for the FNE should be interpreted as follows: Value range. FNE can take any value greater than or equal to 0. The value 0 is assumed when there are no false negatives in the AVA output, which is the goal of any AVA system. Higher values correspond to increasing false negative rates. Best value: 0. All AVA software should target to achieve a FNE value close to 0. The further the FNE value is from 0, the higher is the chance of missed detections. 10
Impression Count Error (ICE) The ICE metric assesses the accuracy of the overall impression count as estimated by the AVA system. The ICE metric is a ratio of the absolute difference between AVA and GT total viewer counts and GT total viewer counts. It provides an idea of whether the AVA system is over counting or under counting, and by what percentage. This metric is only relevant when there is at least one viewer in the video being processed. The values for the ICE should be interpreted as follows: Value range. ICE can take both positive and negative values. It can range from -1 to close to AVA Impression count reported by AVA software. A negative value indicates under counting by the AVA system and a positive value indicates over counting. A value of 0 indicates 100% accurate impression count by AVA with respect to the GT data. A value close to 0 indicates low error rate in impression count. As the ICE values goes away from 0, the error rate gets higher. Best value: 0. All AVA software should target to achieve an ICE value close to 0. The further the ICE value is from 0, the higher is the error in impression counts. Dwell Time Error (DTE) The DTE metric provides a measure of how well the AVA system determines the dwell time of a viewer for the digital sign. First, the Average AVA Dwell Time is calculated as follows: where V is the total number of viewers as detected by AVA and Tj is the dwell time for the i-th viewer. 11
Metrics for Anonymous Video Analytics on Digital Signage where U is the total number of viewers as reported in GT data and Sj is the dwell time for the j-th viewer. Using the Average AVA Dwell time and the Average GT Dwell Time, DTE is calculated as follows: The values for the DTE should be interpreted as follows: Value range: DTE can take any value greater than or equal to 0. The number represents the error in dwell time as calculated by AVA software with respect to that calculated in GT data. Best value: 0. All AVA software should target to achieve a DTE value close to 0. The further the DTE value is from 0, the higher is the error in impression counts. 12
Conclusion In this paper, we presented an accuracy measurement methodology and a set of metrics that we hope will help advertisers and digital signage operators make more informed decisions when assessing the various video analytic solutions available on the market. The goal of our metrics was to measure the viewership detection, tracking, and classification components of an AVA system, using parameters which we feel are applicable to any AVA technology that uses face detection algorithms to detect viewers. Our hope is that such methodologies can move the digital signage industry towards adopting a standard set of metrics to fairly and effectively assess the capabilities of an AVA system, while also helping solution providers to determine the strengths and weaknesses of their systems in order to improve the quality of their measurement technology. The Intel Embedded Design Center provides qualified developers with webbased access to technical resources. Access Intel Confidential design materials, step-by step guidance, application reference solutions, training, Intel s tool loaner program, and connect with an e-help desk and the embedded community. Design Fast. Design Smart. Get started today. http://www.intel.com/p/en_us/embedded. Authors Addicam Sanjay is a Software Architect with Digital Signage at Intel Corporation. Shahzad Malik is a Software Architect with Digital Signage at Intel Corporation. Abhishek Ranjan is a Software Engineer with Digital Signage at Intel Corporation. Shweta Phadnis is a Software Engineer with Digital Signage at Intel Corporation. Acronyms AVA Anonymous Video Analytics DS GT Digital Signage Ground Truth References 1. Arel, S. Video Analytics for Digital Signage Deployments. DigitalSignageToday.com. 13
Metrics for Anonymous Video Analytics on Digital Signage http://download.intel.com/design/intarch/platforms/digitalsignage/intel_ WP_Video_Analytics_To_Launch.pdf 2. White Paper. A Report on a Field Trial of Anonymous Video Analytics (AVA) in Digital Signage. Intel Corporation. http://download.intel.com/embedded/applications/digitalsignage/325223. pdf 3. Solution Brief. Reaching the Right Audience with Digital Signage Systems. Intel Corporation. http://www.intel.com/design/intarch/platforms/digitalsignage/322038.pdf 14
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR. Intel may make changes to specifications and product descriptions at any time, without notice. This paper is for informational purposes only. THIS DOCUMENT IS PROVIDED "AS IS" WITH NO WARRANTIES WHATSOEVER, INCLUDING ANY WARRANTY OF MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR ANY PARTICULAR PURPOSE, OR ANY WARRANTY OTHERWISE ARISING OUT OF ANY PROPOSAL, SPECIFICATION OR SAMPLE. Intel disclaims all liability, including liability for infringement of any proprietary rights, relating to use of information in this specification. No license, express or implied, by estoppel or otherwise, to any intellectual property rights is granted herein. Intel, the Intel logo, Intel Core, Intel vpro and vpro Inside are trademarks of Intel Corporation in the U.S. and other countries. *Other names and brands may be claimed as the property of others. Copyright 2011 Intel Corporation. All rights reserved. 15