Role of Human Factor in Automated Data Analysis

Transcription

1 Role of Human Factor in Automated Data Analysis James Benson Technical Executive, EPRI Ratko Vojvodic Technical Consultant, Areva 35 th EPRI SG NDE and Tube Integrity Workshop July 18-20, 2016

2 Human Factor in NDE Overall performance of the NDE examination process is a complex combination of 3 factors Inherent capability of the technique - determined by the physics of the process and by the technology used SG tube defined by its geometry, material properties, accessibility, surface conditions, noise levels is inspected using particular probe technology, ET instrument, acquisition and analysis software to detect degradation of certain size, shape, depth Application parameters surrounding factors impacting technique s performance, processes applied in the preparation, examination and post-examination phase as well as environmental factors of technical nature such as noise originating from external sources, temperature Human factors include personal factors such as skills, training, experience, qualification, motivation, attitude, stress-tolerance, risk taking, awareness but also system factors such as organization of the examination, team work, leadership aspects, commercial pressure 2

3 Human Factor in Data Analysis Human factor is considered as having the most important impact on the results of the manual analysis Automated analysis is expected to address human factor by Improving reliability (how well the process works) Assuring consistency (always performing in similar way) Improving efficiency (good use of time and energy) Selecting appropriate level of autonomy and authority Different thoughts about possible level of addressing human factor Reduce the impact without fundamental changes of the process Minimize the impact by changing the role from doing to controlling Remove it completely from the process by limiting the reach Move it from one phase to the other where it can be managed and used more efficiently 3

4 Simplified Approach Towards Human Factor Review and editing process is often considered as the weakest point of the automated analysis because of human performance error where relevant indication of degradation reported by the automated system may get deleted Error-like situation is more likely to occur when there is an excessive number of false positive calls. Realistic detection and reporting requirements, adequate data quality, appropriate high quality data analysis software toolbox and well prepared system configuration will reduce false calls and make this situation less likely There is no standard definition of what is excessive number of false calls. It will depend on tubing conditions, degradation assessment, analysis instructions, detection and reporting requirements, expectations and automated analysis capabilities and limitations Comparison of human performance in: (1) manual vs. (2) auto analysis should consider attentiveness, awareness and mindset being the same in both processes and perform evaluation based on the actual differences in workload, stress level, expectations, etc. 4

5 Workload Comparison Manual Analyst Manual analysis is repetitive task with heavy workload where same actions are exercised on each tube over and over Scroll through the tube sheet using differential mix channel Review top of tube sheet expansion transition and zone at and above the top of tube sheet using differential mix channel and turbo mix Scroll through the sludge pile region using the raw channel and the differential mix channel Review the region at and above the tube sheet on low frequency channel(s) searching for the loose parts Scroll through the free span straight portion of the tube using one or more raw channels and process channel Review each tube support on process channel, often on raw channel(s) Scroll through U-bend free span and review U-bend structures on raw and process channel Observe in parallel long strip charts for any evidence of loose parts, evidence of tube proximity, tube-to-tube wear, deposits 5

6 Workload Comparison Auto Analysis Review Analyst Significantly smaller number of signals to deal with during review though typically working in parallel on more than one guide tube Indications retained by auto system are displayed one-by-one on the right channel with right span and with measurement taken Analyst will occasionally change the channel when reviewing the signal prior to deciding to either keep the indication or not Typical number of false calls retained by the auto system per tube in replacement steam generators is in the range 0.2 to 2 Significantly smaller workload than in the manual analysis Smaller number of signals to review Lower stress level Same cognitive effort, less fatigue even with false calls significantly above what is typically seen Under these circumstances, deleting valid indication during the editing process is not a result of excessive number of overcalls but rather a result of other human performance factors 6

7 Presence of Human Factor Implementation strategy framework Procedural requirements Scenario selected Reporting requirements, expectations and flow of information Configuration Complex and critical for the performance of auto analysis Verification Complex, targeting apparent and latent errors Production Analysis Typically in the center of interest Review Analysis Same domain as production analysis 7

8 Configuring Automated Analysis Configuration of the automated data analysis system is very complex task which does not have equivalence in manual analysis Human performance errors in this phase, if not identified in a timely manner and corrected, will have detrimental effects on the system performance Being able to thoroughly and independently verify the configuration prior to deployment helps to reduce the probability of error in this phase Configuration is set and verified prior to the examination so the work load and time pressure that typically impact human performance are not present, but all other factors that impact human performance still exist 8

9 Personal and System Factors in the Automated Analysis Human factors describe interactions of individuals with each other, with equipment, processes and management systems Interactions are influenced by the working environment and the culture (safety, technical, social, regulatory, ethical ) Personal factors Skills Training Experience Motivation Questioning attitude Stress and fatigue tolerance Risk taking Awareness System factors Organization of the process Team work Leadership Analyst involvement in pre and post-job tasks Commercial pressure Business factors Technical factors 9

10 Misconceptions about Human Factor Deeper analysis of human errors, not as isolated events but rather in the broader context of the examination process*, shows that the source of error is not only by the analyst but also by the team, technology, organization and even broader in the examination environment replacing manual analysis with automated analysis will not eliminate human errors but it may make it easier to reduce them changing analyst's task from doing to controlling can eliminate one source of error but may create new sources of errors systems approach that builds defense-in-depth through creating conditions that will 1. prevent or reduce errors and 2. mitigate their impact, if they occur, is much more efficient than the person approach which sees the source of errors primarily in the data analyst * - Ref: M.Bertovic, C.Müller, B.Fahlbruch, U.Ronneteg, J.Pitkänen; Holistic Risk Assessment and Risk Prevention Approach to the Mechanized NDT and Inspection Procedure; 5 th European-American Workshop on Reliability of NDE; Berlin,

11 Person vs. Systems Approach Definition of human factors and human performance error Narrow and broad definition of human factors (NDE, elsewhere) Human error human action that failed to achieve intended outcome Traditional or person approach considers the source of human error in the individual committing the error Error could be avoid if individual was more attentive or just tried harder Solution is to add more technology and/or add more procedural requirements It may help but it may also only change how error manifests itself and it may move it elsewhere Systems approach Focuses on the conditions under which the task is performed and identifies the defenses to prevent further occurrences of errors and to minimize their effects; problem may be deeper within the system High hazard potential defense-in-depth applied 11

12 Human Factor Is Here To Stay Solving the human performance issue by designing the automated process, which will take human impact out of the picture, does not seem plausible Automation helps minimizing human performance errors in one phase of the process The role of human will shift from one phase of the process to another, but human will always be the part of the process Process that results in shifting human factor to a more appropriate time and place, where it can be better utilized, verified and controlled, is desirable Shifting data analysis tasks from production analysts to configurators Performing most of work prior to the examination when resources can be managed more efficiently 12

13 How Is Human Factor Being Addressed Human factor within the data analysis process, and within automated data analysis in particular, is being continuously addressed by the industry through: Two party or dual analysis Resolution/review process Independent QDA oversight Generic (AAPDD) and site specific (SSPD) performance demonstration Analysis feedback loop Structured analysis guidelines content Tightly controlled process Data quality requirements for both data and equipment Rev 8 / Appendix L shape auto analysis process further and address some aspects of human factor 13

14 Steps To Improve Human Performance Specific training and instructions know typical steps prior to performing the actual data analysis work Knowing expectations (false calls, hits, misses, expected signals, locating, speed ) Reasonable requirements SSPD (written and practical) Improved ergonomics (control panel, dashboard, signal visualization, integrated baseline display) Insight in the auto analysis process Monitoring auto analysis process 14

15 References M.Bertovic, C.Müller, B.Fahlbruch, U.Ronneteg, J.Pitkänen; Holistic Risk Assessment and Risk Prevention Approach to the Mechanized NDT and Inspection Procedure; 5 th European-American Workshop on Reliability of NDE; Berlin, 2013 R.Parasuraman, V.Riley; Humans and Automation: Use, Misuse, Disuse, Abuse; Human Factors, 1997, 39(2) Human Performance in Nondestructive Inspections and Functional Tests; EPRI NP-6052, Project , 1988 H. M. Stephens Jr, NDE Reliability - Human Factors - Basic Considerations, in Proceedings of the 15th WCNDT, Rome, Italy, 2000 M.Dennis, G.Selby, S.Swilley; Improving NDE Reliability through Performance Demonstration and Attention to Human Factors; 4 th European-American Workshop on Reliability of NDE; Berlin,

16 Together Shaping the Future of Electricity 16