DESIGN OF A PROCEDURE ANALYSIS TOOL (PAT) FOR AFFORDABLE HUMAN FACTORS CERTIFICATION OF AVIATION DEVICES

Transcription

1 DESIGN OF A PROCEDURE ANALYSIS TOOL (PAT) FOR AFFORDABLE HUMAN FACTORS CERTIFICATION OF AVIATION DEVICES New Regulation More thorough device certification DER Team Members: Heather Johnson, Houda Kerkoub, Shaida Latifi, Mumtahina Mahmud, Alexis Nittoli

2 Agenda 2 Context Stakeholders Analysis Problem Statement Proposed Solution Objectives Hierarchy and Requirements Method of Analysis Prototype: Procedure Analysis Tool (PAT) Business Case - Alternatives

3 Flightcrew Errors as Major Accidents Factor commercial jet aircraft accidents between % of error was due to flight crew In 2009, flight crew errors were 56% Source: Statistic analysis of airplane accidents, Copyright

4 In Response to Accidents 4 Complemented by AC Issuance of FAR enforcing Human Factors evaluation in the device certification process. Requirements the applicant1 must show that [ ] flightcrew [ ] can safely perform all of the procedures associated with the systems' and equipment's intended functions Section (b)(2) requires controls and information be accessible and usable by the flightcrew in a manner consistent with the urgency, frequency, and duration of their tasks

5 Airline Standard Operating Procedures (SOP) and Tasks 5 Standard Operating Procedures (SOP): Instructions with purpose of documenting how to perform a routine or emergency activity and to ensure safety, consistency, and quality of a product. SOP can be found in: Airline Operations Manual (part B) Training Manuals Airline Computer Based Training (CBT) Test Case: Swiss European RJ100 (CBT) Item Count Procedures 104 Tasks 642 Sub-tasks 1,263

6 Device 6 Example device is the Multifunction Control Display Unit (MCDU) incorporates: A small screen A keyboard or touchscreen Main input to the flight management computer (automation) Swiss European CBT displays 62 tasks

7 Selected Tasks on Device 7

8 Time on Task 8 LOGINV(RAND(), ) EXPON.DIST(RAND(), 1,1) 500 i=1 NORMINV(RAND(), 2, 1) Replication

9 Tasks and Operator Actions 9 Example Procedure: Change Hold Parameters Stimulus: ATC call: Swiss XX Hold at Present Position, 1 nm legs, left turns, ECHT 1220Z Note: this task is a second task after completing the Hold at Present Position Task, hence the Working Memory retrieval as a trigger to the task Identification # Operator Action (OA) Operator Action Category (OAC) Rand Num 1 from clearance to hold, remember the hold parameters working memory item retrieval Identify task "change the directiion of turn, inbound course, and speed" decision (TSL Id) select functions TURN DIR, INBD CRS, and TGT SPD TSL Select L2: turn dirction is right visual cue: Rare frequency LM: remember TURN DIR means direction of the turn long term memory item retrieval LM: remember you can change the direction by a direct scratchpad entry long term memory item retrieval LM: remember how to use the scratchpad: make the entry then select the field long term memory item retrieval letter L on the alphanumerical keys visual cue: Rare frequency

10 Task Decomposed into 3 Operator Actions (OA) 10 LOGINV(RAND(), ) EXPON.DIST(RAND(), 1,1) 500 i=1 NORMINV(RAND(), 2, 1) Replication

11 Probability of Failure to Complete Procedure 11 L. Sherry and e. al., "Estimating the Benefits of Human Factors Engineering in NextGen Development: Towards a Formal Definition of Pilot Proficiency," in 9th AIAA Aviation Technology, Integration, And Operations Conference (ATIO), 2009.

12 Probability of Failure to Complete Working memory retrieval TSL ID TSL Select Visual Cue Simulating flightcrew performance and keeping track of their timing while executing procedures on a device enables affordable automation for task/usability analysis.

13 Regulation Guidelines (Advisory Circular) 13 Advisory Circular provides guidelines for Methods of Compliance with FAR Focusing on requirement b-2 Implementation of accessible and usable control by flight crew Urgency Frequency Task duration Source: AC Methods for Means of Compliance: Inspection testing Human in the loop testing

14 The Current way of Certification (Pre-Regulation) 14 Inspection Human in the Loop 4 hours per crew per 30 functions crew Tactile evaluation Color differences Position of device and reachability Readability in both sunlight and darkness Captain and first officer (2 crew members) are tested within in a simulator or flight test aircraft This test must be completed times to guarantee statistical significance

15 The Problem: Certification as Required by FAR Inspection Human in the Loop New Regulation More thorough device certification crew 4 hours per crew per 30 functions For 600 functions device, increase above time 20 folds

16 Agenda 16 Context Stakeholders Analysis Problem Statement Proposed Solution Objectives Hierarchy and Requirements Method of Analysis Prototype: Procedure Analysis Tool (PAT) Business Case - Alternatives

17 Stakeholders 17 Aircraft Operator Employs Pilots Provide Airworthiness Certification Federal Aviation Administration Support the Certification Certify HF Device Aircraft Build Product Manager Aviation Manufacturer Pilot Communicate Air Traffic Information Air Traffic Controller Human Factor Specialist (Inspector) Congress Certify HF Device Maintain Safety through Certification Approve Technical Data Push to Sell Device Designated Engineer Representative (DER) Tension Interactions Primary Secondary Tertiary Demand Increase Safety, Lower Cost of Travel Change Regulations Passenger The AC describes Evaluation as assessments of the design conducted by the applicant (or their agent), who then provides a report of the results to the FAA

18 Stakeholder Tensions upon DER 18 FAA Human Factors Specialist Aviation Manufacturer Product Manager Designated Engineering Representative DERs DER must perform certification tests and report to FAA inspector while being employed by applicant for certification

19 Agenda 19 Context Stakeholders Analysis Problem Statement Proposed Solution Objectives Hierarchy and Requirements Method of Analysis Prototype: Procedure Analysis Tool (PAT) Business Case - Alternatives

20 Costs of Tasks Pre and Post FAR Certification time: T Process (x) = T INSP (x) + T HITL (x) = 5x C Process (x) = C INSP (x) + C HITL (x)= 1600x PRE FAR : Evaluation of 30 over 600 tasks ONLY Inspection Costs $500 5 hours Human in the Loop Costs $14, hours Total $14, hours POST FAR : Evaluation of 600 tasks MUST be completed Inspection Costs $10, hours Human in the Loop Costs $979, hours Total $989, hours

21 Problem Statement 21 The new regulation FAR requirement to evaluate all Procedures is cost and time prohibitive if it has to be accomplished by the currently approved Human-in-the-Loop testing. Pre FAR Post FAR $14,500 $989, Hours 1700 Hours Time and Cost increase dramatically due to FAR

22 Additional Problems Stakeholders tension creates pressure on the DER* who is both employed by the applicant and must report to the FAA inspector during the certification process 2. Current means of compliance are subjective 3. Lack of cockpit interaction in the current certification process *DER: Designated Engineering Representative

23 Agenda 23 Context Stakeholders Analysis Problem Statement Proposed Solution Objectives Hierarchy and Requirements Method of Analysis Prototype: Procedure Analysis Tool (PAT) Business Case - Alternatives

24 Concept of Operations 24 The PAT simulates crew performance and tracks their timing while executing procedures on a device enabling affordable automation for task/usability analysis. DER Tool for DER that substitutes the current certification process by PAT testing on ALL tasks. Dedicate HitL testing to only procedures with high PFtoC.

25 Current Process vs. Solution 25 Current Process T Process (x) = T INSP (x) + T HITL (x) = 5x C Process (x) = C INSP (x) + C HITL (x)= 1600x SOLUTION- New Process with PAT T Alt2,2 = T INSP (x) + T PAT +10%T HITL (x) = 2x C Alt2.2 (x) = C INSP + C PAT + 10%C HITL = 500x Note: Calculations based on inspector wage, pilot wage, and inspection time

26 Proposed Solution 26 Provide automated evaluation which is significantly faster and cheaper Eliminate stakeholders tensions by providing rapid, yet objective means of analysis Enable implementation of accessible and usable control and information consistent with (1) urgency, (2) frequency, and (3) task duration

27 Agenda 27 Context Stakeholders Analysis Problem Statement Proposed Solution Objectives Hierarchy and Requirements Method of Analysis Prototype: Procedure Analysis Tool (PAT) Business Case - Alternatives

28 Mission Requirements 28 MR.1 PAT shall enable compliance with the FAR requirements MR.2 PAT shall reduce the post FAR certification cost by at least 65% to match the pre-far cost MR.3 PAT shall remove the stakeholder tensions between the FAA, the Applicant, and the DER

29 Technology Requirements 29 TR1. PAT shall use a Graphical User Interface (GUI) to enable user input and tool output display TR2. PAT shall use a Database to store Operator Performance data TR3. PAT shall accept up to 600 procedures for decomposition TR4. PAT shall use a lookup function to map the OA to their statistical distributions DER TR5. PAT shall use an algorithm to generate random numbers and sum them into total procedure time TR6. PAT shall use Monte Carlo Simulation to simulate 500 operators performing the SOP procedure

30 Objectives Hierarchy for PAT 30 O.1 Maximize Compliance with FAR (w=58%) O.2 Minimize Operational Costs (w=16%) O.2.1 Minimize Set Up Costs O.2.2 Minimize evaluation cost O.3 Optimize Performance (w=26%) O.3.1 Maximize Availability of PAT tool O.3.2 Maximize Maintainability O.3.3 Maximize Reliability of Procedure Analysis Tool

31 Agenda 31 Context Stakeholders Analysis Problem Statement Proposed Solution Objectives Hierarchy and Requirements Method Of Analysis Prototype: Procedure Analysis Tool (PAT) Business Case - Alternatives

32 Process Modeled 32 OA: operator action OAC: operator action categories SOP: standard operating procedure PFtoC: probability of failure to complete

33 IDEF0 Diagram System Steps 33 DER decomposes the procedures into OA and categorizes them OAC PAT maps the OA to its statistical distribution PAT sums up all the OA times PAT performs Monte Carlo simulation PAT displays output DER inputs threshold DER inputs acceptable PFtoC PAT displays results PAT provides option to perform sensitivity analysis DER selects option to perform SA DER inputs SA parameters PAT shows results

34 Human Factors OAC Database 34 Literature Review Experiment Desing not started Operator Action Category (OAC) Distribution button push Normal(0.3, 0.01) decision (TSL Id) Normal(0.5,0.002) decision/choice Lognormal(2,1) flight controls manipulation (feet) Normal(2.31, 0.651) flight controls manipulation (hands) Lognormal(1.12, 0.231) listen to audio (ATC Clearance, Aural Warning) exponential (1) long term memory item retrieval Lognormal(1.21, 2.1) MCC callout Normal(2,1) MCC readback Normal(1.5, 0.2) Talk to ATC (including Readback) exponential (0.9) thrust levers manipulation Triangular(0.1,2,3.5) TSL Select Normal(0.1,0.002) visual check Normal (1, 0.5) visual cue: Routine Normal( 0.632, ) visual cue: Moderate frequency Normal( 1, 0.02 ) visual cue: Rare frequency Normal( 1.632, 0.03 ) working memory item retrieval Lognormal(0.737, 1.21)

35 Frequency Frequency Operator Actions Distribution 35 Working Memory Log (0.737, 1.21) Long term memory Expo(1.21,2.1) Participant 1: Time keeper Completion time (sec) From the Standard Operating Procedures dictate a task. Ex: Press HOLD Press 6R Review the default Press EXEC to activate holding patterns After a 51 second interval Ask designated questions Repeat it for 10 times with 5 subjects 50 data points Tabulate the data in Microsoft Excel Participant 2: Subject Follow Participant 1 s instructions Completion time (sec) Same experiment was performed after 10 days but without showing the task steps to subjects.

36 Sequential Variables Process 36 LOGINV(RAND(), ) EXPON.DIST(RAND(), 1,1) 500 i=1 NORMINV(RAND(), 2, 1) Replication Assumptions: Independent Variables No Dual Tasks Performance

37 Agenda 37 Context Stakeholders Analysis Problem Statement Proposed Solution Objectives Hierarchy and Requirements Method of Analysis Prototype: Procedure Analysis Tool (PAT) Business Case - Alternatives

38 Example Task Decomposition Initialize Position 38

39 Prototype PAT DER enters Decomposition here 2. DER categorizes the OA here 3. PAT generates random numbers here and computes sum on the last row 4. Pressing this button displays the 500 task times here 5. Pressing this button displays frequency distribution

40 Results - 15 Tasks 40 # Procedure PAT Entry time [hr] PAT Run Time [sec] Mean Std Threshold = 3std from mean *PFtoC Percent OAC - Visual Cue (%) 1 Initialize the Position 1: % 47.27% 2 Enter Company Flightplan 1: % 40.82% 4 Check Flgihtplan Progress 1: % 45.16% 5 Switch MCDU ON 0: % 41.18% 6 Enter Runway and SID info 0: % 59.57% 7 Enter flightplan manually 1: % 52.00% 8 Hold at Present Postion 2: % 39.00% 9 Change turn direction, speed 0: % 39.29% 10 Check Climb/Cruise Settings 0: % 48.00% 11 Change Speed and Altitude 0: % 51.51% 12 Select Arrival and Approach 0: % 61.70% 13 Enter a Runway Extension 0: % 41.18% 14 Enter Hold ExitTime 0: % 50.00% 15 Select Speed Restriction 0: % 48.0% -- Total Time 12: Average 00: % 47% Acceptable PFtoC the task was set to 2.5% *PFtoC: Probability of Failure to Complete the task

41 Verification and Validation 41 Preliminary verification and validation results using GMU students performing tasks on a device. Experiment without Cognition Experiment with Cognition Normal Distribution with Mean of 1.06 and Standard Deviation of 0.43 Lognormal Distribution with Mean of 2.93 and Standard Deviation of 1.04

42 Agenda 42 Context Stakeholders Analysis Problem Statement Proposed Solution Objectives Hierarchy and Requirements Method of Analysis Prototype: Procedure Analysis Tool (PAT) Business Case - Alternatives

43 Business Models used for PAT A La Carte Model: $200 per function 2. License Model: $75,000 per year with recurring annual fee of $25,000 thereafter 3. Application Purchase Model: $500,000

44 Market Analysis 44 Market prospection:

45 Acquisition of Customers Phase 1: First 5 years Year 1 Year 2 Year 3 Year 4 Year 5 1 new A La Carte customer 2 new A La Carte customers 1 customer shift to License Model 1 new A La Carte customer 2 customers shift to License Model 7 new customers use License Model 1 customer shifts to Purchase Model Phase 2: Following 5 years Radom numbers generated per following uniformly distributed intervals A la Carte License Purchase New: between(0,3) New between(0,7) Shift to License: between(1,2) Shift to Application: between(0,2) *Lost customer: between(0,1) *Lost customer: between(0,2) Application Purchase New: between(0,1) *Lost: customer 0 * Lost customers modeled over the 10 years period 45

46 Market Penetration per Business Model 46

47 Annual Costs 47 Startup Costs Software development: $250,000 Marketing, sales, business management: $250,000 Recurrent Costs: $250,000 scales with consulting business and product support services growth Time Cumm. Revenue Cumm. Cost Profit ROI year0 $ - $ 500,000 $ (500,000) -200% year1 $ 50,000 $ 750,000 $ (700,000) -193% year2 $ 350,000 $ 1,000,000 $ (650,000) -165% year3 $ 875,000 $ 1,250,000 $ (375,000) -130% year4 $ 2,550,000 $ 1,500,000 $ 1,050,000-30% year5 $ 3,850,000 $ 1,750,000 $ 2,100,000 20%

48 Return on Investment and Breakeven Point 48 At end of year 5, ROI is 20% Breakeven point is forecasted to occur after year 4: ROI -193% -165% -130% -30% 20%

49 Risk Analysis 49 FMEA Matrix (1 to 10 scale) Risk Area Failure Severity (S) likelihood (L) Detectability (D) RPN market size is too small business market is not growing uncertainty regulatory trends are not aligned with service/product not able to deliver service in time operational risks staff issues financial model fall under worse scenario or below (breakeven and ROI) uncertainties recession

50 Conclusions and Recommendations 50 The PAT reduces the evaluation time by 70% per function and therefore solves the problem imposed by FAR The PAT was demonstrated to work on 15 Procedures on the MCDU device. Evaluating the 15 procedures with PAT took 12:47 hr. entry time, sec. average runtime, and analysis showed that PFtoC ranged between 0.4% and 2.6%. Also, 3 over 15 procedures (20%) were flagged for HitL testing. The Visual Cue category gathered the greatest OAC percentage which highlights the importance of semantic cues to recognizing emerging mission situations and prompting the next step of the procedure process. Further exploration of the Operator Action Categories and their impact on the PFtoC is recommended.

51 Thank You

52 Questions?

53 Task Decomposition follows TSL 53 TSL framework showing interaction between all three agents (lifelines) Arrows show flow of information between agents through six steps Cognitive actions are shown as recursive arrows on the Operator TSL Steps are shown in red Top to bottom shows time progression