Critical STAT Questions DoD Leadership Should Ask:

Transcription

1 Critical STAT Questins DD Leadership Shuld Ask: Injecting Rigr int the Test and Evaluatin Prcess thrugh Effective Inquiry Authred by: Jennifer Kensler, PhD Francisc Ortiz, PhD Lenny Truett, PhD Jim Crdeir, PhD Luis Crtes Michael Harman 18 March 2015 The gal f the STAT T&E COE is t assist in develping rigrus, defensible test strategies t mre effectively quantify and characterize system perfrmance and prvide infrmatin that reduces risk. This and ther COE prducts are available at STAT T&E Center f Excellence 2950 Hbsn Way Wright-Pattersn AFB, OH 45433

2 Table f Cntents Executive Summary... 2 Intrductin... 2 Questins t Ask... 2 STAT Differences in Develpmental (DT) and Operatinal Testing (OT)... 3 Strategy/Planning... 4 Design Develpment... 5 Pre-Executin Planning... 6 Analysis Planning (befre any data is cllected)... 6 Pst-Executin Review... 7 Analysis Review (after data has been cllected)... 7 Cnclusin... 7

3 Executive Summary The Deputy Assistant Secretary f Defense (Develpmental Test and Evaluatin) (DASD(DT&E)) and Directr Operatinal Test and Evaluatin (DOT&E) have advcated the use f mre rigrus test methds t ensure risk is identified and quantified as early as pssible. Scientific test and analysis techniques (STAT) are the scientific and statistical methds and prcesses which enable the develpment f efficient, rigrus test strategies that will effectively quantify and characterize system perfrmance and prvide infrmatin that reduces risk and infrms decisins. This paper fcuses n the leadership aspect f the planning prcess; playing the quality assurance rle in identifying if the prpsed testing will be sufficient. Critical leadership questins are prpsed alng with acceptable respnse cntent. Keywrds: STAT, scientific test and analysis techniques, leadership, test and evaluatin, critical questins, test planning, test readiness review The views expressed in this paper are thse f the authr(s) and d nt reflect the fficial plicy r psitin f the United States Air Frce, the Department f Defense, r the United States Gvernment. Intrductin The Deputy Assistant Secretary f Defense (Develpmental Test and Evaluatin) (DASD(DT&E)) and Directr Operatinal Test and Evaluatin (DOT&E) have advcated the use f mre rigrus test methds t ensure risk is identified and quantified as early as pssible. Scientific test and analysis techniques (STAT) are the scientific and statistical methds and prcesses which enable the develpment f efficient, rigrus test strategies that will yield defensible results. STAT encmpasses such techniques as design f experiments, bservatinal studies, reliability grwth, and survey design. The suitability f each methd is determined by the specific bjective(s) f the test. These techniques can be applied t a brad array f missins, systems, and functins. Despite this wide applicability, sme testing des nt lend itself t the applicatin f STAT. This is typically due t existing standards r safety reasns which dictate hw a test must be perfrmed. Test planners shuld make every effrt t apply STAT until a reasn t deviate is justified. This paper fcuses n the leadership quality assurance rle f the STAT planning prcess. It utlines the critical questins that thse in a leadership psitin shuld be asking. These questins are apprpriate at Test Plan Wrking Grups (TPWG), Wrking Integrated Prduct Team (WIPT) meetings, Test Readiness Reviews (TRRs), and milestne reviews. Their purpse is t help the leader understand if the prpsed testing will be sufficient and ensure prgram risk is quantified and minimized. Nte: Practitiners requiring mre technical prcess details shuld refer t the Guide t Develping an Effective STAT Test Strategy available at under STAT INFORMATION. Questins t Ask The questins that fllw are brken up by phase (refer t Figure 1) t make it easier t understand why they are being asked. Sub-bullets belw each questin utline acceptable respnse cntent. The phases shuld be addressed in rder because effective planning at ne stage sets up the next stage fr Page 2

4 gd results. Nte that there are sme questins that pertain t planning befre any testing is cnducted and sme that are relevant nly after data has been cllected. These pre- and pst-testing questins are identified. This is nt an exhaustive list but a reference tl t assist in meeting preparatin. Mre detailed technical questins and prcess details can be fund in the Guide t Develping an Effective STAT Test Strategy. Plan STAT in Test & Evaluatin Understand Requirements Screening Characterizatin Requirement(s) ID/Clarify/Quantify STAT Candidates Decmpse Missin/System Determine Objective(s) Verificatin Optimizatin Prgram Level Basic Test Prcess STAT Infusin Define Design Space Design Factrs Recrded Cnditins Nise Select Factr(s) / Level(s) Define Respnse(s) Identify Cnstraint(s) Randmizatin Blcking Disallwed Cmbinatins Design Cnfidence & Pwer Signal & Nise Cverage Replicatin Create Test Matrix Orthgnality Mdel Specificatin Optimality Criteria Execute Analyze Run Card Generatin Experimental Prtcl/Training Recrd Deviatins Analysis f (C)Variance Respnse Surface Methdlgy Regressin Mdeling Multivariate Analysis Execute Test Reprt Decisin Quality Inf Distributin Fitting Hypthesis Testing Mnte Carl Simulatin Optimizatin Prgram Decisin(s) Emply a STAT Wrking Grup fr Best Results Figure 1: STAT Prcess fr Test Planning STAT Differences in Develpmental (DT) and Operatinal Testing (OT) The applicatin f STAT will differ in DT and OT fr varius reasns. In early DT the subsystems may be tested individually befre executing fully integrated systems testing and the test designs will reflect this. Later in DT the full-up system testing may resemble mre peratinally realistic cnditins as shuld the test design. In general, OT is the mst difficult realm t apply STAT due t the ptential lack f cntrl f many factrs. Despite these differences, the STAT planning principles apply in all test phases: determine apprpriate bjectives, define the respnse and factrs, create a design fr the required data, execute the testing in the prper manner, and analyze the results t address the requirement. Mrever, a well integrated test team (ITT) will ensure that early testing will supprt later testing. Test Page 3

5 planners shuld always strive t cllect data that relates t peratinal gals s results can be rigrusly cmpared, risk can be assessed, and the test readiness determinatin is well infrmed. Strategy/Planning What STAT resurces have yu cnsulted t plan yur strategy? Leverage supprt frm the STAT COE. Utilize lcal STAT/DOE Subject Matter Expert (SME). STAT/DOE class/trainer cnsultatin r ntes. STAT/DOE knwledge base/website. Hw has STAT planning been incrprated int yur test team? STAT is a cmpnent f the Integrated Test Team and Test Integrated Prductin Team. The STAT sectin was develped thrugh an integrated prcess, nt a separate wrking grup. Bth DT & OT rganizatins address STAT cnsideratins and the ITT apprves the apprach and dcuments this in the TEMP test strategy. TEMP STAT plans are translated int the system level test planning activities by the respnsible test rganizatin. Are the test bjectives clearly articulated? Nte: Clear bjectives are fundamental t the planning prcess and facilitate gd design. Objectives shuld be specific and define desired utcmes. Example bjectives: Screening: a lw reslutin design fcused n identifying main effects Characterizatin: a high reslutin design with lw predictin variance Sftware factr cverage: a cmbinatrial design cvering all triplets What is yur strategy t ensure early testing infrms later testing? Operatinal testers are part f the planning team frm the beginning. Fllw-n testing will emply the same designs and/r respnses and be used t validate previus findings. The empirical predictive mdels will allw later testing t validate earlier results. Earlier testing with simulatins, labs, and hardware in the lp (HWIL) will cver and cntrl factrs unavailable in later real-wrld testing. A data sharing prcess flw map shws what data is being cllected thrughut testing and hw it is being used. What prcess was used t determine which test requirements need STAT fr assessment? Wrking grup meetings emply SMEs frm all relevant disciplines. All test requirements are reviewed by the grup and STAT requirements are identified based n the test bjective at each particular phase. Hw are cnstraints (inability t randmize, lack f factr cntrl, resurce r range limitatins, system limitatins, etc) limiting ur ability t gather a statistically relevant size data set? Page 4

6 Early testing (lab, HWIL, simulatin) will cver the full perating regin and factrs. Present a cmparisn f the cst ($ and number f pints) required fr a relevant data set against the prpsed plan. Can the test labs/ranges facilitate the fundamentals f STAT/DOE (randmizatin, replicatin, and blcking)? Ranges allw randmizatin withut negatively impacting resurces r time t test. Range persnnel understand DOE cncepts including why pints are tested in a particular rder. Data cllectin, reductin, and analysis can be dne in a timely manner t facilitate screening designs and sequential testing. Fr the nn-stat designs what determined the test pint selectin? Nte: Previus designs are nt necessarily a gd reference (What has changed? Was it a rbust design?). Accepted standards are referenced (e.g. Industry, Mil-Spec, SysCm prcedures). Hw was reliability test time determined? Statistical sftware shuld be used; `rules f thumb shuld be avided (e.g., threetimes rule is insufficient). Operatinal characteristic (OC) curves shuld shw a lw prbability f acceptance (Pa) at the metric (e.g. MTBF) threshld value and high Pa at the bjective value. Reliability grwth curve starting pint shuld be based n data frm a similar system. Reliability will nt grw unless imprvement insertin pints are planned during develpment. Sftware tl assumptins shuld be realistic (e.g. fix effectiveness shuld nt be >70%). Has the system (r functins) been decmpsed t facilitate lwer level designs? Nte: Single designs that cver the entire missin are typically t brad. Many smaller designs (instead f ne large design) can mre effectively address requirements thrughut a missin. Designs shuld fcus n smaller cmpnents r functins. The designs are usually the right size when the respnses and factrs are clear, preferably cntinuus, and peratinally meaningful. Design Develpment Will the designs adequately explre the expected regin f perability? If nt, what regins will we nt be able t assess perfrmance? M&S shuld cver the entire regin f perability, bundary t bundary. Test range cnstraints culd limit certain perating regins. The prgram needs t identify, quantify, and mitigate the risk f nt examining these scenaris. Did yu minimize the use f categrical factrs and respnses? NOTE: The data type chsen t represent factrs and respnses can have a majr effect n the resurces needed t cnduct an experiment and the quality f its respective Page 5

7 analysis. Cntinuus data types (e.g. mile per hur vs. slw/fast) are preferable when pssible. The reasn fr any categrical data types left in the design shuld be discussed. Respnses must be bjective and measureable and nt require a value judgment like satisfactin r success. Hw is test risk quantified? Lt sampling plans: Number f trials and acceptable failures can be pltted as peratinal characteristic (OC) curves. The accepted curve shuld pass thrugh a minimum 80% prbability f acceptance at bjective and 20% (r less) prbability f acceptance at threshld. Characterizatin testing: Minimum 80% pwer and cnfidence gals shuld be used t size the design f experiments. Simulatins: Space filling designs shuld reprt hw the space was cvered (design type and density). Cmbinatrial (sftware) testing: Designs shuld have at least 2-way (all pairs) cverage. Three way (triplet) r higher cverage will reduce risk further. Hw wuld the design need t change t reduce this risk? Nte: This risk may be driven by test cnstraints (e.g. range limitatins). Prvide a cmparisn f the existing design with a perfect design and analyze the difference in the number f pints (and assciated cst). Fr reliability: estimate the additinal perating time (after testing cmpletes) needed t generate a sufficiently accurate estimate. Pre-Executin Planning What STAT-related training has the test team received? All testers shuld receive STAT familiarizatin training. This can be prvided by the test designers r STAT subject matter experts. This prvides awareness as t why events shuld be run in the planned (randmized) rder. Testers shuld be aware that any deviatins frm the test prtcl must be recrded. Testers must understand that the design recrded in the TEMP (especially if it is apprved) must nt be changed withut test team cncurrence. Analysis Planning (befre any data is cllected) What analysis d yu have planned and hw will it address the requirement? Traceability frm requirements t design must be clear. Selected analytical methds shuld directly infrm acquisitin decisins. Multiple analytical methds (e.g., simulatin, lgistic regressin, parametric analysis, etc.) may be required t fully address the requirement and quantify perfrmance and risk acrss the space. Analysis f variance and an empirical mdel alne may nt suffice t address the requirement as stated. Hw will the analysis methd help predict perfrmance fr pints nt actually tested? If a predictin is required the design shuld be capable f prducing a predictive mdel with acceptable accuracy acrss the factr space. Page 6

8 Use f cntinuus factrs will enable predictin between tested pints (imprtant in OT when cnditins are less cntrllable). Pst-Executin Review Were there restrictins in running design pints randmly? Hw was it handled? Analysis shuld be based n hw the data was actually cllected, nt the way it shuld have been cllected. Recnfiguring the run rder t be mst efficient may create statistical analysis issues. If this is the case then the actual run rder and the cllected data shuld be thrughly analyzed by a highly cmpetent DOE practitiner t ensure the statistical data validity befre cncluding testing. Analysis Review (after data has been cllected) Can yu describe the analysis methds used t cme t yur cnclusins? The analysis shuld address the requirement. Multiple analytical methds shuld cmplement each ther r act t uncver risk r validate ther cnclusins. Any changes frm the riginal design and analysis plan shuld be detailed as t hw they impacted the assessment and cnclusins. Validatin f the raw data shuld be accmplished whenever pssible. Cnclusin DOD leadership have advcated the use f mre rigrus test methds t ensure risk is identified and quantified as early as pssible as determined by the specific bjective(s) f the test. Test planners shuld make every effrt t apply STAT until a reasn t deviate is justified. STAT must be an integral part f the test planning prcess and the active invlvement f prgram leadership enfrces effective implementatin. Page 7