Settlement System Replacement (SSR) Project Test Process Review. BAWG August 16, 2007

Transcription

1 Settlement System Replacement (SSR) Project Test Process Review BAWG August 16,

2 Overview of SSR Test Process Development Unit Testing Quality Assurance Testing Functional Testing Integration Testing Regression Testing Performance Testing User Acceptance Testing SSR Certification Process 2

3 SSR Development Unit Testing Use Case (UC) development broken out into 8 iterations Consistent Design reviews held for each iteration before beginning development For each iteration, developers tested each individual UC, as well as conducted full iteration testing prior to hand off to QA team Team also conducted full comparisons of the tables by reviewing BAS bills against bills produced using new settlement engine Used automated comparison tools to conduct full analysis on differences found in bills produced within the new engine 3

4 SSR Quality Assurance Testing Functional Testing Use Case testing was conducted by preparing test cases based on calculations in the SSR design documents Automated comparison tools (table compare utility tool) were used to ensure accuracy of all results (BAS tables results compared to same day results for new Settlement System) Integration Testing Rebills run on Settlement System platform, same billing days run in BAS Teams including CS and DSS test team pulled the data for verification (to test both DSS as well as CS Audit modules) Simultaneously all QA teams tested integration between different applications with SSR. i.e. testing from Bid to Bill (Market Run, PTS, DSS, Calculation verification through SAS audits modules testing, and posting through to financials) 4

5 SSR Quality Assurance Testing Continued Regression Testing After each scheduled deployment in a testing environment, QA verified the functionality of all Use Cases, Rebills and Daily Bills to ensure code drops did not introduce other unintended consequences or impact existing functionality Performance Testing Stress and load tested on various existing applications to see the impacts while running bills. Coordination across Product lines and applications have been identified as good target areas for our testing. During this test, the DBA group monitored the databases to ensure there are no issues Tested performance of new engine Average time to run a bill was greatly reduced relative to the current system. (Average for each bill was minutes compared to 120 minutes in the current system) DSS pulls have remained unchanged, Improvements are internal at this point. MPs will not see a gain in the performance 5

6 SSR QA Testing Quick Facts QA Testing and Preparation January 1, 2006 through June 11, 2007 No critical or major defects remain open 6 months of Rebills, including special days identified by QA and Customer Settlements were matched out from BAS to SSR (Special days included DST days, and multiple peak load days) Over two full weeks of daily bills were run and certified complete and accurate by QA and UAT teams 6

7 SSR User Acceptance Testing Selected over 100 rebill days to conduct comparisons against BAS and new settlement engine Using SAS audit modules ran audits against BAS settlement calculations Ran the same audits against the corresponding days in the new settlement engine Performed analysis on each audit module to ensure settlement calculation results matched in both BAS and new settlement system. Validated accuracy of full billing runs from QA integration testing 7

8 SSR Certification Process Mapped the Use Cases, which contain the specifications for the SSR, to the specific sections of the tariffs Validated that the formulations in the SSR, based on the Use Cases, were consistent with PA s formulation interpretations in the tariffs Developed tests to ensure that appropriate logic paths for market rules would be tested Included formulation interpretations based on tariff language Executed tests to ensure/validate SSR application returned results as expected 8

9 SSR Certification Results Identified one tariff compliance issue, which was resolved via a code fix Identified a future compliance issue, in regard to penalizing over generation, that will impact the market in 2010 if not addressed by the NYISO Certification letter, endorsed by PA Consulting, distributed to Market Participants on June 18,