Best in Service Program Overview

Transcription

1 Best in Service Program Overview Effective November 1, 2016

2 Contents Overview What is Best in Service? Program Goals Data Center Considerations Designation Types: Orange and Blue Levels... 4 Evaluation Process Point-of-Entry Criteria... 5 Blue Level... 5 Orange Level Technical Evaluation... 7 Performance Dashboard Quality of Service... 7 Best in Service Trust Site Customer Service Audit... 8 Enrollment and Recertification Process Data Center Mergers, Additions, and Relocations Maintaining your Best in Service Status Status Removal Scoring Appeals Benefits and Marketing Contact Information Appendix A: Quality of Service - Scoring Details Document Relativity Best in Service Program Overview - 2

3 Overview What is Best in Service? The Relativity Best in Service program recognizes Relativity Premium Hosting Partners (PHPs) whose data centers provide an exceptional end user experience. kcura defines a Best in Service partner as one whose data center exemplifies best-in-class Relativity delivery through: Proven customer satisfaction based on positive feedback from customer references. Staying current with Relativity releases, giving customers access to the most feature-rich tools. Extensively monitoring and maintaining their hosting environment, ensuring their system performance scores meet or exceed service delivery standards. Transparency about the quality of their service delivery via the Relativity Best in Service Trust site. Retaining Relativity-certified staff who demonstrate proficiency in Relativity administration and infrastructure management. Program Goals kcura developed the program with these goals in mind: Create a meaningful program that gives partners an opportunity to differentiate themselves among other Relativity providers. Strive to make the Best in Service designation a key criterion for those evaluating litigation service providers offering Relativity. Recognize groups who provide a great experience to our collective customers, independent of the purchasing volume of their Relativity account. Hold our partners accountable to maintaining fast and scalable Relativity hosting environments by measuring their performance with technical audits. Make Best in Service acceptance criteria objective by using a transparent scoring methodology. Motivate partners to deliver the highest standard of support by gauging customer satisfaction. Encourage partners to execute best practices in delivering Relativity, resulting in better customer experience with the Relativity platform. To keep these goals at the forefront of the Best in Service program, kcura continues to improve our evaluation methods, increase our transparency, and add value to the designation. Relativity Best in Service Program Overview - 3

4 Data Center Considerations A data center is defined as a location where Relativity is installed that has its own physical or cloud-based infrastructure. A data center may run one or many instances of Relativity. An instance of Relativity is not necessarily confined to a single, physical data center if the instance has file shares, web servers, or agent servers located in remote data centers. You may choose to apply for Best in Service with as many of your data centers as you d like, but only those data centers that successfully complete all phases of the application process can be marketed with the Best in Service designation. Designation Types: Orange and Blue Levels As the volumes of data and number of cases under management grow, we understand that you experience unique challenges and require different types of resources to effectively support your Relativity customers. As such, we've created two tiers for Best in Service, Orange level and Blue level, which recognize PHPs for providing an excellent Relativity experience. The point-of-entry criteria, outlined in Section 2.1, differ between the Orange and Blue levels. Relativity Best in Service Program Overview - 4

5 Evaluation Process Point-of-Entry Criteria All partners who meet the program s point-of-entry criteria are eligible to apply. These criteria vary, depending on the level(s) at which you are applying. The following sections describe the point-of-entry criteria for both the Blue and Orange levels. Please note that a single partner may certify any combination of Blue- and Orange-level data centers. Blue Level To apply for the Blue level of Best in Service, you must: Complete one year as a Relativity Premium Hosting Partner. Meet the following requirements on all instances: o Run a supported version of Relativity (standard support is provided for 15 months following a release). o Have the most recent version of Relativity Performance Dashboard (PDB) installed for at least 90 days prior to applying. Additionally, we recommend using Dr. Feelgood to proactively monitor your environment for at least 90 days prior to applying for each instance that will be included in your application. o Achieve a minimum average score of 80 percent across the PDB Quality of Service (QoS) scores, as shown in Section To ensure your environment can support the latest versions of Relativity and PDB, please keep any third party software up-to-date with recommended software and firmware updates. Submit a disaster recovery plan that provides access to a Relativity environment, housed in an alternative data center location, within 48 hours of a disaster occurring, with no more than 24 hours of data loss for active cases. The Recovery Point Objective (RPO) and Recovery Time Objective (RTO) should be as follows: o RPO: 24 hours o RTO: 48 hours For more information on disaster recovery planning, please see our Relativity Blog or our Backup and Data Management Best Practices guide. Submit the name and contact information for one Relativity Certified Administrator (RCA) per data center. You must maintain one RCA per data center on staff throughout your certification. Relativity Best in Service Program Overview - 5

6 Submit the name and contact information for one certified Relativity Infrastructure Specialist per data center, who can submit requests pertaining to QoS scoring appeals. You must maintain one Infrastructure Specialist per data center on staff throughout your certification. Please see Section for further considerations about this requirement. Maintain a user count of more than 100 unique named users per data center, measured as an average count of active users in the three months preceding application. Please note that any single user may only be allocated to one data center for application purposes. Maintain reasonable financial standing with kcura. Provide contact information for seven to 10 customer references for each Relativity instance, per data center, all from unique accounts (i.e. you cannot use two references from the same organization). See Section 2.3 for more details on references. Sign a Memorandum of Understanding (MOU). Submit the name and contact information for one business representative from your organization, who will be kcura s point of contact for the point-of-entry criteria check, MOU, references, and customer service audit. Submit the name and contact information for one technical representative from your organization, who will be kcura s technical point of contact for point-of-entry exemption, and Best in Service program notifications. Orange Level To apply for the Orange level of Best in Service, you must meet all of the Blue-level criteria outlined above, in addition to meeting the following requirements: Maintain a user count of more than 1,000 unique named users per data center, measured as an average count of active users in the three months preceding your application. Please note that any single user may only be allocated to one data center for application purposes. Maintain one RCA on staff per 1,000 users per data center (we will not require more than five RCAs per data center.). NOTE: Data centers that are certified at the Blue level may become eligible to change to Orange level certification if found to meet the requirements. If this occurs, your data center will be considered Orange level for the remainder of your Best in Service certification year. Relativity Best in Service Program Overview - 6

7 Infrastructure Specialist Considerations If you plan to certify multiple data centers, you may assign one unique named Relativity certified infrastructure specialist to multiple data centers. A single infrastructure specialist may be assigned to a maximum of: Two Blue-level data centers or One Orange-level data center and one Blue-level data center Technical Evaluation Successful delivery of Relativity requires the right hardware, a proficient technical staff to support that hardware, and implementation of kcura s best practices for optimizing a Relativity environment. As indicated in the point-of-entry criteria, you must meet system performance thresholds that will be measured by PDB. These technical requirements are the same for both Orange and Blue levels. The Best in Service technical evaluation relies on the Quality of Service system, conducted through PDB. This scoring system provides an objective and complete analysis of the environment performance, as well as an automated reporting system, eliminating the need for partners to manually send the reporting data. Performance Dashboard Quality of Service Your technical evaluation score will be calculated by the Quality of Service tab in PDB. PDB collects data over time, and you must have at least three months of usage data available in the application before applying for Best in Service. To earn a passing status on your technical evaluation, you must maintain a minimum average QoS score of 80 percent throughout the three months prior to your application. This methodology evaluates four categories: 1. User Experience 2. Infrastructure Performance 3. Recoverability and Integrity 4. Uptime These four areas are scored in a transparent method, allowing participants to gain a complete understanding of what the different scores mean and how to use them to improve service delivery. For a thorough explanation of how this score is calculated, please refer to the Quality of Service Scoring Details document (Attachment A). Best in Service Trust Site The Relativity Best in Service Trust site (Trust site) is a public website that provides visibility into Relativity performance at the data center level. On the site, viewers can find weekly performance data from Best in Service partners. Upon certifying for Best in Service, your weekly quality of service score status will be displayed next to your data center listing on the site. Relativity Best in Service Program Overview - 7

8 Your data center s weekly QoS score reflected on the Trust site is calculated by averaging the individual scores of each active instance within it. These individual instance scores are weighted based upon the number of active users associated with the instance. If scores for individual instances are not received: The Trust site will display data pending if the missing data accounts for at least 50 percent of the total active users for the data center. The Trust site will display a score calculated using the received data if the missing data accounts for less than 50 percent of the total active users for the data center. In both cases, the score will be recalculated and updated once all missing data has been received. You may store audit data in data grid provided: At least 14 days of audit data must be preserved in SQL across the board Scores must be sent to the Trust site on a weekly basis Weekly backups must be maintained for the EDDSPerformance and EDDSQoS databases You ll need to have at least 90 days of reporting to the Trust site to have enough data for an evaluation. Customer Service Audit To ensure your clients are receiving the support they need for Relativity, we ve designed a short customer service survey. Please provide seven to 10 client references for each Relativity instance, per data center, all from unique accounts (i.e. you cannot use two references from the same organization). You must adhere to the following guidelines when selecting your references: If you are applying for the first time, please send us seven to 10 references from unique organizations per instance, per data center. If you are applying for recertification, please also send us seven to 10 references from unique organizations per instance, per data center. In this case, you may include references from the organizations used in your previous submission(s) or from new organizations, but only up to four of these references can be specific contacts from your previous submission(s). We will send the survey to all submitted references. Once the surveys have been sent, references will be given two weeks to respond. We will keep the survey process open for the entire two weeks to allow ample time for responses. We must receive at least five survey responses per instance, per data center, within that timeframe, and all survey responses must be positive. If not, you will be required to wait until the next annual enrollment period before reapplying to the program with seven to 10 new references per instance. We encourage you to actively engage your references to guarantee survey responses. One week after sending the surveys, we will let your Best in Service business contact know if you have not yet met the minimum requirement for returned reference surveys. This will give you an opportunity to contact each of your references to remind them to reply to the survey. Because of the confidential nature of the surveys, we will not be able to share the response status for each reference, or whether the survey results Relativity Best in Service Program Overview - 8

9 are positive or negative per individual response. After two weeks, we will review the survey feedback to determine if you have passed this category of the process, and we will notify you with a pass or fail result. Relativity Best in Service Program Overview - 9

10 Enrollment and Recertification Process We are accepting applications from November 1 through January 16. During this enrollment period, we will accept both new applications and recertification applications. To apply or recertify, you must access the Best in Service Online Application in the Customer Portal. Once you ve initiated the process, the online application will automatically check to ensure that your data center meets the point-of-entry criteria and passes the technical evaluation to allow you to proceed with entering your customer references and other contact information. NOTE: Although the online application page automatically audits your records to ensure you ve met all point-of-entry criteria listing in Section 2.1, kcura will manually review and approve your disaster recovery plan. For instructions on how to use the online application, please reference the Best in Service Online Application Guide. Once you ve completed your online application, we ll begin the customer service audit. On average, this process takes at least two weeks to complete. If you have questions or need assistance with using the Best in Service online application, please contact BestinService@kcura.com. NOTE: Partners renewing their Best in Service certification can maintain their status past December 31 during the time the application is being processed. Once your online application is complete, we will reach out to you with an MOU to sign and return. We will then move forward with the customer service evaluation. Best in Service status will be awarded to those data centers that satisfy the point-of-entry criteria, which includes the technical evaluation, and the customer service audit. NOTE: In the event of a data center merger, addition of instances, or relocation, please contact BestinService@kcura.com for information on how this may impact your Best in Service status or weekly QoS scores. Relativity Best in Service Program Overview - 10

11 Maintaining your Best in Service Status Because the Best in Service designation is valid for one year, we expect partners to maintain the high levels of service, support, and accountability required by the program throughout the duration of their certification. Status Removal The Best in Service status can be revoked for any of the following reasons: Failure to maintain all Best in Service point-of-entry criteria throughout your certification period Tampering with quality of service data reporting Failure to continue reporting quality of service data throughout your certification year (see Section 2.2.1) Failure to publish your weekly performance data on the Trust site Loss of customer data because of insufficient data backup procedures Failure to fulfill the disaster recovery plan requirements Negative feedback from end users / customers Complaints issued by the government / undergoing government investigation Delinquencies or defaults under the partner s kcura Relativity agreement Your certification and related marketing benefits may be revoked if any of the above occur. However, this list of examples is not exhaustive. Ultimately, removal of a partner s Best in Service designation will be at kcura s sole discretion. You may only reapply to the Best in Service program in the following year if the applicable issue has been resolved to kcura s complete satisfaction and kcura considers a new application in its sole discretion. Failure to adhere to the program guidelines may result in loss of Best in Service status. Scoring Appeals Best in Service status is contingent on maintaining QoS scores that meet the program s standards. If, at any point, you wish to appeal your scores, you may do so by completing the appeal form located in the Customer Portal. NOTE: Scoring appeal forms can only be submitted by certified Infrastructure Specialists. Relativity Best in Service Program Overview - 11

12 Benefits and Marketing The Best in Service designation is valid for one year, during which you will receive the following benefits, per certified data center: Priority listing on the Relativity Premium Hosting Partners (PHP) webpage Recognition on the Trust site Use of the Best in Service logo, which includes a digital badge that can be applied to your website, signatures, and other marketing collateral. Please refer to our Relativity Trademark and Branding Guidelines when using the logo. Support with writing a press release announcing your Best in Service designation, including: o A press release template o A quote from a kcura representative o A link to your press release on the: kcura website kcura RSS feed kcura LinkedIn page kcura Twitter page On a company level, you will also receive the following benefits: Access to multi-tenancy licensing NOTE: If a PHP with a Best in Service data center wishes to obtain a tenant license for a non-certified data center, the non-certified data center must maintain the system performance standards required of certified data centers by deploying Performance Dashboard and enabling an agent to release weekly scores. Five free burst Relativity seats per month $4,000 Relativity marketing development funds (in addition to the annual $2,000 stipend provided to all partners) for marketing activities that promote your Relativity offering. Please contact marketing@kcura.com for more information on marketing development funds and marketing support. Contact Information We're proud to have you as a partner and hope to have the opportunity to recognize your achievements through the Best in Service program. If you have any questions about the program, please contact BestinService@kcura.com. Relativity Best in Service Program Overview - 12

13 Appendix A: Quality of Service Scoring Details Document

14

15 Quality of Service Scoring Details July 18, 2016 Relativity Quality of Service Scoring Details - 1

16 Contents Purpose...3 Overview...3 The Sample...3 User Experience...4 Measurements...4 Search Analysis...4 User Experience Sample...4 Infrastructure Performance...5 Measurements...5 CPU Utilization (Web + SQL Servers)...5 Available Memory (Web + SQL Servers)...5 File Level Latency (SQL Only)...5 SQL Server Memory Signal State & Paging (SQL Only)...6 Virtual Log Files (SQL Only)...6 Waits Score (SQL Only)...6 Scoring...7 Scoring the Web Servers...7 Scoring the SQL Servers...7 Overall Score...7 Recoverability and Integrity...8 Measurements...8 Scoring...8 Database Backups and Database Consistency Checks (DBCCs)...8 Maximum Data Loss...8 Time to Recover...9 Overall Scoring...9 Relativity Quality of Service Scoring Details - 2

17 Purpose Performance Dashboard (PDB) is an environment monitoring and performance assessment tool that provides Relativity infrastructure administrators with feedback and alerts on the performance of their Relativity environments. This document describes the information and algorithms PDB uses to collect, analyze, and score environments. Overview The Quality of Service scoring system is what PDB uses to calculate an overall weekly score. The Quality of Service scoring system measures the following four key aspects of an environment: User experience Infrastructure performance Recoverability and integrity Uptime PDB examines each category individually on a weekly basis and determines an overall weekly score by averaging the score of all four concepts together. All four categories are equally weighted when calculating the average. The overall quarterly score is an average of the previous twelve overall weekly scores. The Sample PDB takes a weekly sample that represents the busiest periods of time (hours) for each workspace across the entire Relativity environment and uses that data to compute the user experience and infrastructure performance scores. This focuses the scoring for these categories on times when an environment is most likely to compromise infrastructure performance and user experience. PDB determines the busiest hours per workspace by user arrival rate. The user arrival rate is the average arrival of queries per second (total requests within an hour/3,600 seconds). PDB bases a second sample extraction from the larger weekly sample based on concurrency (total execution time within an hour (seconds) / 3600 seconds). This concurrency sample is used in the calculation of the user experience score. The scoring process ignores hours when there are no users hence no activity in a workspace. This eliminates maintenance windows from the infrastructure performance scoring process and allows off-hours reviewers and project managers to run operations that otherwise affect the experience of other users in the system. Relativity Quality of Service Scoring Details - 3

18 User Experience The execution time of simple document query audits is used to determine the user experience score. This is because it reflects a user s experience running searches and navigating views within in Relativity. Measurements PDB analyzes user searches and assigns a complexity rating based on the criteria within the search (nesting, use of Is Like, etc.) Searches fall into the following two categories that carry different scoring weights: 1. Simple Query 2. Complex query If a search s execution time is greater than 2 seconds, it is classified as a Long Running Query (LRQ). Search Analysis The user experience category is an average of the arrival rate sample and the concurrency sample. PDB averages the scores generated for each of these samples to form the overall weekly score for the user experience category. User Experience Sample Each user receives a score based on the percentage of simple document searches that are not longrunning. Long-running searches between 2-5 seconds count against the score as though they were run twice, and searches exceeding 5 seconds count as though they were run three times. The server s score for the hour is an average of the user scores. User experience is the average of the following two component scores: Arrival rate sample hours (top 20% by arrival rate) - the average hourly score in the arrival rate sample hours forms one component of the user experience score. The top 33 hours by arrival rate in the last week are used for the sample. Concurrency sample hours - the average hourly score in the concurrency sample hours forms the second component of the user experience score. The top 33 hours by execution time in the last week that are also in the arrival rate sample are used for the concurrency sample. Relativity Quality of Service Scoring Details - 4

19 Infrastructure Performance Infrastructure performance scoring reflects the underlying infrastructure s ability to handle the volume of activity. This category is an analysis of the performance of the web and SQL servers. Measurements PDB analyzes the following items to determine an infrastructure performance score: CPU Utilization (Web + SQL Servers) This metric analyses the CPU utilization on the SQL Servers and the Web Servers. PDB collects this information every five minutes and averages it every hour. Scoring Details If CPU utilization is less than or equal to 60 percent, the score is 100. If CPU utilization greater than 85 percent, the score is zero. If CPU utilization is between 60 and 85 percent, a linear scale determines the score. Available Memory (Web + SQL Servers) Scoring Details Web Servers o If the available memory is greater than or equal to 1GB, the score is 100. o If the available memory is less than 1GB, a logarithmic scale determines the score. SQL Servers o If the available memory is greater than or equal to 4GB, the score is 100. o If the available memory is less than 4GB, a logarithmic scale determines the score. File Level Latency (SQL Only) PDB checks the read and write latency on database each hour, using SQL Server s virtual file stats function (sys.dm_io_virtual_file_stats). If a SQL Server has a high amount of RAM, latency on reads becomes less of an issue, hence the file level latency is only scored when PAGEIOLATCH waits are the highest wait type. Scoring Details File latencies are checked against the following thresholds: Data file reads (100ms) Data file writes (30ms) Log file writes (10ms) Relativity Quality of Service Scoring Details - 5

20 If file latency exceeds the threshold and PAGEIOLATCH waits are the highest wait type, the score is zero; otherwise, the score is 100. SQL Server Memory Signal State & Paging (SQL Only) SQL tracks the allocated memory state that indicates high or low memory. If the low memory state is flagged, SQL server attempts to trim in process memory. Performance suffers if large sections of in process memory are paged out. Scoring Details If the SQL low memory state occurrence is greater than 80 percent, the score is zero. SQL low memory state occurrence is greater than 0 percent and page outs occurred, the score is zero. PDB determines the score with a linear scale from 0 to 80 percent low memory occurrences. Virtual Log Files (SQL Only) Too many virtual log files (VLFs) can cause long startup and backup times. Depending on the database size, more than 1,000 VLFs indicate a problem. SQL Server issues a warning if there are more than 10,000 VLFs. Scoring Details If the number of VLFs is less than or equal to 10,000, the score is 100; otherwise the score is zero. Waits Score (SQL Only) A high signal-to-resource wait ratio indicates CPU pressure. Threads spend an excessive amount of time on the runnable queue as opposed to actually running. Poison waits are detrimental to the environment and should not appear during hours of user activity. If any poison wait type exceeds one second for the hour, the hourly system load score is zero. Scoring Details If the total waits for the hour exceed 75% of the available processor time for the hour, the signal waits ratio is used to determine the score. Otherwise (in the absence of poison waits), the score is 100. For example, if there are eight logical processors on the server, total wait time must exceed six hours before the signal waits ratio is used. SOS_SCHEDULER_YIELD waits are excluded from the signal waits ratio if their average wait time for the hour is less than 1ms. If the signal waits ratio is less than or equal to 10 percent and there are no poison waits, the score is 100. If the signal waits ratio is greater than or equal to 25%, the score is zero. Relativity Quality of Service Scoring Details - 6

21 If the signal waits ratio is between 10 and 25 percent, a linear scale is used to calculate the score. If poison waits exceed one second for the hour, the score is zero. Scoring PDB uses the following methods to determine a final infrastructure performance score. Scoring the Web Servers Each web server s score is the average of the scores of the following scores: CPU Utilization Available Memory PDB aggregates all web server data (across sample hours) to generate one score. Scoring the SQL Servers Each SQL server s score uses the following weighted average of the scores: CPU Utilization 22.5% Available Memory 22.5% File Level Latency 5% SQL Server Memory Signal State & Paging 22.5% Virtual Log Files 5% Waits Score 22.5% PDB scores each SQL server separately. Overall Score The overall score for Infrastructure Performance is taken as the lowest score of all the SQL and web servers. Relativity Quality of Service Scoring Details - 7

22 Recoverability and Integrity Relativity workspace databases store all of the underlying metadata within Relativity. The databases include the entire work product and the reviewers coding decisions. Maintaining these critical databases is important to the integrity of the entire Relativity environment. This category analyses the frequency and coverage of database backups and integrity checks (DBCCs). PDB bases the analysis on the length of time it takes to recover a database from a backup and the maximum potential data loss. Measurements PDB uses the following four measurements to determine a score. Database backups (frequency and coverage) Database consistency checks (DBCCs) (frequency and coverage) Maximum data loss Time to recover Scoring Database Backups and Database Consistency Checks (DBCCs) PDB calculates a database backup score according to frequency and coverage. Frequency Frequency represents the largest gap size that exceeds the allotted 10 day window. PDB deducts five points per day until the maximum gap size is reached (after 30 days). Coverage Coverage represents the percentage of databases in violation of the allotted 10 day window. PDB deducts points after 0.75% of databases exceed allotted 10 day window. If 10% or more of databases are in violation, the score is zero. Maximum Data Loss This metric measures the maximum amount of data lost if corruption or accidental user deletion occurs. This is measured as the time between backups. Fifteen (15) minutes between backups is a score of 100. Four hours between backups is a score of 80. Greater than four hours, uses a linear scale down to 24 hours of lost time, where 24 hours between backups is a score of 0. Relativity Quality of Service Scoring Details - 8

23 Time to Recover This metric measures the length of time it would take to restore the most recent full backup if corruption occurs within the database. It accounts for the most recent differential and any subsequent transaction log backups. If recovery time is four hours, the score is 100. If recovery time is one day, the score is 80. Between 24 to 48 hours uses a linear scale to calculate the score, where 48 hours is a score of 0. Overall Scoring PDB determines the overall recoverability and integrity score by averaging the scores of the above four categories. Relativity Quality of Service Scoring Details - 9