IBM Traveler Performance via Console Monitoring

Transcription

1 IBM Traveler Performance via Console Monitoring OPEN MIC WEBCAST DECEMBER 15, 2015 QUERVIN LLOYD L BUCO L2 SOFTWARE ENGINEER IBM TRAVELER

2 Agenda Introduction Improvements Tell Traveler DBMaint APARs Monitoring Console Commands Tools Troubleshooting Status/Error Messages Logs Tips and Best Practices Q&A IBM Corporation

3 Introduction Planning is critical Tuning depending on demands. Being proactive versus reactive Keeping the server updated with the latest server releases. Different components affect the overall performance of an IBM Traveler Deployment. Knowing which logging to turn on, and gathering logs during the issue, not after it has occurred. IBM Corporation

4 Improvements DBMAINT COMMAND APAR FIXES IBM Corporation

5 Tell Traveler DBMaint <command> Introduced with IBM Traveler A new way to conveniently perform regular database maintenance on the IBM Traveler database. will perform an online reorganization of the IBM Traveler table s indexes, as well as run statistics on all indexes in the Database. Operation is online and should not affect the IBM Traveler service. Recommended to be performed during non-peak hours. Mainly used for enterprise database. When run on a standalone (Derby) deployment, sets NTS_DEFRAG_ONCE to 1, and the Derby database will be defragmented on the next restart of the IBM Traveler server. IBM Corporation

6 Tell Traveler DBMaint <command> Configuring automatic database maintenance Time and Interval subcommands must be set. Tell Traveler DbMaint Set Auto On to turn on automatic database maintenance schedule IBM Corporation

7 Tell Traveler DBMaint <command> Know when your database is fragmented Tell Traveler DBMaint Fragmentation Setting Fragmentation Thresholds Set Indexes sets the number of indexes needed to be fragmented before considered as above fragmentation threshold Set Percent defines a percentage of fragmentation in which an index must be above to be considered fragmented (applies to Microsoft SQL) Set Functions defines the number of functions (out of 4) needed to consider an index fragmented (for DB2) IBM Corporation

8 Tell Traveler DBMaint <command> Know when the next maintenance is scheduled and other information about the database fragmentation Tell Traveler DBMaint Show IBM Corporation

9 APARs LO83667: ERRORS SEEN ON TRAVELER STARTUP AT THE END OF HA MIGRATION LO84596: SERVER PERFORMANCE DEGRADATION WHEN REPEATED HTTP getstatus REQUESTS ARE MADE LO84947: TRAVELER DEFRAG ON DERBY RUNS WITH ERRORS LO84846: USE OF INDEX HINT MAY REDUCE PERFORMANCE OF DATABASE QUERY LO85260: WHEN FIRST ENABLING TRASH SYNC, START WITH TODAY AND FORWARD FOR ENHANCED PERFORMANCE LO85582: ONLY SET TRANSACTION ISOLATION LEVEL IF NOT ALREADY DESIRED VALUE IBM Corporation

10 APARs LO86733: LOCK TIME OUT IN SELECTMASTERBINDINGFORUSER.GETONEROW COULD CAUSE TRANSACTION LOG GROWTH. LO86496: SERVER CRASH ON BUFFER OVER RUN ON LONG LOG LINES LO86341: ADDED A NEW DATABASE INDEX WHICH WILL IMPROVE PERFORMANCE LO83617: NATIVE MEMORY LEAK OVER TIME WHEN S ARE READ ON THE DEVICE. LO85584: EXPLICIT COMMIT FOR SELECT STATEMENTS MAY NOT BE NEEDED LO86445: TRAVELER CRASHES FROM STREAMING ATTACHMENTS IN VERY SMALL CHUNKS For the list of IBM Traveler fixes by release (Starting Traveler ) including release date and related documentation. Refer to this link: IBM Corporation

11 Monitoring CONSOLE COMMANDS TOOLS IBM Corporation

12 Things to Monitor On the Traveler Server: CPU Utilization Memory Usage Disk I/O utilization of volume where Domino Data directory resides. Disk Capacity of volume where Domino Data directory resides On Database Server: CPU Utilization Memory Utilization Disk I/O utilization and capacity of volume where database resides. Disk I/O utilization and capacity of volume where transaction log(s) reside. IBM Corporation

13 Console Commands tell traveler status Provides the general status of the server. GREEN generally no issue YELLOW denotes of an issue (warning) RED there is an issue (critical) IBM Corporation

14 Console Commands tell traveler mem Outputs a snapshot of the memory and CPU usages over the last 24 hrs in 15 minute intervals. IBM Corporation

15 Console Commands tell traveler threads Displays the server thread pools, and thread information/statistics IBM Corporation

16 Console Commands tell traveler stat show Dumps all of the IBM Traveler statistics to the Domino console. Statistics from the time the IBM Traveler server was started up to the current up time of the server. Cleared every restart of Traveler or via tell traveler stat clear IBM Corporation

17 Console Commands Caution: there are some stats that are cumulative and only set at startup, and won t get rebuilt if cleared while the server is running. For example, the number of users or devices (Push.Devices.* OR Push.Users.*) IBM Corporation

18 Console Commands tell traveler stat show Show specific IBM Traveler statistics via tell traveler stat show <pattern> Tell Traveler Stat Show DCA.* IBM Corporation

19 Console Commands The following IBM Traveler statistics can be checked to determine if Traveler is performing okay or not: Stat NameLookup.Time.Histogram.<returncode>.<bucket> Monitor.LoopTime.<server name>.<bucket> DCA.DB_OPEN.Time.Histogram.<server>.<bucket> Database.Query.Histogram.GudSelect.<bucket> PrimeSync.Time.Histogram.<SyncReturnCode>.<bucket> Errors.<UserName> Description Histogram of the time (in seconds) to complete the name lookup request. The time between successive checks for changes to a users database. Histogram of the time spent (in seconds) to open a database on the given server using the Domino Java API call. Server is the name of the Domino server on which the database was opened. Histogram of the time spent (in seconds) to execute the specified query against Traveler's internal database. Histogram of the time spent (in seconds) to complete the prime sync with the given sync return code Number of errors logged for the specified user. IBM Corporation

20 Console Commands tell traveler dbusage Displays the number of records being tracked across all IBM Traveler users. Displays the top 10 consumers of mail, calendar, contacts, todos and folder on the server or server pool, along with the selected PIM filters. IBM Corporation

21 Console Commands tell traveler dbusage * To get a list of all users on the system ordered by most to least usage. tell traveler systemdump dbusage Outputs the dbusage information into a systemdump log file. NOTE: This command performs some heavy DB queries, so it shouldn t be done often without proper planning. IBM Corporation

22 Console commands Other console commands User commands Tell Traveler User <userid> -> provides a complete user check. Tell Traveler Users -> outputs all users registered on the IBM Traveler server Tell Traveler Active -> displays the current users who are syncing data with the server. Show Stat Displays all Domino statistics, including Traveler stats. Tip: You can use the -f option on tell commands to write the output to a file. Sample: tell traveler f test.txt stat show Reference: nds.dita?lang=en IBM Corporation

23 Tools How to schedule regular output of Traveler server statistics and usage data? A Domino program document can be used to establish regular output of Traveler server statistics and usage. TN # Use of 3 rd Party server monitoring tools GSX Monitor - Monitoring Traveler with Watchit - h+lotus+notes+traveler#action=opendocument&res_title=8.8_monitoring_lotus_notes_travel er&content=pdcontent&sa=true IBM Corporation

24 Troubleshooting STATU S/ERROR M ES SAGES SCENARIOS LOGGING IBM Corporation

25 How to Troubleshoot Performance issues? Execute a tell traveler status command to show the current status of the server, and check for Yellow or Red status messages. Based on the status messages, perform necessary corrective actions. Also check for error messages in the server console. Important: Before restarting the Traveler service, make sure to get a systemdump (tell traveler systemdump). IBM Corporation

26 Troubleshooting High CPU Symptoms: Status Message: The IBM Traveler's CPU usage is {current percentage} percent over the last {minutes} minutes of processing. Slow sync between the server and devices Common Causes: CPU cores are not enough HTTP threads are not set correctly (insufficient or over-allocation) Database is fragmented Slow performing disk Corrective Actions: Upgrade to the latest version of IBM Traveler to obtain all relevant APAR fixes. Add more CPU cores when necessary Tune HTTP threads to match current load of devices connected to the server. Defragment the database for improved performance. Check for hard disk latency/issues. IBM Corporation

27 Troubleshooting Memory Issues Native Memory Status Message: The current native memory usage is {current percentage} percent of the available memory. Execute a tell traveler mem command to show the history of memory usage. Probable causes: Over-allocation of HTTP threads Other Domino applications are eating up the native memory. Possible issues in the Domino native API calls causing bottlenecks. Corrective actions: Tune HTTP threads Reduce or remove other Domino applications running alongside the Traveler task in the Domino server. Upgrade to the latest IBM Traveler version to get all relevant APAR fixes. Apply latest Domino Fix packs to get memory related fixes. IBM Corporation

28 Troubleshooting Memory Issues Java Memory Status Message: The current Java memory usage is {current percentage} percent of the available memory. Execute a tell traveler mem command to show the history of memory usage. Probable causes: Not enough Java heap memory for the number of users on the system Corrective actions: Increase the Maximum Memory Size in the Domino server document under the IBM Traveler tab. IBM Corporation

29 Troubleshooting Memory Issues Error Messages Traveler: SEVERE userid[deviceid] Exception Thrown: java.lang.outofmemoryerror: Java heap space Check the Maximum Memory Size set from the server document. If insufficient for number of connected users/devices in server, may cause the server to run out of java heap memory. Increase this setting when necessary. For a 32bit server, the Maximum Memory Size cannot exceed 512MB 64bit server, can be increased up to 4GB, where 16GB of physical memory has been deployed. IBM Corporation

30 Troubleshooting Memory Issues Error Messages HTTP JVM: CLFAD0317E: The XPages runtime engine faced an OutOfMemoryError HTTP JVM: CLFAD0316E: You can fix this by increasing the value of the HTTPJVMMaxHeapSize variable in notes.ini HTTP JVM: CLFAD0315E: Out of memory exception occurred servicing request for: /traveler/microsoft-server- ActiveSync - HTTP Code: 500. For more detailed information, please consult error-log-0.xml A heapdump file will be generated which is essential to determine where the memory is consumed and being held resulting in the OOM error. Increasing the notes.ini parameter HTTPJVMMaxHeapSize is advisable, if there are devices expected to sync data (especially with attachments) at any given time, which may consume all the available memory. Upgrade to the latest Traveler release to obtain relevant fixes related to Memory issues. IBM Corporation

31 Troubleshooting too busy status Status Message: There have been {number of 503 RC} device sync failures because the server is too busy and returned status code Error usually means the server is over capacity. Either increase the memory or move users to another IBM Traveler server. Also check for network latency between Traveler and the mail servers. Status Message: There have been {number of error code other than 503 RC} device sync failures for reasons other than the server is too busy. There are factors causing the issue that is outside of Traveler. Most of the time, this is an indication of a network connectivity issue between IBM Traveler, and the users devices. IBM Corporation

32 Troubleshooting too busy status We can also check the server status by issuing tell traveler status, and from there we can identify on what is the current status of the server: The response times for opening databases on mail server <ServerName> are above the acceptable threshold. - The status above is a clear indication that the server is taking a long time to reach and open mail database on that mail server over the network. If you have a systemdump you can check this parameter: DCA.DB_OPEN.Time.Histogram.CN, make sure that there are no large number of connections dropping of this value (120-Inf) because it means 120 Inf seconds before Traveler can access that specific mail server. IBM Corporation

33 Network response times between Traveler and Domino How to determine network latency between IBM Traveler and Domino Mail Servers The bandwidth and speed of the network connection between the IBM Traveler server and the backend Domino mail servers impacts the response time of synchronization with mobile devices. It is recommended that back-end mail servers be co-located with the IBM Traveler server or be accessible through a high bandwidth connection Starting with Traveler , additional network response times to the Mail Servers have been added to the SystemDump ########## Mail Servers (Tue Jun 02 09:33:13 EDT 2015) ########## -- MailServers -- Record Count (Tue Jun 02 09:33:13 EDT 2015): 3 ServerName=CN=MailServer01/O=IBM/C=US, Status=49de, pingtime=56878 msec, LoadIndex=100, lastupdated= :31: ServerName=CN=MailServer02/O=IBM/C=US, Status=0, pingtime=55 msec, LoadIndex=19, lastupdated= :31: ServerName=CN=MailServer03/O=IBM/C=US, Status=0, pingtime=0 msec, LoadIndex=0, lastupdated= :32: If the value of pingtime is greater than 50ms, then this is an indication of network latency. Contact your resident Network Administrator to assist in the resolution of the network. Reference: IBM Corporation

34 Troubleshooting Threads Status Message: User {User name} on thread {thread name} has been running for {xx} minutes. For first occurrence, take a 2 or more systemdumps (in 5 or 10 minute intervals) which will include the information about all of the threads in the Traveler service. Use tell traveler systemdump and collect the logs including javacore. If the thread is DS (DeviceSync) check if it shows the UNID of the document and collect a DUMPDOC by running a tell traveler dumpdoc <UNID> <userid> command If the thread is PS (PrimeSync) check for network connectivity issues between the IBM Traveler server, and the mail servers. Ping times greater than 50m/s suggests network latency. To workaround the issue you can issue the command tell traveler stopsync <deviceid> <username> Restart the Traveler service. There is a good chance this will require a complete Domino server restart and you may need to kill the Domino server in order for it to shutdown completely NOTE: After collecting the dumpdoc, kindly contact IBM Support by opening a PMR for us to help in investigating why it is causing the thread to go hung. IBM Corporation

35 Troubleshooting - Threads Status Message: The number of active HTTP connections is {current percentage} percent of the available HTTP threads ({HTTP Threads}). The peak number of HTTP connections is {peak percentage} percent of the available HTTP threads ({HTTP Threads}). Increase the number of HTTP threads if there is enough memory and CPU resources. Number of needed active HTTP threads = 1.2 x Number of registered devices (Push.Devices.Total) Note: BlackBerry 10 devices tend to use more connections than other mobile devices. Therefore, the general rule of multiplying the number of devices by 1.2 may not be sufficient in calculating the number of HTTP threads and may need to be increased to 2 times the number of BlackBerry devices plus 1.2 times the number of non-blackberry devices. BlackBerry and later releases include improvements to the number of connections utilized, which may reduce the number of HTTP threads. Please contact BlackBerry for additional information about connection utilization by specific BlackBerry 10 releases. IBM Corporation

36 Troubleshooting Database Related Derby Error Message: The IDX_TS_RECORD_MAP_5 table or index has a large number of pages allocated that are unfilled. You should perform a defrag of the NTSDB at the next restart of Notes Traveler to reduce database space and improve performance. - execute a tell traveler stat show Database.Query.* command, and look for the Database.Query.Histogram.GudSelect.<bucket> stat. Check if there are stats within the up to 120-Inf buckets, which indicates fragmentation in the Derby database. - In Traveler run a tell traveler dbmaint fragmentation command to also check if your database is fragmented. Resolution: Perform maintenance (defrag) of the Derby database to resolve the errors. IBM Corporation

37 Troubleshooting Database Related Enterprise DB Errors related to the enterprise database (e.g. SQL or DB2) are most of the time indicative of an issue at the backend enterprise database, and the Traveler service is just being affected by those issues. The following slides have some sample scenarios where the server is suffering on poor performance and there are errors being thrown for the enterprise db. Kindly consult your DB administrator for assistance in troubleshooting enterprise database related issues IBM Corporation

38 Troubleshooting Database Related Enterprise DB Sync exception being treated as a backend failure. Message: com.microsoft.sqlserver.jdbc.sqlserverexception: Lock request time out period exceeded. (SQLState: S00056 ErrorCode: 1222) Exception Thrown: com.lotus.sync.db.persistenceexception: com.microsoft.sqlserver.jdbc.sqlserverexception: Lock request time out period exceeded. (SQLState: S00056 ErrorCode: 122 Error code 122 means lock request time out period exceeded, occurs when a query waits longer than the lock timeout setting. The lock timeout setting is the time in millisecond a query waits on a blocked resource and it returns error when the wait time exceeds the lock time out setting. Usually, if this happens to the backend database of the Traveler Server, most of the devices will not be able to synch or the performance of the Traveler Server will also suffer as the Availability Index will probably drop. Troubleshooting: Check for any issues with the Database Servers during the timestamp the errors were reported. IBM Corporation

39 Troubleshooting Database Related Enterprise DB Sync exception being treated as a backend failure. Message: com.microsoft.sqlserver.jdbc.sqlserverexception: The transaction log for database 'Traveler_P2' is full. To find out why space in the log cannot be reused, see the log_reuse_wait_desc column in sys.databases This message explicitly indicates that the Transaction Log for the Database Server is full which impacts the Traveler server, causing syncs to fail, and performance to drop. Resolution: Clear the transaction logs from the DB servers. IBM Corporation

40 Troubleshooting Database Related Technotes: Traveler HA Servers are not starting and cannot connect to the DB2 Enterprise Server - How should the DB2 or SQL Enterprise Database be maintained in a Notes Traveler environment? - Notes traveler can not start with error: The component CAF could not be started. - Notes Traveler Severe Message : Error trying to determine database version. No suitable driver - Traveler status mesage: "The IDX_TS_RECORD_MAP_5 table or index has a large number of pages allocated that are unfilled. - IBM Corporation

41 Troubleshooting VM (Virtual Machine) Run a show stat Platform.LogicalDisk.* on the server console. Check for the Platform.LogicalDisk.x.AssignedName stat indicating the Drive where the Domino Data directory resides. The Platform.LogicalDisk.x.AvgQueueLen.Avg stat should be below 0.2 on the system/disk where the IBM Traveler data directory resides. Sample: Platform.LogicalDisk.x.AssignedName = * Platform.LogicalDisk.2.AvgQueueLen = 2.26 Platform.LogicalDisk.2.AvgQueueLen.Avg = E+12 Platform.LogicalDisk.2.AvgQueueLen.Peak = E E+12, is an indication that the server is running into a limitation of physical resources. The issue can be solved by either allocating dedicated physical resources to the VM or by moving the Traveler server to a physical machine. Reference: IBM Corporation

42 Logging IMPORTANT: Logs needed to be gathered during, and when the issue is occurring. A systemdump is essential for Support to check the status and statistics of the server, during the issue. Supporting logs such as NSDs, heapdumps and memory dumps also need to be taken during the event of the issue. Before restarting the server, make sure that logs are gathered completely. Reference for logging IBM Traveler performance issues: ita?lang=en IBM Corporation

43 Tips Keep the IBM Traveler servers up to date with the latest releases. See Recommended Maintenance for IBM Traveler Server - Monitor memory, CPU, and other hardware resources regularly. Perform regular maintenance of the IBM Traveler database for improved performance. Always refer to the Capacity Planning Guidelines for IBM Traveler to know what factors to consider when deploying a server, as well as to check if your server is hitting its capacity and needs to be tuned. Always clear the temporary files on Windows OS, when Domino server is shut down. You can do this by going to Start > Run > %TEMP%. If you are having too much clutter in TMP folder, then you may face problems in startup. When servers are in High Availability Configuration and you experienced a degraded server performance kindly check the following: - Performance of the Relational Database - There s no antivirus scanning the domino directory IBM Corporation

44 References Featured Documents for IBM Traveler High Availability Performance Best Practices for High Availability - Maintaining the Enterprise Database - base.html?lang=en Database Maintenance for HA Deployments - Performance Benchmark Results for Traveler 9.0 HA - Availability_Configuration_Performance IBM Corporation

45 References Featured Documents for IBM Traveler Performance: Tuning Active HTTP Threads straveler.dita?lang=en Tuning Performance of the Server r.dita?lang=en Maintaining the IBM Traveler Database tabase.html?lang=en Status Commands Capacity Planning Guidelines - IBM Corporation

46 Questions Press *1 on your telephone to ask a question. Visit our Support Technical Exchange page or our Facebook page for details on future events. To help shape the future of IBM software, take this quality survey and share your opinion of IBM software used within your organization: IBM Collaboration Solutions Support page IBM Collaboration Solutions Support IBM Corporation