IBM HPC Cluster Health Check IBM Redbooks Solution Guide

Transcription

1 IBM HPC Cluster Health Check IBM Redbks Slutin Guide The IBM Cluster Health Check (CHC) tlset is an extensible framewrk and cllectin f tls t check and verify the health f an IBM Cluster. The CHC framewrk prvides an envirnment fr integrating and running a subset f individual checks that can be perfrmed n a single nde r a grup f ndes. The tests are categrized int different tl types and grups s that nly checks fr a specific part f the cluster are perfrmed. Because the framewrk and tlset is extensible, users can als create their wn grups and tls fr checking different cmpnents f the cluster. Figure 1 shws the different cluster cmpnents and the verificatin pyramid. Each level f verificatin depends n the crrectness f the underlying stages. The higher in the pyramid yu are befre a prblem is fund, the mre difficult it might be t attribute it t a mre fundatinal prblem. Figure 1. Cluster cmpnents verificatin pyramid IBM HPC Cluster Health Check 1

2 Did yu knw? The IBM Cluster Health Check tlset (CHC) is an extensible framewrk and cllectin f tls that is used t check and verify the health f an IBM Cluster. The main wrapper tl is hcrun, which prvides access t all tls and passes them the envirnment. Currently, the tlset is available n an as-is supprt basis with the intentin f receiving input frm users regarding its usefulness. Althugh many f the tls wrk in bth the x86 and IBM POWER slutins, the initial tlset cncentrates n x86 slutins. There is nthing in the initial framewrk that precludes integrating tls that are geared tward a POWER slutin. Business value Health checking invlves a series f checking and verificatin steps. Basically, yu must have unifrmity amng the different individual cmputers within a cluster system. Usually, thusands f individual cmputers are invlved in ne HPC cluster system. T check them ne by ne manually is nt a gd apprach because manual peratins are needlessly time-cnsuming and errr prne. Thus, sme kind f autmated checking and verificatin facilities are necessary. A single, fully autmated checking and verificatin facility fr an HPC cluster system d nt exist because it is t difficult t develp such a cmprehensive facility. There are t many different cmputer architectures, different types f perating systems, different types f input/utput adapters, and different types f strage systems. T implement such a fully autmated checking and verificatin facility, the verificatin tl itself must be tested and verified in an envirnment that includes varius cmbinatins f all these types f cmpnents. This is a difficult gal t achieve because there are t many cmbinatins. Based n these reasns, the administratrs f an HPC cluster system shuld implement their wn checking and verificatin tls t leverage and adapt existing tls t their unique cluster envirnment. The IBM Cluster Health Check tlset prvides a set f tls that helps check and ensure cnsistency and stability f the cluster. The fcus fr the tls is t help reduce initial envirnment deplyment time, minimize the effrts that are required fr health state checking, and helps decrease perfrmance variatin. Slutin verview There are significant challenges abut hw t btain pertinent health data that is in many layers f clustered systems. Reprting and crrelating critical system data and transfrming it int usable infrmatin is nt trivial. Tday s cluster health check tls mstly test cmpnents and sme pint-t-pint netwrk perfrmance. They cannt cmprehensively mnitr and assess verall system health, truly aggregate perfrmance, and recncile cnflicting usage, user expectatins, and demands. In this first release, the health check tlkit attempts t establish a few reference baselines f perfrmance and verify the prerequisite services at the cmpnent level, which allws prductive testing and detectin f anmalus cnditins and can even extend t peridic cnsistency checking at the cmpnent level when the tests are nt invasive. This apprach is particularly imprtant when yu try t islate the cause f perfrmance prblems as initial deplyment system testing uncvers prblems. These tls d nt intend t devalue benchmarking and perfrmance-based mdeling as an alternative apprach. Sme researchers had success with perfrmance mdeling results that used strict cluster start and acceptance criteria. This perfrmance-mdeling apprach des extend the time that it takes t start a cluster and adds cnsiderable expense. Even thse wh are successful with perfrmance-based mdeling admit that there are significant challenges in setting reasnable perfrmance expectatins fr bradly diversified wrklads n new HPC systems t realistically set accurate expectatins fr perfrmance at all stages f system integratin. IBM HPC Cluster Health Check 2

3 Slutin architecture The CHC framewrk has the fllwing characteristics: Display available tls in an rganized fashin. Organize and display results f the tls with as much r as little infrmatin as the user requests. Cnfiguratin file driven t allw fr custmizatin and extensin. Custmize tl executin rder thrugh the usage f cnfiguratin files. Extensible. Yu can add yur wn tls and make them available under CHC by updating the cnfiguratin files and using key envirnment variables frm CHC. The CHC tls cver the fllwing brad areas: Nde health and cnfiguratin cnsistency, and test tls. InfiniBand fabric health and cnfiguratin cnsistency, and test tls t verify health f the fabric. Utilities. Basic cnfiguratin checking fr cnsistency acrss a grup f ndes and devices, as well as against a baseline. Usage scenaris Frm build-up t final decmmissining, a cluster passes thrugh different phases during its lifetime in iterative cycles. Updates, changes, maintenance, r utages require methds t run thrugh the stages cnsistently. Each phase has its wn prcesses, characteristics, and needs. The gal f the methdlgy is t reduce nnprductive time t a minimum. The lifecycle starts with the deplyment stage, where all the different cmpnents are assembled, cnnected, pwered, installed, and cnfigured. While the cluster grws, it must be carefully and cntinuusly examined. Basic functins are verified and unifrmity between similar devices must be ensured. Discrepancies and abnrmalities must be analyzed and eliminated r they increase with the size f a cluster and ultimately result in negative impacts t the ability t meet expectatins. IBM prvides preferred practices and recmmendatins t help prevent yu frm running int knwn prblems and traps. Mre cmpnents are cllated and tested until the whle cluster is ready fr verificatin. During that phase, a cluster must prve that it is able t perfrm estimated wrklads and meet expectatins. This phase is when yu attest functinality and perfrmance thrugh extensive testing and evaluatin. Verificatin starts with the smallest entity in the cluster and then scales t the whle cluster size. Fr example, it is a preferred practice t prve the perfrmance f each single cmpute nde befre running a test fr the whle cluster. Reducing the size and cmplexity t a minimum is ne f the basic rules abut health. If a test fails, the surce f the failure must return t the deplyment stage until it can meet the requirements. Finally, after all inspectins, a cluster reaches the prductin phase and must carry ut its duty: Run user jbs and return results. T keep the cluster at that state, it is mandatry t mnitr all key characteristics cntinuusly. Failures r discrepancies frm standards must be quickly identified, the cause investigated, and the surce islated. If it is nt pssible t separate the cause f a prblem, then the cluster n the whle must be verified again, and if a system-wide change is required t slve a prblem, then the lifecycle starts again at the beginning f the deplyment phase. Insufficient results in ne f the stages causes a start frm the beginning at the previus phase. IBM HPC Cluster Health Check 3

4 The lifecycle that is shwn in Figure 2 des nt describe nly the phases f the cluster; it als applies t each single cmpnent. There is n difference between a large cluster r a single device when it cmes t changes. S, even if a single cable must be replaced, the ld ne must be islated frm the remaining cluster. Afterward, a new cable must be deplyed accrding t preferred practices and carefully verified befre yu resume prductin. Figure 2. Health lifecycle methdlgy It is an HPC rule f thumb that the slwest device determines the verall perfrmance f the cluster. Therefre, extensive verificatin and cntinuus mnitring is crucial t btaining and maintaining a healthy cluster. After a maintenance actin, there is n excuse fr verlking ne prly perfrming memry DIMM that thrttles all parallel jbs. It is imprtant t have tests in place t ensure that all critical cmpnents wrk accrding t their expectatins. Mst f the testing shuld be dne befre cmbining them with ther similar cmpnents. Integratin The fllwing slutins are als part f the cluster delivery slutins in the areas f file system management, cluster installatin and management, and netwrking: The General Parallel File System is a high perfrmance, shared-disk file system that prvides data access frm all ndes in a hmgeneus r hetergeneus cluster f IBM UNIX servers running either the IBM AIX r the Linux perating system. xcat is pen surce cluster cmputing management sftware that is used fr the deplyment and administratin. It prvides the fllwing functins: Prvisins an perating system n a different architecture, such as Linux, AIX, IBM System x, and IBM Pwer Systems. Creates and manage clusters. Installs and manages many cluster machines in parallel. Remtely manages a system thrugh a remte cnsle r distributed shell. IBM HPC Cluster Health Check 4

5 Fr mre infrmatin abut xcat, see the fllwing website: The OpenFabrics Enterprise Distributin (OFED) is pen surce sftware fr RDMA and perating system kernel bypass applicatins. OFED includes kernel-level drivers, channel-riented RDMA and send/receive peratins, perating system kernel bypasses, and a kernel and user-level applicatin prgramming interface (API). OFED is available fr many Linux distributins, including Red Hat Enterprise Linux, SUSE Linux Enterprise Distributin, and s n. OFED cmes with many user space tls that can be used t verify and diagnse the state f InfiniBand interfaces and tplgy. Fr mre infrmatin abut OFED, see the fllwing website: Supprted platfrms The CHC tlkit shuld be installed n the xcat server because many f the individual checks and framewrk use xcat cmmands. In additin, sme f the health checks are MPI parallel applicatins, which run using the POE runtime envirnment. Here are the required sftware packages: xcat POE envirnment Pythn 2.6 Althugh many f the initial tls wrk in bth the x86 and POWER slutins, the initial tlset cncentrates n x86 slutins. There is nthing in the initial framewrk that precludes integrating tls that are geared tward the POWER slutin. Ordering infrmatin The IBM Cluster Health Check tls can be dwnladed frm the IBM HPC Central at the fllwing website: Related infrmatin Fr mre infrmatin, see the fllwing dcuments: IBM HPC Cluster Health Check, SG IBM HPC Central IBM HPC Cluster Health Check 5

6 Ntices This infrmatin was develped fr prducts and services ffered in the U.S.A. IBM may nt ffer the prducts, services, r features discussed in this dcument in ther cuntries. Cnsult yur lcal IBM representative fr infrmatin n the prducts and services currently available in yur area. Any reference t an IBM prduct, prgram, r service is nt intended t state r imply that nly that IBM prduct, prgram, r service may be used. Any functinally equivalent prduct, prgram, r service that des nt infringe any IBM intellectual prperty right may be used instead. Hwever, it is the user's respnsibility t evaluate and verify the peratin f any nn-ibm prduct, prgram, r service. IBM may have patents r pending patent applicatins cvering subject matter described in this dcument. The furnishing f this dcument des nt give yu any license t these patents. Yu can send license inquiries, in writing, t: IBM Directr f Licensing, IBM Crpratin, Nrth Castle Drive, Armnk, NY U.S.A. The fllwing paragraph des nt apply t the United Kingdm r any ther cuntry where such prvisins are incnsistent with lcal law : INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Sme states d nt allw disclaimer f express r implied warranties in certain transactins, therefre, this statement may nt apply t yu. This infrmatin culd include technical inaccuracies r typgraphical errrs. Changes are peridically made t the infrmatin herein; these changes will be incrprated in new editins f the publicatin. IBM may make imprvements and/r changes in the prduct(s) and/r the prgram(s) described in this publicatin at any time withut ntice. Any references in this infrmatin t nn-ibm Web sites are prvided fr cnvenience nly and d nt in any manner serve as an endrsement f thse Web sites. The materials at thse Web sites are nt part f the materials fr this IBM prduct and use f thse Web sites is at yur wn risk.ibm may use r distribute any f the infrmatin yu supply in any way it believes apprpriate withut incurring any bligatin t yu. Infrmatin cncerning nn-ibm prducts was btained frm the suppliers f thse prducts, their published annuncements r ther publicly available surces. IBM has nt tested thse prducts and cannt cnfirm the accuracy f perfrmance, cmpatibility r any ther claims related t nn-ibm prducts. Questins n the capabilities f nn-ibm prducts shuld be addressed t the suppliers f thse prducts. This infrmatin cntains examples f data and reprts used in daily business peratins. T illustrate them as cmpletely as pssible, the examples include the names f individuals, cmpanies, brands, and prducts. All f these names are fictitius and any similarity t the names and addresses used by an actual business enterprise is entirely cincidental. Any perfrmance data cntained herein was determined in a cntrlled envirnment. Therefre, the results btained in ther perating envirnments may vary significantly. Sme measurements may have been made n develpment-level systems and there is n guarantee that these measurements will be the same n generally available systems. Furthermre, sme measurement may have been estimated thrugh extraplatin. Actual results may vary. Users f this dcument shuld verify the applicable data fr their specific envirnment. COPYRIGHT LICENSE: This infrmatin cntains sample applicatin prgrams in surce language, which illustrate prgramming techniques n varius perating platfrms. Yu may cpy, mdify, and distribute these sample prgrams in any frm withut payment t IBM, fr the purpses f develping, using, marketing r distributing applicatin prgrams cnfrming t the applicatin prgramming interface fr the perating platfrm fr which the sample prgrams are written. These examples have nt been thrughly tested under all cnditins. IBM, therefre, cannt guarantee r imply reliability, serviceability, r functin f these prgrams. Cpyright Internatinal Business Machines Crpratin All rights reserved. Nte t U.S. Gvernment Users Restricted Rights -- Use, duplicatin r disclsure restricted by GSA ADP Schedule Cntract with IBM Crp. IBM HPC Cluster Health Check 6

7 This dcument was created r updated n December 17, Send us yur cmments in ne f the fllwing ways: Use the nline Cntact us review frm fund at: ibm.cm/redbks Send yur cmments in an t: redbks@us.ibm.cm Mail yur cmments t: IBM Crpratin, Internatinal Technical Supprt Organizatin Dept. HYTD Mail Statin P Suth Rad Pughkeepsie, NY U.S.A. This dcument is available nline at Trademarks IBM, the IBM lg, and ibm.cm are trademarks r registered trademarks f Internatinal Business Machines Crpratin in the United States, ther cuntries, r bth. These and ther IBM trademarked terms are marked n their first ccurrence in this infrmatin with the apprpriate symbl ( r ), indicating US registered r cmmn law trademarks wned by IBM at the time this infrmatin was published. Such trademarks may als be registered r cmmn law trademarks in ther cuntries. A current list f IBM trademarks is available n the web at The fllwing terms are trademarks f the Internatinal Business Machines Crpratin in the United States, ther cuntries, r bth: AIX IBM POWER Pwer Systems Redbks Redbks (lg) System x The fllwing terms are trademarks f ther cmpanies: Linux is a trademark f Linus Trvalds in the United States, ther cuntries, r bth. UNIX is a registered trademark f The Open Grup in the United States and ther cuntries. Other cmpany, prduct, r service names may be trademarks r service marks f thers. IBM HPC Cluster Health Check 7