Integration of Middleware for Compute Farms

Transcription

1 3. LS-DYNA Anwenderforum, Bamberg 2004 CAE / IT III Integration of Middleware for Compute Farms Nils Burghard Roland Niemeier science + computing ag Some Facts und Figures about s+c Revenue 2003/04: 20 Mio Employees: 190 (September 2004) Locations: Tübingen, München, Berlin, Duisburg Company established: September 1989 Legal Form of Company: Privately Owned Stock Corporation E - III - 1

2 CAE / IT III 3. LS-DYNA Anwenderforum, Bamberg 2004 Our Focus We offer IT services and solutions for efficient IT management in technical computing Revenue Growth 25,0 20,0 Revenue in Mio. Euro 15,0 10,0 5,0 0,0 95/96 96/97 97/98 98/99 99/00 00/01 01/02 02/03 03/04 E - III - 2

3 3. LS-DYNA Anwenderforum, Bamberg 2004 CAE / IT III Company Growth employees Today: about 190 employees More than 85% hold advanced degrees in sciences, engineering, and computer science An educated guess Most of our customers do not want to operate Compute Farms they want solutions for their simulation problems E - III - 3

4 CAE / IT III 3. LS-DYNA Anwenderforum, Bamberg 2004 IT-Services scitservices System Administration of Linux/Unix/Windows Installation and Configurationof Compute Farms Operation of Compute Farms Integration of Clusters in department networks Update of Applications/Operating System Operating System and Application specific Know-How IT s all about Distributed Resource Management Need for Middleware Global Compute Farms Desktop Aggregation Cluster & Departmental Compute Farms Enterprise Compute Farms Increased complexity Heterogeneous platforms Multiple administrative domains Multiple locations Varying security policies Flexible cooperation structures E - III - 4

5 3. LS-DYNA Anwenderforum, Bamberg 2004 CAE / IT III The Administration with scvenus The functunalityof computers aregroup discribed concept through their group membership Performance, unreachable clients Parallelizationof minimize influence Single Central point of administration no local configuration, administration Automated system administration: Method concept fast, reproducible, solid/stable Multi administrator environment Different administrator accounts, different access rights, log files scvenus Modules Venus scvenus User and filesystem administration Configuration administration Konfiguratio Software administration Inventory Inventar Monitoring scvenus Base Venus Base E - III - 5

6 CAE / IT III 3. LS-DYNA Anwenderforum, Bamberg 2004 Integration and Job Flow - Software Layers Easyaccess to compute resources over a WEB interface LSF LS-DYNA, NASTRAN,... Comfortable job flow management with a graphical editor Optimal utilization of compute resources, LSF Integration of different applications Automate and reproduce system administration node node node Operating systems Integration and Job Flow - I System Environment Compute Farm In a preprocessing process the CAE engineers are building the input decks on their workstations WS... E - III - 6

7 3. LS-DYNA Anwenderforum, Bamberg 2004 CAE / IT III Integration and Job Flow - II System Environment Compute Farm The CAE engineers s batch jobs over a WEB interface, generated from EnginFrame WS... Integration and Job Flow - III System Environment Compute Farm With flowguide the batch jobs are prepared to be ed as s in a load sharing tool, like LSF WS... E - III - 7

8 CAE / IT III 3. LS-DYNA Anwenderforum, Bamberg 2004 Integration and Job Flow - IV System Environment Optimal distribution of batch jobs depending on system requirements Compute Farm scheduler. batch queues WS... LSF Parallel: - Control for MPP Jobs - Proper Accounting The Platform LSF Family of Products E - III - 8

9 3. LS-DYNA Anwenderforum, Bamberg 2004 CAE / IT III Enterprise Grid-level: LSF Multicluster Job Forwarding Model Send queue Receive queue Site A Compute Servers You LSF cares for Job transfer data staging Account mapping Accounting Site B Compute Servers E - III - 9

10 CAE / IT III 3. LS-DYNA Anwenderforum, Bamberg 2004 Resource Leasing Model Single system image, ease of admin, scalability Enable fairshare, preemption, pending reason support, chunk jobs, advance reservation, interactive jobs, parallel jobs, across clusters Easy to Configure Begin Queue QUEUE=lease HOSTS= all@siteb End Queue Site A Compute Servers Begin HostExport PER_HOST = hopper curie DISTRIBUTION = [sitea, 10] MEM = 2 GB End HostExport Site B Compute Servers Begin Queue QUEUE= export SNDJOBS_TO = import@siteb Compute End Queue Servers Site A Send queue Receive queue Begin Queue QUEUE = import RCVJOBS_FROM= sitea End Queue Compute Servers Site B E - III - 10

11 3. LS-DYNA Anwenderforum, Bamberg 2004 CAE / IT III Integration and Job Flow - V System Environment Calculation of the batch job in local working directories on the Compute Farm Compute Farm scheduler. batch queues WS... Integration and Job Flow - VI System Environment For the post processing process the results are automated copied back to the WS or other servers Compute Farm scheduler. batch queues WS... CAE engineers are informed per copy results E - III - 11

12 CAE / IT III 3. LS-DYNA Anwenderforum, Bamberg 2004 Summary Simple and comfortable to use solution for CAE engineers (Compute Power out of the Box) Support of heterogeneous environment (Linux, Unix, Windows) Flexible configuration due to modular software layers: Easy to administrate with scvenus Efficient distributed resource management with Platform LSF Comfortable user oriented workflows with flowguide Portal Solutions with EnginFrame Thank you for your attention E - III - 12