Resource Management Portal CHReME with Web Interface for Scientific Users. Nisha Kurkure C-DAC, India

Transcription

1 Resource Management Portal CHReME with Web Interface for Scientific Users Nisha Kurkure C-DAC, India

2 C-DAC IN HPC

3 High Performance Computing National PARAM SuperComputing Facility. 10 Gbps PARAMnet III. Reconfigurable Computing Systems (RCS): Bio-informatics sequence search commercial deployment. Design and Deployment of High end HPC systems at various premiere institutions like IITM-Pune, JNU Delhi, TIFR Mumbai, BDU Tiruchirapally, NIO Goa, NCL Pune. Maintenance and Application Support for HPC systems at University of Hyderabad, NCMRWF, JNU Delhi, IIT Delhi, NIO Goa, IITM Pune. Deployed Grid File System prototype on Grid test bed between Pune and Bangalore. GFS Addresses data location independence for HPC applications.

4 C-DAC Activities Science & Engineering, Strategic Sectors, Health, e-governance, Education, Power & Industrial Sector, Agriculture, Rural Areas End-to-End Solutions Enabling Technologies HPC, Language Computing, Speech Technology, e-security, Geomatics, Ubiquitous Computing, Embedded Systems, VLSI, Broadband & Wireless, Software Technologies,

5 Need to develop CHReME Growth of Parallel Applications has posed a serious challenge of efficient, easy usage and management of resources of HPC systems In domain specific organizations, Installation and porting of relevant applications is very difficult as the activity is computational in nature Very Tedious for users to run heterogeneous scientific application codes using command line Facilitate ease of use of large clusters as well as their increased adoption Ability to address Large and varied Data sets efficiently Handle the work load Ever increasing users (Novices????)

6 About CHReME Salient Features Developed on an open source platform and ecosystem Fully integrated environment with Industry standard Schedulers User friendly web based GUI providing access to various cluster resources Submission, monitoring and management of jobs through GUI Easy and secure access to all the cluster resources from remote host machine Optimum Utilization of Cluster Resources

7 CHReME Snapshot

8 CHReME Additional Features Creation and Management of various Cluster Resources viz. Queue, Parallel Environment and Users Timely notification regarding job status Provides application specific interface for pre-compiled applications in various domains viz. Weather Forecasting Fully customizable - Scope for integrating numerous HPC Applications

9 Weather Research Forecast Model (WRF) The Weather Research and Forecasting (WRF) Model is designed to serve both operational forecasting and atmospheric research needs It features multiple dynamical cores, a 3- dimensional variational (3DVAR) data assimilation system, and a software architecture allowing for computational parallelism as well as system extensibility WRF is suitable for a broad spectrum of applications across scales ranging from meters to thousands of kilometers

10 WRF & CHReME Portal CHReME WRF Portal is an integrated solution with WRF application execution interface CHReME WRF Portal guides users through the entire cycle of execution i.e. Pre-processing (WPS) Running WRF model Post-processing of the model GUI facilitates users with a modular and seamless approach for WRF execution by Creating various workflows Storing the workflow execution stages Setting up of environment variables Submission of the WRF execution jobs Monitoring status of the workflows Visualization of the output.

11 WRF Portal: Workflow Creation Snapshot

12 WRF Portal: Workflow Creation For creating a new workflow, user has to specify a name for the workflow Workflow name is a single name given to all the steps the user performs for running WRF model i.e. Geogrid Ungrib Metgrid Real.exe Wrf.exe After entering the workflow name, the user will be directed to the step of configuring the workflow details. Initially once, the user will create a workflow and perform all the steps, further the user has to just load the workflow and run the model

13 WRF Portal: Geogrid run snapshot

14 WRF Portal - Geogrid run The user shall enter the values required for running the geogrid The user can view the namelist.wps file which is present in the WPS directory. The namelist.wps file is edited by the portal according to the values entered by the user For browsing the folder path the user is provided with a browsing feature After pressing Execute button, the user will be redirected to a page where the user can see the running status of geogrid, the job of running geogrid is done through the scheduler. Following are the outputs created after running geogrid. The netcdf files generated after running geogrid. The log generated after running geogrid. The Processing details generated while running geogrid. The error generated while running geogrid. The status can be monitored from the job progress area

15 WRF Portal - Ungrib run snapshot

16 WRF Portal - Ungrib run Enter the values required for running ungrib After pressing execute the user will be directed to a page where one can view the running process of Ungrib, done through the scheduler Following are the outputs created after running ungrib The files generated after running ungrib The Log generated after running ungrib The Processing log of Ungrib The errors generated while running Ungrib After successful execution of ungrib, a link for executing metgrid would be highlighted

17 WRF Portal: Metgrid run snapshot

18 WRF Portal: Metgrid run The user will be asked to enter the run directory path of WRF Run the metgrid After execution of the metgrid the following output files are created: The met files generated after running metgrid The log generated The processing log The error generated while running metgrid The user has the option of re running metgrid or proceeding to the next step of running real.exe

19 WRF Portal: Run real.exe snapshot

20 WRF Portal: Run wrf.exe snapshot

21 WRF Portal: Run WRF Model The user can select the number of processors for running real.exe After pressing Run the user will be redirected to page where he can check the status of the run of real.exe After running the real.exe following outputs are created : The files generated after running real.exe The output log generated as the job is submitted to the scheduler The error log generated while running real.exe Update the namelist.input file to run wrf.exe User can re run wrf.exe by pressing Re run wrf.exe User can store the namelist.wps and namelist.input files after running the entire model for future reference At any time, the user can load another workflow to run the WRF model with the entries specific to that workflow The user can only delete the workflow created by him

22 Advantages of CHReME Scientific users can now concentrate on their respective domains without having to worry about computational aspects of supercomputing facilities. It Scales to the environments with thousands of users that can access this Portal through various clusters Impact on the existing network and cluster infrastructure is minimal: The HPC Access Portal server can be completely separate from the cluster, as it uses SSH to communicate with HPC Cluster Scheduler Built using JSP/Struts, a specification making the server installation portable and scalable Integrating new Parallel Applications and additional tools to those that are provided by HPC Resource Access and Management Portal The HPC Portal facilitates submission and monitoring of the job and hence freeing users from the strenuous tasks of command line submission and monitoring It also helps the Administrator to Monitor the entire Cluster Users, Job submitted by the users, Cluster Resources and various reservations by the user that are being used

23 Future Work Development of a application integration module for various other domains like Bioinformatics, Oceanography, Computational Chemistry etc. with customised GUIs. Support for advanced reservation policies of resource manager. Application level check-pointing feature.

24