STUDYING THE TRANSPORT OF POLLUTANTS IN THE ATMOSPHERE USING AN IBM BLUE GENE/P COMPUTER

Size: px

Start display at page:

Download "STUDYING THE TRANSPORT OF POLLUTANTS IN THE ATMOSPHERE USING AN IBM BLUE GENE/P COMPUTER"

Clyde Holland
5 years ago
Views:

1 STUDYING THE TRANSPORT OF POLLUTANTS IN THE ATMOSPHERE USING AN IBM BLUE GENE/P COMPUTER Krassimir Georgiev (1) In cooperation with: Zahari Zlatev (2) and Tzvetan Ostromsky (1) (1) Institute for Information and Communication Technologies, Bulgarian Academy of Sciences, Acad. G. Bonchev str., bl. 25-A, 1113 Sofia, Bulgaria (2) National Environmental Research Institute, Aarhus University Frederiksborgvej 399, P.O. Box 358, DK-4000 Roskilde, Denmark 0 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

2 Outline of the talk Introduction The mathematical background of UNI DEM Organization of the computational process Parallelization strategy and the supercomputer used Analysis of the performed runs of UNI DEM on an IBM BlueGene/P Computer Some conclusions 1 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

3 Introduction Air pollution and especially the reduction of air pollutants to some acceptable levels High resolution comprehensive air pollution models (many pollutants; advanced chemical modules; all physical processes) very much time consuming High-performance computer architectures 2 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

4 Introduction Danish Eulerian Model for long-range transport of air pollutants (UNI-DEM) vector computers (CRAY C92A, Fujitsu, etc.) parallel computers with distributed memory (IBM SP, CRAY T3E, Beowulf clusters, Macintosh G4 clusters, etc.) parallel computers with shared memory (SGI Origin, SUN, etc.) parallel computers with two level of parallelism (IBM SMP, IBM BlueGene/P, clusters of multiprocessor nodes, etc.) 3 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

5 Some important applications Studying the phenomenon itself (better understanding the physical and chemical processes either in scientific studies or in the treatment of tasks required by policy makers); Improving as much as possible the reliability of the control strategies that are to be used for keeping the pollution levels under the prescribed acceptable limits; Studying the ozone episodes; Source-receptor relations; Calculating losses of crops due to high ozone levels; Influence of future climate changes on air pollutants critical levels, etc.; 4 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

6 Major difficulties the need to carry out extensive computations; the need to store and handle very large input-output files; the need to visualize the output data in order to be able to see the trends and relationships hidden behind a great ammount of digital data produced by the model; the need to validate the model results (to show their reliability) 5 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

7 Let the model to be run for one year period with time step of 2.5 minutes times equations RAM 48 GB archive and data storage requirements 400 GB 6 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

8 The 3D version of the Danish Eulerian Model c s t = (uc s) x + x ( (vc s) y K x c s x ) (wc s) z + y ( K y c s y ) + z ( K z c s z ) + E s + Q s (c 1, c 2,... c q ) (k 1s + k 2s )c s, s = 1,2,... q. c s the concentrations of the chemical species; u, v, w the wind components along the coordinate axes; K x, K y, K z diffusion coefficients; E s the emissions; k 1s, k 2s dry / wet deposition coefficients; Q s (c 1, c 2,... c q ) non-linear functions describing the chemical reactions between species under consideration (Gery et al. (1989)). 7 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

9 Splitting into submodels c (1) s t c (2) s t s ) = (uc(1) x = x c K (2) s x x (vc(1) s ) y + y c K (2) s y y horizontal advection horizontal diffusion c (3) s t c (4) s t c (5) s t = E s + Q s (c (3) 1, c(3) 2,... c(3) q ) = (k 1s + k 2s )c (4) s s ) = (wc(5) z + z c K (5) s z z chemistry & emissions deposition vertical transport Related work: Strang (1968); Marchuk (1982); McRae, Goodin & Seinfeld (1984); Lancer & Verwer (1999); Dimov, Farago & Zlatev (1999). 8 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

10 Space discretization and numerical treatment Finite elements (1D linear first order, bilinear, nonconforming) dg (i) dt = f (i) (t, g (i) ) g (i) R N x N y q, f (i) R N x N y q N x, N y = numbers of grid points; q = number of chemical species; f (i) (i = 1,...,4) depend on the discretization methods; g (i) (i = 1,...,4) contain of the concentrations. Predictor-corrector methods with several different correctors in advection-diffusion submodel QSSA (Quasi-Steady-State Algorithm) in chemistry-emission submodel Exact solution in deposition submodel 9 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

11 Size of the computational task Grid No. of species No. of eq. s per system of ODE s at every time step. one month period = advection time-steps one year period advection time-steps six times smaller time-step (150 sec.) in the chemical submodel time-steps per year several hundred of runs (ozone episodes for many years, global climate changes scenarios, etc.) 10 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

12 Parallelization strategy Distributed memory parallelization model via MPI. For maximum portability only standard MPI routines are used in the UNI DEM code. Based on domain partitioning of the horizontal grid Algorithm 1: strips in direction north - south Algorithm 2: grid structure domain overlapping of the advection-diffusion subproblem (requires communication on each time step (communication stage). nonoverlapping subdomains in chemistry-deposition subproblem (improving the data locality for more efficient cache utilization by using chunks to group properly the small tasks) pre-processing and post-processing stages are needed for scattering the input data and gathering the results (cheap and reduce the communications during actual computations, but their relative weight grows up with increasing the number of MPI tasks (affects the total speed-up and efficiency). 11 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

13 IBM Blue Gene/P computer used Location: Bulgarian Supecomputer Center, Sofia consists of two racks, 2048 Power PC 450 based compute nodes, 8192 processor cores, and a total of 4TB random access memory. Each processor core has a double precision, dual pipe floating point core accelerator. 16 I/O nodes are connected via fibre optics to a 10 Gb.s Ethernet switch. The smallest partition size, available currently, is 128 compute nodes (512 processor cores). The theoretical performance of the computer is Tflops while the maximum LINPACK performance achieved is Tlops ( 84%). 12 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

14 What about CHUNKSIZE? Size of chunks Case 1 (CPU in hours) Case 2 (CPU in hours) Case 1: grid (10km 10km cells); advection time step: 150 s.; chemical time step: 150 s.; number of processors used: 120; Computer: IBM BlueGene/P, Algorithm 1. Case 2: grid (10km 10km cells); advection time step: 90 s.; chemical time step: 9 s.; number of processors used: 120; Computer: IBM BlueGene/P, Algorithm Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

15 UniDEM on (10km 10km) mesh Algorithm 1 Computer Number of Communi- CPU time Efficiency type processors cations (in hours) (in %) Cray T3E 32 MPI % IBM SP 32 MPI % IBM BluGene/P 32 MPI % 14 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

16 Runs on IBM BG/P computer Algorithm 1 Computational Measured Number of processors used proces quantities CPU time Advection Speed-up in % CPU time Chemistry Speed-up in % CPU time Overhead Speed-up in % CPU time Total Speed-up in % Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

17 Runs on IBM BG/P computer Algorithm 2 No. of CPU time Speedup Efficiency Communication proc. (in sec) (in % ) time in sec (in %) (3.2) (3.3) (4.1) (5.0) (5.9) (6.3) (8.0) (12.3) (20.7) 16 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

18 Applications of the UniDEM output Seasonal variation of the concentrations; Annual variation of the concentrations; Climatic scenarious; 17 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

19 Applications of the UniDEM output Studying the number of BAD DAYS Assume that c max is the maximum of the eight-hour averages of the calculated by the model ozone concentrations in a given day at site A. If the condition c max > 60 ppb is satisfied at least once in the day under consideration, then the expression a bad day will be used for such a day at site A. Bad days can have damaging effects on some groups of human beings (e. g. people who suffer from asthmatic diseases). Therefore, the number of such days should be reduced as much as possible. Two important aims are stated in the Ozone Directive issued by the EU Parliament in year 2002 (European Parliament 2002): Target aim: The number of bad days in any site of the European Union should not exceed 25 after year Long-term aim; No bad day should occur in the European Union (the year after which the long-term aim has to be satisfied is not specified in the EU Ozone Directive). 18 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

20 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

21 Applications of the UniDEM output Studying exceeded AOT40 values The AOT40 values are normally calculated by using the formula: AOT40 = N max(c i 40.0), i=1 where N is the number of day-time hours in the period under consideration (for crops this period contains the months May, June and July and the notation AOT40C is used, while the period from April to September as well as the notation AOT40F is used for forest trees), c i is the calculated by the model ozone concentration. With UniDEM model we are able to calculate c i for all values i = 1, 2,... N,. The critical levels in the European Union are: (a) 3000 ppb.hours for AOT40C; (b) 10000ppb.hours for AOT40F. see: European Parliament: Directive 2002/3/EC of the European Parliament and the Council of 12 February 2002 relating to ozone in ambient air, Official Journal of the European Communities, Vol. L67 ( ), pp Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

22 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

23 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

24 Applications of the UniDEM output large Studying the pollutant concentrations The output results from the UNI DEM runs can be used for studying the distribution of the concentrations of the involved pollutants over the area contains Europe or some choosen subdomains of interest. On the next Figure one can see an example for studying the distribution of four of the most important pollutants nitrogen di oxide, nitrogen mono oxide, sulphur dioxide and sulphate over the the Southeastern Europe. 23 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

25 24 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

26 Conclusions By using high performance parallel computer IBM Blue Gene/P to run the variable grid-size code UniDEM, detailed air pollution results for a large region (whole Europe or some parts of interest) and for a long periods (many years, climatic scenarios) can be obtained within a reasonable time. The parallel code, created by using MPI standard library, appears to be highly portable and shows good efficiency and scalability. 25 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

27 Acknowledgments This research is supported in part by: Grants DCVP-02/1 and DO /2008 from the Bulgarian NSF the Bulgarian National Center for Supercomputing Applications (NCSA) giving access to the IBM Blue Gene/P computer in Sofia 26 Supercomputing - New Challenge for Science and Industry, December 9-10, 2010

Some typical applications related to environmental studies

Some typical applications related to environmental studies 1. Need for reliable input data 2. Meteorological data 3. Emission data 4. Problems with huge output sets 5. Need for visualization and animation