X
Advanced Search
Dawei Mu, Po Chen, Liqiang Wang (2013). Accelerating the discontinuous Galerkin method for seismic wave propagation simulations using multiple GPUs with CUDA and MPI. Earthq Sci 26(6): 377-393. DOI: 10.1007/s11589-013-0047-7
Citation: Dawei Mu, Po Chen, Liqiang Wang (2013). Accelerating the discontinuous Galerkin method for seismic wave propagation simulations using multiple GPUs with CUDA and MPI. Earthq Sci 26(6): 377-393. DOI: 10.1007/s11589-013-0047-7
shu

Accelerating the discontinuous Galerkin method for seismic wave propagation simulations using multiple GPUs with CUDA and MPI

  • 1.

    Department of Geology and Geophysics, University of Wyoming, Laramie, WY, USA

  • 2.

    Computer Science Department, University of Wyoming, Laramie, WY, USA

More Information
  • Corresponding author:

    P. Chen, e-mail: pochengeophysics@gmail.com

  • Received Date: 24 Jul 2013
  • Accepted Date: 12 Nov 2013
  • Available Online: 30 May 2022
  • Published Date: 30 Dec 2013
Chinese summary

    Abstract

  • We have successfully ported an arbitrary high-order discontinuous Galerkin method for solving the three-dimensional isotropic elastic wave equation on unstructured tetrahedral meshes to multiple Graphic Processing Units (GPUs) using the Compute Unified Device Architecture (CUDA) of NVIDIA and Message Passing Interface (MPI) and obtained a speedup factor of about 28.3 for the single-precision version of our codes and a speedup factor of about 14.9 for the double-precision version. The GPU used in the comparisons is NVIDIA Tesla C2070 Fermi, and the CPU used is Intel Xeon W5660. To effectively overlap inter-process communication with computation, we separate the elements on each subdomain into inner and outer elements and complete the computation on outer elements and fill the MPI buffer first. While the MPI messages travel across the network, the GPU performs computation on inner elements, and all other calculations that do not use information of outer elements from neighboring subdomains. A significant portion of the speedup also comes from a customized matrix-matrix multiplication kernel, which is used extensively throughout our program. Preliminary performance analysis on our parallel GPU codes shows favorable strong and weak scalabilities.

    • FullText(HTML)
  • loading
    • References (49)
  • Abdelkhalek R, Calandra H, Coulaud O, Roman J, Latu G (2009) Fast seismic modeling and reverse time migration on a GPU cluster. The 2009 International Conference on High Performance Computing & Simulation, 2009. HPCS'09, pp 36-43
    Aminzadeh F, Brac J, Kunz T (1997) 3-D salt and overthrust models. SEG/EAGE 3-D modeling series No. 1, 1997. Society of Exploration Geophysicists and European Association of Exploration Geophysicists
    Bao H, Bielak J, Ghattas O, Kallivokas LF, O'Hallaron DR, Shewchuk JR, Xu J (1998) Large-scale simulation of elastic wave propagation in heterogeneous media on parallel computers. Comput Methods Appl Mech Eng 152:85-102 doi: 10.1016/S0045-7825(97)00183-7
    Bell N, Garland M (2009) Efficient sparse matrix-vector multiplication on CUDA. In: Proceedings of ACM/IEE E Conference Supercomputing (SC), Portland, OR, USA
    Carcione JM (1994) The wave equation in generalized coordinates. Geophysics 59:1911-1919 doi: 10.1190/1.1443578
    Chen P, Jordan T, Zhao L (2005) Finite-moment tensor of the 3 September 2002 Yorba Linda earthquake. Bull Seismol Soc Am 95:1170-1180 doi: 10.1785/0120040094
    Chen P, Jordan TH, Zhao L (2007) Full three-dimensional tomography: a comparison between the scattering-integral and adjoint-wavefield methods. Geophys J Int 170:175-181 doi: 10.1111/gji.2007.170.issue-1
    Chen P, Jordan TH, Zhao L (2010) Resolving fault plane ambiguity for small earthquakes. Geophys J Int 181:493-501 doi: 10.1111/gji.2010.181.issue-1
    Cockburn B, Karniadakis GE, Shu CW (2000) Discontinuous Galerkin methods, theory, computation and applications. LNC-SE, 11. Springer
    Dumbser M, Käser M (2006) An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes-Ⅱ. The three-dimensional isotropic case. Geophys J Int 167(1):319-336. doi: 10.1111/j.1365-246X.2006.03120.x
    Dumbser M, Käser M, Toro EF (2007) An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes-Ⅴ. Local time stepping and p-adaptivity. Geophys J Int 171(2):695-717. doi: 10.1111/j.1365-246X.2007.03427.x
    Graves RW (1996) Simulating seismic wave propagation in 3D elastic media using staggered-grid finite differences. Bull Seismol Soc Am 86(4):1091-1106
    Graves RW, Aagaard BT, Hudnut KW, Star LM, Stewart JP, Jordan TH (2008) Broadband simulations for Mw 7.8 southern San Andreas earthquakes: ground motion sensitivity to rupture speed. Geophys Res Lett 35:L22302. doi: 10.1029/2008GL035750
    Graves R, Jordan T, Callaghan S, Deelman E (2010) CyberShake: a physics-based seismic hazard model for Southern California. Pure Appl Geophys 168:367-381
    Hughes TJR (1987) The finite element method—linear static and dynamic finite element analysis. Prentice Hall, Englewood Cliffs
    Igel H (1999) Wave propagation in three-dimensional spherical sections by the Chebyshev spectral method. Geophys J Int 136:559-566 doi: 10.1046/j.1365-246x.1999.00758.x
    Igel H, Mora P, Riollet B (1995) Anisotropic wave propagation through finite-difference grids. Geophysics 60:1203-1216 doi: 10.1190/1.1443849
    Käser M, Dumbser M (2006) An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes-Ⅰ. The two-dimensional isotropic case with external source terms. Geophys J Int 166(2):855-877 doi: 10.1111/gji.2006.166.issue-2
    Käser M, Dumbser M (2008) A highly accurate discontinuous Galerkin method for complex interfaces between solids and moving fluids. Geophysics 73(3):T23-T35 doi: 10.1190/1.2870081
    Käser M, Dumbser M, la Puente de J, Igel H (2007) An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes-Ⅲ. Viscoelastic attenuation. Geophys J Int 168(1):224-242. doi: 10.1111/j.1365-246X.2006.03193.x
    Käser M, Pelties C, Castro CE (2010) Wavefield modeling in exploration seismology using the discontinuous Galerkin finite-element method on HPC infrastructure. Lead Edge 29:76-84 doi: 10.1190/1.3284056
    Klöckner A, Warburton T, Bridge J, Hesthaven JS (2009) Nodal discontinuous Galerkin methods on graphics processors. J Comput Phys 228(21):7863-7882. doi: 10.1016/j.jcp.2009.06.041
    Komatitsch D, Tromp J (1999) Introduction to the spectral-element method for 3-D seismic wave propagation. Geophys J Int 139:806-822 doi: 10.1046/j.1365-246x.1999.00967.x
    Komatitsch D, Tromp J (2002) Spectral-element simulations of global seismic wave propagation-Ⅱ. Three-dimensional models, oceans, rotation and self-gravitation. Geophys J Int 150(1):303-318 doi: 10.1046/j.1365-246X.2002.01716.x
    Komatitsch D, Vilotte JP (1998) The spectral-element method: an efficient tool to simulate the seismic response of 2D and 3D geological structures. Bull Seismol Soc Am 88:368-392
    Komatitsch D, Liu Q, Tromp J, Suss P, Stidham C, Shaw J (2004) Simulations of ground motion in the Los Angeles basin based upon the spectral-element method. Bull Seismol Soc Am 94:187-206 doi: 10.1785/0120030077
    Komatitsch D, Michéa D, Erlebacher G (2009) Porting a high-order finite-element earthquake modeling application to NVIDIA graphics cards using CUDA. J Parallel Distrib Comput 69(5):451-460 doi: 10.1016/j.jpdc.2009.01.006
    Komatitsch D, Göddeke D, Erlebacher G, Michéa D (2010) Modeling the propagation of elastic waves using spectral elements on a cluster of 192 GPUs. Comput Sci Res Dev 25(1):75-82
    la Puente De J, Käser M, Dumbser M, Igel H (2007) An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes-Ⅳ. Anisotropy. Geophys J Int 169(3):1210-1228 doi: 10.1111/gji.2007.169.issue-3
    la Puente De J, Käser M, Cela JM (2009) SeisSol optimization, scaling and synchronization for local time stepping. Science and Supercomputing in Europe, pp 300-302
    Lee E, Chen P, Jordan T, Wang L (2011) Rapid full-wave centroid moment tensor (CMT) inversion in a three-dimensional earth structure model for earthquakes in Southern California. Geophys J Int 186:311-330 doi: 10.1111/gji.2011.186.issue-1
    Levander AR (1988) Fourth-order finite difference P-SV seismograms. Geophysics 53:1425-1436 doi: 10.1190/1.1442422
    Liu Q, Tromp J (2006) Finite-frequency kernels based on adjoint methods. Bull Seismol Soc Am 96:2383-2397 doi: 10.1785/0120060041
    Lysmer J, Drake LA (1972) A finite element method for seismology. In: Alder B, Fernbach S, Bolt BA (eds) Methods in Computational Physics, vol 11. Academic Press, New York, Ch. 6, pp 181-216
    Madariaga R (1976) Dynamics of an expanding circular fault. Bull Seismol Soc Am 65:163-182
    Michéa D, Komatitsch D (2010) Accelerating a three-dimensional finite-difference wave propagation code using GPU graphics cards. Geophys J Int 182(1):389-402
    Moczo P, Kristek J, Vavrycuk V, Archuleta RJ, Halada L (2002) 3D heterogeneous staggered-grid finite-difference modeling of seismic motion with volume harmonic and arithmetic averaging of elastic moduli and densities. Bull Seismol Soc Am 92:3042-3066 doi: 10.1785/0120010167
    Mora P (1989) Modeling anisotropic seismic waves in 3-D, 59th Ann. Int. Mtg Exploration Geophysicists, expanded abstracts, pp 1039-1043
    Mu D, Chen P, Wang L (2013) Accelerating the discontinuous Galerkin method for seismic wave propagation simulations using the graphic processing unit (GPU)—single-GPU implementation. Comput Geosci 51:282-292 doi: 10.1016/j.cageo.2012.07.017
    Okamoto T, Takenaka H, Nakamura T, Aoki T (2010) Accelerating large-scale simulation of seismic wave propagation by multi-GPUs and three-dimensional domain decomposition. Earth Planet Space 62(12):939-942. doi: 10.5047/eps.2010.11.009
    Olsen K (2000) Site amplification in the Los Angeles basin from three-dimensional modeling of ground motion. Bull Seismol Soc Am 90:577-594
    Tessmer E (1995) 3-D Seismic modelling of general material anisotropy in the presence of the free surface by a Chebyshev spectral method. Geophys J Int 121:557-575 doi: 10.1111/gji.1995.121.issue-2
    Tessmer E, Kosloff D (1994) 3-D elastic modeling with surface topography by a Chebyshev spectral method. Geophysics 59(3):464-473 doi: 10.1190/1.1443608
    Toro EF (1999) Riemann Solvers and Numerical Methods for Fluid Dynamics. Springer, Berlin
    Toro EF, Titarev VA (2002) Solution of the generalized Riemann problem for advection-reaction equations. Proc R Soc Lond 458:271-281 doi: 10.1098/rspa.2001.0926
    Tromp J, Komatitsch D, Liu Q (2008) Spectral-element and adjoint methods in seismology. Commun Comput Phys 3:1-32
    Virieux J (1984) SH-wave propagation in heterogeneous media: velocity-stress finite-difference method. Geophysics 49:1933-1942 doi: 10.1190/1.1441605
    Virieux J (1986) P-SV wave propagation in heterogeneous media: velocity-stress finite-difference method. Geophysics 51:889-901 doi: 10.1190/1.1442147
    Wang Z, Peng S, Liu T (2010) Modeling seismic wave propagation using graphics processor units (GPU). The Second International Symposium on Networking and Network Security (ISNNS 2010), pp 129-132
    • Related Articles

Catalog

    Liqiang Wang
    • 2.

      Computer Science Department, University of Wyoming, Laramie, WY, USA

    1. On this Site
    2. On Google Scholar
    3. On PubMed

    Figures(10)  /  Tables(1)

    Get Citation
    PDF
    XML
    Article views (438) PDF downloads (1) Cited by()
    Turn off MathJax
    Article Contents

    震球期刊

    EQS

    E-mail Alert RSS

    Supported by: Beijing Renhe Information Technology Co., Ltd.  Baidu Analytics

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return