TDDC78 literature (2023)

Course material (mandatory):

• C. Kessler:
  Slide material for all lectures.
  Accessible electronically to registered students on this page (login and password sent to registered participants by email shortly before course start).

• C. Kessler:
  Design and Analysis of Parallel Algorithms: An Introduction.
  Compendium for TDDC78 and TDDD56, Edition Spring 2020 (may be updated during the course).
  Accessible electronically to registered students on this page (login and password sent to registered participants by email).
  Covers Lectures 4-5 (Parallel program design), 11-12 (Analysis of parallel algorithms) and 13-14 (Parallel basic linear algebra, parallel solving of linear equation systems).

• L. Elden, H. Park and Y. Saad.
  Scientific Computing on High Performance Computers (compendium),
  version 2008.
  Accessible electronically to registered students on this page (login and password sent by email).
  Background reading for Lectures 13-14 (parallel linear algebra).

Complementary reading:

If you prefer to work with a textbook beyond the above material, we recommend the following one, or the alternatives given below:

• B. Wilkinson, M. Allen: Parallel Programming, Second Edition. Prentice Hall, 2005. ISBN 0-13-140563-2.
  Available for loan and as course reference copy in the Campus Valla library and in the Bokakademin shop on the campus.

Alternative textbooks:

• C. Lin, L. Snyder: Principles of Parallel Programming. Addison-Wesley, 2009.
• Ian Foster: Designing and building parallel programs. Addison-Wesley 1995.
  (the previous course book, out of print, but available online.)
  Errata
  This excellent book still matches the course's level and contents well, especially the PCAM method collection for the design of parallel programs, but the chapters on tools and programming models are outdated, topics such as OpenMP, MPI-2 and newer MPI versions are missing completely.
• T. Rauber, G. Rünger: Parallel Programming: for Multicore and Cluster Systems. Springer, 2013.

These books can be reused in the course TDDD56 Multicore and GPU Programming.

Advanced reading

The following book gives a more in-depth treatment of the topic (beyond what we take up in the course):

• Georg Hager, Gerhard Wellein: Introduction to High Performance Computing for Scientists and Engineers. CRC Press, 2010. Print ISBN: 978-1-4398-1192-4 eBook ISBN: 978-1-4398-1193-1
  Campus-Valla library has one printed copy for loan and one e-book copy for electronic loan.

Online resources:

• Online version of Ian Foster: Designing and building parallel programs.
• A nice tutorial giving an Introduction to Parallel Computing by B. Barney, LLNL
• TOP-500 list
• Pthreads tutorial at LLNL
• Simple MPI example codes: PI calculation in C pi_mpi.c, in Fortran pi_mpi.f
  Many more MPI source code examples can be found on the web.

• Some open-source MPI implementations (also usable on uniprocessor and multicore systems):
  • MPICH2 for Windows (x86-32, x86-64, IA-64), Unix, also Linux, FreeBSD
  • OpenMPI for Linux (32 and 64bit), OS-X, Solaris
• OpenMP forum (www.openmp.org): contains the official standard documentation, a comprehensive tutorial, a few sample programs, links to OpenMP workshops, benchmarks, and more.

• Some open-source OpenMP compilers:
  • GCC version 4.2 or later
  • Open64 Compiler and Tools

• VAMPIR
• Tutorial: Using VAMPIR by Pallas GmbH, 1998
• SCALASCA performance analysis toolset
• On best practices in performance measurement and reporting:
  T. Hoefler, R. Belli: Scientific Benchmarking of Parallel Computing Systems - Twelve ways to tell the masses when reporting performance results. Proc. SC'15, Nov. 2015. ACM.

• A survey of tools for parallel debugging and performance analysis can be found in Chapter 4 of:
  High Performance Tools & Technologies by Michael Collette, Bob Corey, and John Johnson, Lawrence Livermore National Laboratory, Technical report UCRL-TR-209289, December, 2004.
• W. Hillis, G. Steele: Data parallel algorithms. Comm. ACM 29(12), Dec. 1986
• Jack Dongarra, a HPC software pioneer, receives the ACM 2021 Turing Award (which is widely considered as the Nobel prize in computer science). Dongarra made important contributions to many software frameworks and tools covered in our course, such as BLAS, LINPACK, and MPI.
  For a recent presentation by Dongarra, see this youtube video: A Not So Simple Matter of Software (2022).

Further reading:

• Rohit Chandra, Leonardo Dagum, Dave Kohr, Dror Maydan, Jeff McDonald, Ramesh Menon:
  Parallel Programming in OpenMP.
  Morgan Kaufmann Publishers, 2001.
  The book is available in the Campus Valla (TekNat) library.

• Barbara Chapman et al.: Using OpenMP - Portable Shared Memory Parallel Programming. MIT press, 2007.
  Covers OpenMP 2.5

• Jörg Keller, Christoph Kessler, Jesper Träff:
  Practical PRAM Programming.
  Wiley Interscience, 2001.
  Chapter 2 gives a summary of the PRAM model and introduces basic theory of parallel processing (Time, work, cost, Amdahl's law, Brent's theorem, isoefficiency). The book is available in the Campus Valla (TekNat) library.

• W. Petersen and P. Arbenz:
  Introduction to Parallel Computing. A Practical Guide with Examples in C.
  Oxford University Press, 2004.
  Covers exclusively numerical algorithms. Examples in MPI and OpenMP. A text for those who want to specialize in computational science and numerical analysis. The book is available in the Campus Valla (TekNat) library.

• J. Hennessy, D. Patterson: Computer Architecture, A Quantitative Approach, 4th ed.. Morgan Kaufmann, 2007.
  The second edition (1996), third edition (2002) and fourth edition (2007) each contain a chapter on multiprocessor architecture. The Campus Valla (TekNat) library has copies. As an alternative:

• D. Culler, J. Singh, A. Gupta: Parallel Computer Architecture. Morgan Kaufmann, 1998.
  Contains an in-depth treatment of SMP and CC-NUMA shared-memory architectures.

• Articles on concepts and trends in high-performance computing:
  
  • IEEE Computer 30(9), Sep. 1997 Special issue on Chip multiprocessors, and IEEE Micro Sep/Oct 1997 on simultaneous multithreading.
  • On Beowulf clusters:
    T. Sterling: The scientific workstation of the future may be a pile of PCs, Communications of the ACM 39(9), Sep. 1996.
  • IEEE Computer 40(3), March 2007 Special issue on High-Performance Reconfigurable Computing with articles on FPGA accelerators and GPU programming.
  • Technical details on HPC interconnects:
    PRACE: PRACE Best Practice Guide of Modern Interconnects, PRACE technical report, Feb. 2019.
  • J. Gebis, D. Patterson: Embracing and Extending 20th Century Instruction Set Architectures. IEEE Computer 40(4): 68-75, April 2007. Summary of SIMD instruction set extensions in standard microprocessors.
  • M. Hill, M. Marty: Amdahl's Law in the Multicore Era. IEEE Computer 41(7): 33-37, July 2008.

Page responsible: Webmaster
Last updated: 2023-04-13