Course detail
Practical Parallel Programming
FIT-PPPAcad. year: 2024/2025
The course covers architecture and programming of parallel systems with functional and data parallelism. First, the parallel system theory and program parallelization are discussed. The detailed description of most proliferated supercomputing systems, interconnection network typologies and routing algorithms is followed by the architecture of parallel and distributed storage systems. The course goes on in message passing programming in standardized interface MPI. Consequently, techniques for parallel debugging and profiling are discussed. Last part of the course is devoted to the description of parallel programming patterns and case studies from the are of linear algebra, physical systems described by partial differential equations, N-Body systems and Monte-Carlo methods.
Language of instruction
Number of ECTS credits
Mode of study
Guarantor
Department
Entry knowledge
Von-Neumann computer architecture, computer memory hierarchy, cache memories and their organization, programming in C/C++. Knowledge gained in courses PRL and AVS.
Rules for evaluation and completion of the course
- Assignment 60b
- 30b implementation
- 20b documentation and measuremetns
- 10b oral defense
- Written tests 40b
- 5 short tests, 8b each
Credit Requirements:
-
A minimum of 30 points from the project and a minimum of 20 points from ongoing tests.
Aims
To get familiar with the architecture of distributed supercomputing systems, their interconnection networks and storage. To orientate oneself in parallel systems on the market, be able to assess communication and computing possibilities of a particular architecture and to predict the performance of parallel applications. Learn how to write portable programs using standardized interfaces and languages, specify parallelism and process communication. To learn how to practically use supercoputer for solving complex engineering problems.
Overview of principles of current parallel system design and of interconnection networks, communication techniques and algorithms. Survey of parallelization techniques of fundamental scientific problems, knowledge of parallel programming in MPI. Knowledge of basic parallel programming patterns. Practical experience with the work on supercomputers, ability to identify performance issues and propose their solution.
Knowledge of capabilities and limitations of parallel processing, ability to estimate performance of parallel applications. Language means for process/thread communication and synchronization. Competence in hardware-software platforms for high-performance computing and simulations.
Study aids
Prerequisites and corequisites
Basic literature
Ananth Grama, Anshul Gupta, George Karypis, Vipin Kumar: Introduction to Parallel Computing, Addison-Wesley, 2003, 978-0201648652.Slides: download
Hennessy, J.L., Patterson, D.A.: Computer Architecture - A Quantitative Approach. 5. vydání, Morgan Kaufman Publishers, Inc., 2012, 1136 s., ISBN 1-55860-596-7.
Pacecho, P.: Introduction to Parallel Programming. Morgan Kaufman Publishers, 2011, 392 s., ISBN: 9780123742605 URL: download
Victor Eijkhout: Parallel Programming in MPI and OpenMP Full book: download web version: https://theartofhpc.com/pcse/
William Gropp, Ewing Lusk, Anthony Skjellum: Using MPI - 2nd Edition: Portable Parallel Programming with the Message Passing InterfaceUsing MPI - 2nd Edition: Portable Parallel Programming with the Message Passing Interface, MIT Press, 978-0262571326
Recommended reading
MPI Tutoriál: http://mpitutorial.com/
Elearning
Classification of course in study plans
- Programme MITAI Master's
specialization NGRI , 0 year of study, summer semester, elective
specialization NADE , 0 year of study, summer semester, elective
specialization NISD , 0 year of study, summer semester, elective
specialization NMAT , 0 year of study, summer semester, elective
specialization NSEC , 0 year of study, summer semester, elective
specialization NISY up to 2020/21 , 0 year of study, summer semester, elective
specialization NNET , 0 year of study, summer semester, elective
specialization NMAL , 0 year of study, summer semester, elective
specialization NCPS , 0 year of study, summer semester, elective
specialization NHPC , 1 year of study, summer semester, compulsory
specialization NVER , 0 year of study, summer semester, elective
specialization NIDE , 0 year of study, summer semester, elective
specialization NISY , 0 year of study, summer semester, elective
specialization NEMB , 2 year of study, summer semester, compulsory
specialization NSPE , 0 year of study, summer semester, elective
specialization NEMB , 2 year of study, summer semester, compulsory
specialization NBIO , 0 year of study, summer semester, compulsory
specialization NSEN , 0 year of study, summer semester, elective
specialization NVIZ , 0 year of study, summer semester, elective
Type of course unit
Lecture
Teacher / Lecturer
Syllabus
- Introduction to parallel processing
- Programming patterns for parallel programming
- Message passing interface, pair-wise communications.
- Collective communications.
- Communicators and typologies
- Datatypes
- One-sided communications.
- MPI-IO
- Lustre, HDF5
- Parallel code profiling and tracing.
- Hybrid programming OpenMP/MPI
- Technologies of interconnection networks (Infiniband), topology and routing algorithms, switching, flow control.
- Case studies: Fluid dynamics, N-Body systems, Monte-Carlo.
Exercise in computer lab
Teacher / Lecturer
Syllabus
- MPI: Point-to-point communications
- MPI: Collective communications
- MPI: Communicators
- MPI: Data types
- MPI: One-sided communications
- MPI: Parallel IO
- HDF5: Parallel IO
- Profiling and tracing of parallel applications
Project
Teacher / Lecturer
Syllabus
- A parallel program in MPI on the supercomputer.
Elearning