Course detail

Practical Parallel Programming

FIT-PPPAcad. year: 2020/2021

The course covers architecture and programming of parallel systems with functional and data parallelism. First, the parallel system theory and program parallelization are discussed. The detailed description of most proliferated supercomputing systems, interconnection network typologies and routing algorithms is followed by the architecture of parallel and distributed storage systems. The course goes on in message passing programming in standardized interface MPI. Consequently, techniques for parallel debugging and profiling are discussed. Last part of the course is devoted to the description of parallel programming patterns and case studies from the are of linear algebra, physical systems described by partial differential equations, N-Body systems and Monte-Carlo methods. 

Language of instruction

Czech

Number of ECTS credits

Mode of study

Not applicable.

Guarantor

doc. Ing. Jiří Jaroš, Ph.D.

Department

Department of Computer Systems (UPSY)

Learning outcomes of the course unit

Overview of principles of current parallel system design and of interconnection networks, communication techniques and algorithms. Survey of parallelization techniques of fundamental scientific problems, knowledge of parallel programming in MPI. Knowledge of basic parallel programming patterns. Practical experience with the work on supercomputers, ability to identify performance issues and propose their solution.
Knowledge of capabilities and limitations of parallel processing, ability to estimate performance of parallel applications. Language means for process/thread communication and synchronization. Competence in hardware-software platforms for high-performance computing and simulations.

Prerequisites

Von-Neumann computer architecture, computer memory hierarchy, cache memories and their organization, programming in assembly and in C/C++. Knowledge gained in courses PRL and AVS.

Co-requisites

Not applicable.

Planned learning activities and teaching methods

Not applicable.

Assesment methods and criteria linked to learning outcomes

Assessment of a project, 10 hours in total and a midterm examination.
Exam prerequisites:
To get 20 out of 40 points for projects and midterm examination.

Course curriculum

Not applicable.

Work placements

Not applicable.

Aims

To get familiar with the architecture of distributed supercomputing systems, their interconnection networks and storage. To orientate oneself in parallel systems on the market, be able to assess communication and computing possibilities of a particular architecture and to predict the performance of parallel applications. Learn how to write portable programs using standardized interfaces and languages, specify parallelism and process communication. To learn how to practically use supercoputer for solving complex engineering problems.

Specification of controlled education, way of implementation and compensation for absences

  • Missed labs can be substituted in alternative individual dates
  • There will be a place for missed labs in the last week of the semester.

Recommended optional programme components

Not applicable.

Prerequisites and corequisites

Not applicable.

Basic literature

Aktuální PPT prezentace přednášek v systému Moodle (CS)
Ananth Grama, Anshul Gupta, George Karypis, Vipin Kumar: Introduction to Parallel Computing, Addison-Wesley, 2003, 978-0201648652.Slides: download
Hennessy, J.L., Patterson, D.A.: Computer Architecture - A Quantitative Approach. 5. vydání, Morgan Kaufman Publishers, Inc., 2012, 1136 s., ISBN 1-55860-596-7.
Pacecho, P.: Introduction to Parallel Programming. Morgan Kaufman Publishers, 2011, 392 s., ISBN: 9780123742605 URL: download
Victor Eijkhout: Parallel Programming in MPI and OpenMP Full book: download web version: https://theartofhpc.com/pcse/
William Gropp, Ewing Lusk, Anthony Skjellum: Using MPI - 2nd Edition: Portable Parallel Programming with the Message Passing InterfaceUsing MPI - 2nd Edition: Portable Parallel Programming with the Message Passing Interface, MIT Press, 978-0262571326

Recommended reading

Alternativní kurz o paralelním programování http://www.cs.kent.edu/~jbaker/ParallelProg-Sp11/
MPI Tutoriál: http://mpitutorial.com/

Classification of course in study plans

  • Programme IT-MSC-2 Master's

    branch MGM , 0 year of study, summer semester, compulsory-optional
    branch MBI , 0 year of study, summer semester, compulsory-optional
    branch MBS , 0 year of study, summer semester, elective
    branch MIN , 0 year of study, summer semester, elective
    branch MIS , 0 year of study, summer semester, elective
    branch MMM , 0 year of study, summer semester, elective
    branch MPV , 1 year of study, summer semester, compulsory
    branch MSK , 1 year of study, summer semester, compulsory

  • Programme MITAI Master's

    specialization NISY , 0 year of study, summer semester, elective
    specialization NADE , 0 year of study, summer semester, elective
    specialization NBIO , 0 year of study, summer semester, compulsory
    specialization NCPS , 0 year of study, summer semester, elective
    specialization NEMB , 2 year of study, summer semester, compulsory
    specialization NHPC , 1 year of study, summer semester, compulsory
    specialization NGRI , 0 year of study, summer semester, elective
    specialization NIDE , 0 year of study, summer semester, elective
    specialization NISD , 0 year of study, summer semester, elective
    specialization NMAL , 0 year of study, summer semester, elective
    specialization NMAT , 0 year of study, summer semester, elective
    specialization NNET , 0 year of study, summer semester, elective
    specialization NSEC , 0 year of study, summer semester, elective
    specialization NSEN , 0 year of study, summer semester, elective
    specialization NSPE , 0 year of study, summer semester, elective
    specialization NVER , 0 year of study, summer semester, elective
    specialization NVIZ , 0 year of study, summer semester, elective

Type of course unit

 

Lecture

26 hod., optionally

Teacher / Lecturer

doc. Ing. Jiří Jaroš, Ph.D.

Syllabus

  1. Introduction to parallel processing.
  2. Parallel algorithm design and metodology.
  3. MPI: Point-to-point communications.
  4. MPI: Collective communications.
  5. MPI: Communicators and typologies. 
  6. MPI: Data types.
  7. MPI: One-sided communications.
  8. Distributed file systems and MPI-IO.
  9. Knihovny pro paralelní vstup a výstup.
  10. Performance analysis of parallel and distributed applications.
  11. Case studies (Matrix multiplications, BLAS).
  12. Cace studies (Jacobi methods, FDTD).
  13. Interconnection networks: topology and routing algorithms, switching, flow control.

Exercise in computer lab

16 hod., compulsory

Teacher / Lecturer

doc. Ing. Jiří Jaroš, Ph.D.
Ing. Kristián Kadlubiak

Syllabus

  1. MPI: Point-to-point communications
  2. MPI: Collective communications
  3. MPI: Communicators
  4. MPI: Data types
  5. MPI: One sided communications
  6. MPI: Parallel input and output with MPI-IO
  7. MPI: Parallel input and output with HDF5
  8. Profiling and tracing of parallel applications

Project

10 hod., compulsory

Teacher / Lecturer

doc. Ing. Jiří Jaroš, Ph.D.
Ing. Kristián Kadlubiak

Syllabus

  • A parallel program in MPI on the supercomputer.