Single-Node Parallelization and Optimization of Python for HPC

Overview of Modular Lesson Material

Python is productive, but a performance-critical loop and a collection of independent tasks need different remedies. Modern HPC nodes provide many CPU cores and optimized native execution resources; effective Python applications must cross the interpreter boundary or expose work at a useful parallel granularity when measurement shows that these changes are warranted.

This module follows a decision-oriented progression. It first establishes why single-node efficiency matters, surveys Python execution choices, and teaches learners to recognize which problems fit native-code acceleration or task parallelism. Learners then compile and type a numerical kernel with Cython, schedule independent work with Dask on one node, and combine the techniques in a reproducible optimization report.

The goal is not “make everything parallel.” It is to choose an optimization whose execution model, overhead, correctness risk, and maintenance cost fit the measured bottleneck.

Prerequisites

  • Comfortable writing Python functions, loops, and imports.

  • Basic familiarity with NumPy arrays and timing code.

  • A computer or compute node with at least two CPU cores and 4 GB of RAM.

  • A C compiler and the software listed in the setup episode.

No prior C, Cython, Dask, MPI, or cluster-scheduler experience is required.

Table of Contents

Setup

Episodes

Instructor note

The canonical episodes are maintained in MyST Markdown. The structural template demonstrates optional parallel notebook copies, but this module keeps one source form to prevent lesson and exercise drift. Instructors may generate notebooks with Jupytext when a notebook delivery format is required.

Reference

Learning Outcomes

Estimated commitment: 4 hours, including type-alongs, exercises, discussion, and the knowledge check.

This modular lesson material is intended for Python developers, computational scientists, and research software engineers who need to improve performance on one multicore HPC node before considering distributed scaling.

By the end of this module, learners should be able to:

  • explain when native-code integration is suitable for accelerating a Python HPC application;

  • apply Cython types and compilation to improve a measured Python bottleneck;

  • explain how the GIL affects threads and why native libraries can behave differently;

  • use Dask for single-node parallel work and choose between thread and process execution;

  • select task and chunk sizes that balance useful work against scheduling overhead;

  • check correctness and interpret measurements before claiming a speedup.

In the EVITA CQF this is module PP.PJC1-SD1.2.9.4 — Single-node parallelization & optimization, part of PP.PJC1 — Python for HPC. Its related HPC Certification Forum skills are:

Credits and Licenses

Credit

This module adapts material from the EVITA courses Python for HPC, Parallel Computing using Python, and Writing Performant Python. Its repository, course, and material structure follows Introduction to GPU Programming. Exact source revisions are recorded in the repository README.md.

See additional EVITA modular lesson materials for related HPC topics.

License

Pedagogical material and media are available under Creative Commons Attribution-ShareAlike 4.0. Source code and code snippets are available under the MIT License. See LICENSE and LICENSE.code in the repository root for the complete terms.