Why single-node optimization and parallelization?¶

Objectives

Explain why Python applications may leave multicore node resources unused.
Distinguish reducing work, accelerating a kernel, and parallelizing independent work.
Define a performance goal that preserves correctness and fixes the useful workload.

Instructor note

15 min teaching
5 min discussion

Why the node matters¶

Contemporary HPC nodes contain multiple CPU cores, vector units, deep memory hierarchies, and native numerical libraries. A productive Python program can use only a fraction of that capability when its expensive work remains in interpreter-managed loops or when independent tasks run serially.

Single-node work is therefore the first scaling boundary. Improving it can shorten interactive analysis, reduce the number of nodes needed for a production run, and make later multi-node scaling more efficient.

Three different interventions¶

Performance problems that look similar can require different changes:

Reduce the work. Improve the algorithm, eliminate repeated computation, or call an existing optimized library.
Accelerate the kernel. Move a stable, loop-heavy region from Python execution into native code. This module uses Cython.
Execute work concurrently. Divide independent tasks or data chunks among the cores of one node. This module uses Dask.

Cython and Dask are complementary, not competing, when an application contains both an expensive kernel and many independent calls to it.

Performance is a measured requirement¶

A useful performance statement names:

the fixed useful workload;
an expected result and tolerance;
the target machine or allocation;
elapsed time, memory, throughput, or another relevant metric;
the baseline used for comparison.

A speedup without these facts is not reproducible. Faster code that changes the scientific result is incorrect.

Serial work still limits scaling¶

If a fraction \(s\) of an application remains serial, ideal speedup on \(p\) workers is bounded by

\[ S(p) \leq \frac{1}{s + (1-s)/p}. \]

Actual execution also pays for compilation, task construction, scheduling, communication, allocation, and synchronization. This is why the module begins with a single-node model and ends with measured configuration choices rather than a promise of linear speedup.

Discussion

Think of one Python workload from your work. Which intervention would you investigate first: less work, a faster kernel, or concurrent tasks? What evidence would you collect before changing it?

Keypoints

Single-node efficiency matters even when an application will later use multiple nodes.
Algorithmic improvement, kernel acceleration, and task parallelism solve different bottlenecks.
Correctness, workload, hardware, and baseline are part of every performance claim.