The Python single-node performance ecosystem¶

Episode 1 separated algorithmic improvement, kernel acceleration, and concurrent execution. Before choosing among them, we need to know who performs the expensive work—Python, a native library, or another process—and what that implies for sharing data and using CPU cores.

Objectives

Distinguish concurrency from parallel execution.
Explain how the GIL affects CPU-bound Python code.
Choose among threads, processes, native code, and Dask for a workload.

Instructor note

12 min teaching
8 min prediction and type-along
15 min exercises and discussion
Ask learners to predict both threaded results before running thread_gil_demo.py; treat timing differences as observations, not fixed expected values.

Modern nodes gain throughput mainly from multiple cores and vector units. Python does not automatically use those resources, and dividing work creates scheduling, communication, and memory costs. Start by identifying the unit of work and the resource that limits it.

Four useful models¶

Model	Memory	Good first fit	Main cost or risk
Python threads	Shared	Waiting for files, networks, or other I/O	The GIL limits parallel Python bytecode
Native-library threads	Shared	NumPy/BLAS or compiled kernels that release the GIL	Oversubscription and race conditions
Python processes	Separate	Independent CPU-bound Python tasks	Serialization, copying, and process startup
Dask local scheduling	Threads or processes	A graph of tasks or chunked array/table work	Scheduler overhead and poor chunking

Concurrency means multiple tasks can make progress. Parallelism means work is executing at the same instant on multiple resources. Threads can provide useful concurrency even when they cannot execute Python bytecode in parallel.

The Global Interpreter Lock¶

In the standard CPython runtime, a thread must hold the Global Interpreter Lock (GIL) while executing Python bytecode. Two threads in one process therefore do not normally accelerate a CPU-bound loop written in Python.

The important boundary is not “Python versus threads,” but who executes the hot work:

a Python loop typically holds the GIL;
I/O releases the thread while it waits;
many NumPy, compression, and other native kernels release the GIL;
a separate process has its own interpreter and GIL.

This is why a threaded Dask scheduler can accelerate a native array kernel but fail to accelerate the same loop expressed as Python objects.

Observe the boundary¶

The following type-along runs two independent tasks first in sequence and then with two threads. One task performs a pure-Python calculation; the other represents time spent waiting. Before running it, predict which threaded case can overlap effectively.

"""Compare serial and threaded execution for CPU work and waiting."""

from concurrent.futures import ThreadPoolExecutor
from time import perf_counter, sleep
from typing import Callable, TypeVar


Input = TypeVar("Input")
Result = TypeVar("Result")


def cpu_work(size: int) -> int:
    """Perform deterministic arithmetic in the Python interpreter."""
    return sum((value % 97) ** 2 for value in range(size))


def waiting_work(seconds: float) -> float:
    """Represent work that spends its time waiting rather than using the CPU."""
    sleep(seconds)
    return seconds


def run_serial(function: Callable[[Input], Result], inputs: list[Input]) -> list[Result]:
    """Run each independent input in the current thread."""
    return [function(value) for value in inputs]


def run_threads(
    function: Callable[[Input], Result], inputs: list[Input], workers: int = 2
) -> list[Result]:
    """Run independent inputs using a fixed-size thread pool."""
    with ThreadPoolExecutor(max_workers=workers) as pool:
        return list(pool.map(function, inputs))


def measure(label: str, function: Callable[[], object]) -> None:
    """Run a callable and print its elapsed wall-clock time."""
    start = perf_counter()
    function()
    elapsed = perf_counter() - start
    print(f"{label:<28} {elapsed:.3f} s")


if __name__ == "__main__":
    cpu_inputs = [2_000_000, 2_000_000]
    waiting_inputs = [0.25, 0.25]

    measure("Python work, serial", lambda: run_serial(cpu_work, cpu_inputs))
    measure("Python work, 2 threads", lambda: run_threads(cpu_work, cpu_inputs))
    measure("Waiting, serial", lambda: run_serial(waiting_work, waiting_inputs))
    measure("Waiting, 2 threads", lambda: run_threads(waiting_work, waiting_inputs))

Run it from the example directory:

$ cd content/episodes/code
$ python thread_gil_demo.py

The threaded waiting tasks should finish in roughly the time of one wait because neither thread needs the CPU while it sleeps. The pure-Python calculation normally does not become faster with threads under the standard CPython GIL and may become slightly slower. Exact timings vary, so the evidence to retain is the contrast between the two kinds of work rather than a particular speedup.

Discussion

Why would replacing cpu_work with a native function that releases the GIL change the prediction? Which measurement would you record before choosing threads for a real workload?

Runtime costs¶

The serial-work bound introduced in episode 1 is only the ideal limit. Real execution also adds task creation, scheduling, data transfer, allocation, and synchronization. A thousand microsecond tasks are often worse than a few tasks that each do substantial work. More workers can also make performance worse when each worker starts its own multithreaded BLAS kernel.

Control oversubscription

For predictable measurements, record worker counts and native thread settings such as OMP_NUM_THREADS, MKL_NUM_THREADS, or OPENBLAS_NUM_THREADS. A common starting point is one native thread per process.

Decision exercise¶

Exercise

Choose a first implementation for each workload and name one measurement that could change your decision.

Download 500 independent files.
Apply a NumPy universal function to four large array chunks.
Run a CPU-heavy pure-Python parser on 100 independent files.
Accelerate one deeply nested numerical loop called thousands of times.

Solution

Threads are a good first fit because the workload waits for I/O. Measure network/server saturation.
Threads or Dask’s threaded scheduler may work because NumPy commonly releases the GIL. Measure memory-bandwidth saturation and native-library threads.
Processes or Dask’s process/distributed scheduler avoid the single-process GIL. Measure serialization and task duration.
First improve the kernel with Cython (or an appropriate native implementation). Parallel scheduling cannot remove interpreter overhead inside each loop iteration.

Race-condition exercise¶

Exercise

Why is “threads share memory” both an advantage and a risk? Describe a safer shape for a parallel sum than having every worker update one shared total.

Solution

Sharing avoids serialization and copies, but unsynchronized read-modify-write operations can lose updates. Give each worker a private partial sum and combine the partials in a controlled reduction. This also reduces synchronization frequency.

Keypoints

Choose an execution model from the bottleneck and data-sharing needs.
The GIL limits simultaneous Python bytecode, not all native work called from Python.
Processes bypass the single-process GIL but add startup and data-transfer costs.
Parallel speedup is bounded by serial work and overhead; measure with realistic task sizes.