Reference for learners

Selection guide

Workload evidence

Start with

Existing optimized NumPy/SciPy operation

Use and benchmark that operation

Stable Python numerical loop dominates

Cython

I/O-bound independent tasks

Threads or Dask threads

CPU-bound pure-Python independent tasks

Processes or Dask processes

Independent native kernels release the GIL

Threads or Dask threads

Chunked array or table workflow on one node

Dask Array or DataFrame

Measurement checklist

  • State the fixed workload and correctness tolerance.

  • Record hardware/allocation, package versions, scheduler, worker count, task/chunk size, and native thread settings.

  • Include a serial baseline and repeat timings.

  • Separate compilation or startup time from steady-state timing when appropriate.

  • Report elapsed time and speedup, not only a percentage improvement.

  • Note memory use and variability when they influence the decision.

Glossary

chunk : A block of an array or table partition processed as a unit of work.

compiled extension : Native machine code exposed as a module importable by Python.

concurrency : Multiple tasks making progress during overlapping periods; they need not execute simultaneously.

Dask graph : A representation of tasks and the dependencies among them.

GIL : The Global Interpreter Lock used by standard CPython to coordinate execution of Python bytecode within one interpreter.

lazy execution : Recording operations for later execution rather than computing immediately.

native code : Machine code produced by a compiler, including Cython extensions and numerical-library kernels.

oversubscription : Creating more runnable workers or native threads than the allocated hardware can execute effectively.

parallelism : Work executing simultaneously on multiple resources.

partition : A DataFrame or Bag subdivision handled by Dask as one or more tasks.

scheduler : The component that selects ready tasks and assigns them to execution resources.

speedup : Baseline elapsed time divided by optimized elapsed time for the same useful work.

task : A schedulable unit of work with defined inputs and outputs.

typed memoryview : Cython’s typed, low-overhead view of an object exposing the Python buffer protocol.

Further learning