Idiomatic and Performant Python: Writing It Well, Then Measuring Before You Optimise

Iterate over the thing, not its indices

names = ["ada", "grace", "alan"]

# not idiomatic — simulates a C loop
for i in range(len(names)):
    print(i, names[i].title())

# idiomatic — the loop variable is the useful thing itself
for i, name in enumerate(names):
    print(i, name.title())

enumerate produces (index, value) pairs lazily, without a separate len() call or a manual counter. The same shift applies to iterating two sequences together:

scores = [98, 95, 88]

# not idiomatic
for i in range(len(names)):
    print(names[i], scores[i])

# idiomatic
for name, score in zip(names, scores):
    print(name, score)

zip stops at the shorter input rather than raising, which is usually what you want; itertools.zip_longest is there for the other case. Unpacking replaces manual indexing again when a function returns a fixed-size tuple:

def min_max(xs):
    return min(xs), max(xs)

lo, hi = min_max(scores)   # not: result = min_max(scores); lo = result[0]; hi = result[1]
print(lo, hi)   # -> 88 98

first, *rest = names
print(first, rest)   # -> ada ['grace', 'alan']

Truthiness and with

items = []

if len(items) == 0:      # works, but says more than necessary
    print("empty")

if not items:             # idiomatic — an empty container is already falsy
    print("empty")

bool() of None, 0, 0.0, "", [], {}, set(), and any object whose __len__ returns 0 is False; everything else, including a list containing only False, is truthy. This is what makes while queue: a correct loop condition and if response: a correct way to check a non-empty string, without a len() or an explicit is not None. with, covered in files and context managers, is idiomatic for the same reason: it says “acquire, use, release” directly instead of a try/finally that says the same thing in more words.

The performance model: why loops are slow

A CPython for loop executes bytecode one instruction at a time, and each BINARY_OP re-checks the runtime types of its operands before deciding what + even means for them — there is no compile-time specialisation the way there is in C. That per-iteration overhead is fixed regardless of what the loop body computes, which is why the fix for a slow loop is usually to run fewer, larger operations rather than to hand-optimise the loop body.

# quadratic: each += allocates a new, longer string and copies the old one in
result = ""
for word in ["a"] * 10_000:
    result += word + " "

# linear: join allocates once, having measured the total length up front
result = " ".join(["a"] * 10_000)

str is immutable, so result += word does not extend the existing string — it builds an entirely new one and copies everything before it, every single time. Over \(n\) iterations that is \(O(n^2)\) total copying. str.join computes the final length once and allocates once.

import time

def local_lookup():
    total = 0
    x = 5              # local: stored in a fast array slot on the frame
    for _ in range(5_000_000):
        total += x
    return total

x = 5   # global: looked up by name in a dict each time

def global_lookup():
    total = 0
    for _ in range(5_000_000):
        total += x
    return total

local_lookup is reliably faster than global_lookup on CPython — a local variable is a slot in an array indexed by position, resolved at compile time; a global is a dictionary lookup by name, resolved every time the line runs. This is also why a tight loop inside a function is faster than the same loop at module level, and why def f(x, n=len) binding a global as a default argument is a real, if minor, optimisation you will see in hot code.

When NumPy replaces the loop entirely

import numpy as np

xs = list(range(1_000_000))
ys = [v * 2 + 1 for v in xs]        # a Python-level loop, one BINARY_OP pair per element

arr = np.arange(1_000_000)
arr2 = arr * 2 + 1                  # one C-level loop over contiguous memory, no per-element bytecode

The list comprehension is still a Python loop — faster than the explicit for form because the append is implicit and the loop body is one bytecode-level construct, but it still pays per-element interpreter overhead. arr * 2 + 1 does the entire operation inside NumPy’s C implementation over a contiguous block of memory, with no per-element Python bytecode at all. The rule of thumb: once you are doing uniform numeric work over more than a few thousand elements, express it as array operations rather than a Python loop, and let NumPy own the loop.

Profile before you optimise

import timeit

t1 = timeit.timeit('"".join(str(i) for i in range(1000))', number=1000)
t2 = timeit.timeit('"-".join(map(str, range(1000)))', number=1000)
print(t1, t2)   # -> two float seconds; compare them, don't guess which is faster

timeit isolates one expression and runs it many times, which is right for comparing two small alternatives. For “where is the actual time going in my program”, cProfile is the tool:

$ python -m cProfile -s cumulative slow_script.py

That prints every function called, how many times, and how much time it and its callees consumed — usually pointing at one function responsible for most of the runtime, which is where effort belongs. Optimising a function that consumes 2% of total runtime, however cleverly, cannot move the total by more than 2%.

Common mistakes and their fixes

The mutable-default-argument row deserves its own emphasis: a default value is evaluated once, when the def statement runs, and the same object is reused on every call that does not supply that argument.

def add_item(item, basket=[]):   # the list literal runs once, not per call
    basket.append(item)
    return basket

print(add_item("apple"))   # -> ['apple']
print(add_item("bread"))   # -> ['apple', 'bread']   — the SAME list, carried over

The second call did not start from an empty basket — it kept appending to the list created when add_item was defined. The fix in the table (None as the sentinel, a fresh list built inside the function) is the standard idiom, and it applies to any mutable default: lists, dicts, sets, and custom mutable objects alike.

This closes the book. Start from the overview if you arrived here mid-way, or revisit errors and exceptions and files and context managers for the two topics every real program touches first.

References

1. Python documentation. Performance Tips — Python Wiki.
2. Python documentation. timeit — Measure execution time of small code snippets.
3. Python documentation. profile and cProfile — Python profilers.
4. Python documentation. numpy.ndarray operations — NumPy project documentation.
5. Python documentation. Truth Value Testing and zip, enumerate built-ins.
6. van Rossum, G., Warsaw, B., & Coghlan, N. PEP 8 — Style Guide for Python Code, 2001.