Memory and Concurrency in Python: References, the GIL, and Why Data Loaders Fork

Stack and heap

The stack stores one frame per active call: arguments, locals, return address. It grows and shrinks in LIFO order, allocation is one pointer bump, and space is reclaimed the instant a function returns. It is also small — a few megabytes by default — which is why unbounded recursion crashes rather than swaps. The heap stores objects whose lifetime is not tied to a call: anything returned, stored in a container, or shared.

Everything is a reference

In Python a name is a binding to a heap object; assignment binds a name, it never copies. Passing an argument binds the parameter to the same object, so whether the caller sees a change depends on whether the callee mutates the object or rebinds the name.

def mutate(xs):
    xs.append(4)        # mutates the caller's list

def rebind(xs):
    xs = [9, 9, 9]      # rebinds a local name; caller unaffected

a = [1, 2, 3]
mutate(a); print(a)     # [1, 2, 3, 4]
rebind(a); print(a)     # [1, 2, 3, 4]  -- unchanged

b = a                   # NOT a copy: two names, one object
b.append(5); print(a)   # [1, 2, 3, 4, 5]
print(a is b)           # True   -- same object
c = a[:]                # shallow copy
print(a is c, a == c)   # False True

grid = [[0] * 2] * 3    # three references to ONE inner list
grid[0][0] = 1
print(grid)             # [[1, 0], [1, 0], [1, 0]]
grid = [[0] * 2 for _ in range(3)]   # three distinct lists

The [[0]*2]*3 case is the same defect as a mutable default argument: one object reached through several names. Use copy.deepcopy when nesting matters.

How objects are freed

CPython counts references. Binding a name increments an object’s count, rebinding or leaving scope decrements it, and at zero the object is freed immediately — which is why del frees a large tensor promptly.

Reference counting cannot free cycles: if a.ref = b and b.ref = a, both counts stay at one after the last external reference disappears. CPython therefore also runs a generational cyclic collector over container objects, collecting young generations more often on the observation that most objects die young; the gc module exposes it. Runtimes without refcounting (the JVM, Go) trace only, so destruction there is not prompt.

Processes and threads

A process has its own address space: isolation by default, crashes contained, communication only through pipes, sockets or shared memory, expensive start-up. A thread lives inside a process and shares its heap: cheap to create and to communicate with, and every shared mutable object is now a hazard.

In CPython, threads carry an extra constraint. The global interpreter lock allows only one thread to execute Python bytecode at a time. A thread releases it when it blocks on I/O, and C extensions may release it around long computations — NumPy, PyTorch and BLAS do. So:

• I/O-bound work (HTTP requests, file and socket reads, database calls): threads help, because the lock is free while a thread waits.
• CPU-bound pure-Python work (parsing, tokenising, augmentation in Python loops): threads do not help. The work serialises and you pay switching overhead on top.

Python 3.13 ships an optional free-threaded build (PEP 703) that removes the GIL, but it is opt-in and extensions must be adapted, so the reasoning above still governs ordinary deployments.

Races, locks, deadlock

Shared mutable state plus concurrency gives race conditions: an outcome that depends on scheduling. counter += 1 is not atomic — it loads, adds and stores, and a switch between load and store loses an update. The GIL does not protect you: it serialises bytecodes, not your logical operations, and CPython switches threads periodically (sys.setswitchinterval).

A Lock makes a critical section mutually exclusive. Two locks acquired in different orders give deadlock, which needs four conditions at once (Coffman): mutual exclusion, hold-and-wait, no pre-emption, circular wait. Break the last with a global lock ordering, or avoid sharing.

Why data loaders use processes

Loading and augmenting data is CPU-bound Python: decoding JPEGs, tokenising text, building tensors. Under the GIL that would serialise against the training loop, so PyTorch’s DataLoader with num_workers > 0 starts worker processes, each with its own interpreter and lock, shipping batches back through shared memory rather than copying tensors over a pipe.

The costs are the ones processes always have. Workers start with fork (copy-on-write, unsafe alongside threads) or spawn (a fresh interpreter that re-imports your module — hence the if __name__ == "__main__" guard). Copy-on-write saves less than expected, because touching an object’s reference count writes to its page and forces a copy; keep bulk data in arrays or tensors rather than lists of Python objects.

References

1. Python Software Foundation. Global Interpreter Lock and threading.
2. Python Software Foundation. PEP 703 — Making the Global Interpreter Lock Optional in CPython.
3. Python Software Foundation. gc — Garbage Collector interface and multiprocessing (start methods: fork, spawn, forkserver).
4. PyTorch. torch.utils.data — DataLoader and multi-process data loading.
5. Coffman, E. G., Elphick, M., & Shoshani, A. System deadlocks. ACM Computing Surveys 3(2), 67–78, 1971.
6. Jones, R., Hosking, A., & Moss, E. The Garbage Collection Handbook, 2nd ed. CRC Press, 2023.