Choosing the Right Container
Every data pipeline starts with raw Python objects. Picking the wrong container costs both time and clarity. The four workhorses are:
- list — ordered, mutable, O(1) append, O(n) membership test.
- dict — key→value map, O(1) average lookup and insert (hash table). Python 3.7+ preserves insertion order.
- set — unordered unique values, O(1) membership, O(min(n,m)) intersection.
- tuple — immutable list; its immutability lets Python intern and hash it, making it suitable as a dict key or set element.
For lightweight records, prefer collections.namedtuple or dataclasses.dataclass over raw dicts when the schema is fixed — you get attribute access, repr, and optional type checking for free.
from dataclasses import dataclass, field
from typing import List
@dataclass
class Experiment:
name: str
scores: List[float] = field(default_factory=list)
@property
def mean_score(self) -> float:
return sum(self.scores) / len(self.scores) if self.scores else float('nan')
expt = Experiment(name="baseline", scores=[0.82, 0.79, 0.85])
print(expt.mean_score) # 0.82Comprehensions and Generator Expressions
Comprehensions are not merely syntactic sugar — they are compiled to optimised bytecode (LIST<em>APPEND / MAP</em>ADD) that outperforms equivalent append loops by 15–30 % in CPython benchmarks.
# List comprehension: square every odd number
squares = [x**2 for x in range(1_000_000) if x % 2 == 1]
# Dict comprehension: invert a mapping
orig = {'a': 1, 'b': 2, 'c': 3}
inverted = {v: k for k, v in orig.items()}
# Set comprehension: unique domains from email list
emails = ['alice@example.com', 'bob@corp.io', 'carol@example.com']
domains = {e.split('@')[1] for e in emails}When you only need to iterate over a result rather than store it, replace the square brackets with parentheses to get a generator expression. Generators are lazy: they yield one item at a time and hold O(1) memory regardless of the logical size of the stream.
import csv, pathlib
def parse_prices(path: str):
"""Generator: yields (date_str, float) tuples from a CSV."""
with open(path) as fh:
reader = csv.DictReader(fh)
yield from ((row['date'], float(row['price'])) for row in reader)
total = sum(price for _, price in parse_prices('prices.csv'))Pitfall: Comprehension Variable Leakage
In Python 3, comprehension variables are properly scoped — x in [x for x in range(5)] does not leak into the enclosing scope. This is the correct behavior; do not rely on the old Python 2 leak.
collections Utilities
The collections module provides several containers that replace verbose patterns:
Counter— multiset, replaces{k: counts.get(k,0)+1 for k in items}.defaultdict— eliminatessetdefaultboilerplate in grouping loops.deque— O(1) popleft for sliding-window algorithms; lists are O(n).ChainMap— a live view over multiple dicts, useful for layered configs.
from collections import Counter, defaultdict
words = 'the quick brown fox jumps over the lazy dog'.split()
word_freq = Counter(words)
print(word_freq.most_common(3)) # [('the', 2), ...]
# Group words by first letter
by_letter = defaultdict(list)
for w in words:
by_letter[w[0]].append(w)itertools and functools for Data Pipelines
Built-in higher-order tools compose cleanly with generators to build memory-efficient pipelines:
itertools.islice(it, n)— take first n items without materialising the rest.itertools.groupby(sorted<em>iter, key)— streaming group-by (input must be sorted).functools.reduce— fold a sequence; rarely needed sincesum/maxcover common cases.functools.lru</em>cache— memoize pure functions; critical for recursive feature engineering.
import itertools, functools
# Running product via reduce
from operator import mul
factorial = lambda n: functools.reduce(mul, range(1, n+1), 1)
# Sliding window of size k
def windows(iterable, k):
it = iter(iterable)
win = list(itertools.islice(it, k))
if len(win) == k:
yield tuple(win)
for item in it:
win.append(item)
win.pop(0)
yield tuple(win)