Beginner Intermediate 24 min read

Chapter 42: Intermediate SQL: Joins & Aggregation

SQL's power as an analytical tool becomes fully apparent once you move beyond single-table queries. Real data lives across many related tables — orders reference customers, line items reference products, events reference users — and extracting meaning requires weaving those tables together and summarizing the results. This chapter builds the core toolkit that separates a SQL beginner from a working data engineer or analyst: the full family of join operations, the GROUP BY aggregation pipeline, and the expressive power of subqueries, set operations, and CASE expressions.

We work throughout with a realistic e-commerce schema comprising five tables: customers, orders, order_items, products, and categories. This single schema is rich enough to motivate every concept — from the simplest INNER JOIN that pairs orders with their customers, to correlated subqueries that compare each customer's spending against the overall average, to CASE expressions that bucket revenue into cohorts. Every query in the chapter runs against this schema so you can follow the logic end-to-end.

By the end you will be able to write the kinds of multi-table analytical queries that power real dashboards, data pipelines, and ad-hoc investigations. You will also understand why SQL behaves as it does — the relational algebra that underpins joins, the two-phase nature of HAVING versus WHERE, and the performance implications of correlated subqueries — giving you the mental model to debug unexpected results and optimize slow queries.

Runnable companion notebook for this chapter. Download the Jupyter notebook (.ipynb)
Libraries covered: PandasPostgreSQL

Learning Objectives

  • Construct INNER, LEFT, RIGHT, FULL OUTER, CROSS, and self joins and predict their output row counts for given data.
  • Apply GROUP BY with aggregate functions (COUNT, SUM, AVG, MIN, MAX) to summarize data at any granularity.
  • Distinguish WHERE from HAVING and apply each correctly in multi-step aggregation queries.
  • Write scalar and correlated subqueries and understand when to prefer them over joins or CTEs.
  • Combine result sets with UNION, UNION ALL, INTERSECT, and EXCEPT and apply them to practical reporting problems.
  • Use CASE expressions for conditional logic, pivoting, and cohort bucketing inside SELECT and aggregate contexts.
  • Design and execute multi-join analytical queries against a normalized relational schema.

42.1 The Schema and Join Fundamentals Beginner

The Working Schema

All examples in this chapter use the following five-table e-commerce schema. Understanding its structure before writing any SQL pays dividends throughout.

SQL
CREATE TABLE customers (
    customer_id  SERIAL PRIMARY KEY,
    name         TEXT NOT NULL,
    email        TEXT UNIQUE NOT NULL,
    country      TEXT,
    joined_at    DATE
);

CREATE TABLE categories (
    category_id  SERIAL PRIMARY KEY,
    name         TEXT NOT NULL
);

CREATE TABLE products (
    product_id   SERIAL PRIMARY KEY,
    name         TEXT NOT NULL,
    price        NUMERIC(10,2) NOT NULL,
    category_id  INT REFERENCES categories(category_id)
);

CREATE TABLE orders (
    order_id     SERIAL PRIMARY KEY,
    customer_id  INT REFERENCES customers(customer_id),
    placed_at    TIMESTAMPTZ NOT NULL,
    status       TEXT  -- 'pending', 'shipped', 'cancelled'
);

CREATE TABLE order_items (
    item_id      SERIAL PRIMARY KEY,
    order_id     INT REFERENCES orders(order_id),
    product_id   INT REFERENCES products(product_id),
    quantity     INT NOT NULL,
    unit_price   NUMERIC(10,2) NOT NULL
);

The foreign-key graph is: order<em>itemsorderscustomers, and order</em>itemsproductscategories. This is a classic star-like normalized schema.

What a Join Actually Does

A join combines rows from two tables by evaluating a join predicate — a Boolean condition on the columns of both tables. The relational algebra operation is called a theta join; the overwhelmingly common special case where the predicate tests equality of a column from each table is the equi-join.

Conceptually, the database engine first forms the Cartesian product of both tables (every row paired with every row), then filters by the predicate. In practice, query optimizers avoid the full Cartesian product by using hash joins, merge joins, or nested-loop index seeks, but the logical result is identical.

The cardinality of a join result depends on how many rows from the left table match rows from the right table:

  • A 1-to-1 join (primary key ↔ primary key) returns at most as many rows as the smaller table.
  • A many-to-1 join (foreign key ↔ primary key) preserves the row count of the many side.
  • A many-to-many join (no unique constraint on either join key) can multiply row counts and is a common source of unintended row duplication.
  • INNER JOIN — the Default

An INNER JOIN returns only rows where the predicate is satisfied on both sides. Rows with no match are silently discarded.

SQL
-- Every order paired with its customer's name
SELECT o.order_id,
       c.name        AS customer_name,
       o.placed_at,
       o.status
FROM   orders o
INNER JOIN customers c ON c.customer_id = o.customer_id
ORDER  BY o.placed_at DESC
LIMIT  10;

Because orders.customer<em>id is a NOT NULL foreign key to customers, every order has exactly one matching customer row, so no orders are dropped. If the foreign key allowed NULLs, orders with customer</em>id IS NULL would vanish from the result — a subtle bug that INNER JOIN silently introduces.

Alias convention. Table aliases (o, c) are essential in multi-table queries. Use short, memorable aliases that reflect the table name. Always qualify ambiguous column names.

LEFT / RIGHT / FULL OUTER JOIN — Preserving Unmatched Rows

Outer joins extend the inner join result by preserving rows from one or both tables even when no match exists, padding the missing side with NULLs.

SQL
-- All customers, with their order count (0 for customers who never ordered)
SELECT c.customer_id,
       c.name,
       COUNT(o.order_id) AS order_count
FROM   customers c
LEFT JOIN orders o ON o.customer_id = c.customer_id
GROUP BY c.customer_id, c.name
ORDER BY order_count DESC;

Here the LEFT JOIN ensures every customer appears in the result. Customers with no orders will have o.order<em>id = NULL, so COUNT(o.order</em>id) returns 0 (COUNT ignores NULLs). Using COUNT(<em>) instead would incorrectly return 1 for those customers.

A RIGHT JOIN is the mirror image — all rows from the right table are preserved. In practice, most practitioners convert RIGHT JOINs to LEFT JOINs by swapping the table order, improving readability.

A FULL OUTER JOIN preserves unmatched rows from both* sides. It is useful for reconciliation queries — finding rows present in one source but missing from another:

SQL
-- Reconcile two inventory snapshots: find products in either but not both
SELECT COALESCE(a.product_id, b.product_id) AS product_id,
       a.quantity AS qty_snapshot_a,
       b.quantity AS qty_snapshot_b
FROM   inventory_snapshot_a a
FULL OUTER JOIN inventory_snapshot_b b
       ON a.product_id = b.product_id
WHERE  a.product_id IS NULL OR b.product_id IS NULL;

CROSS JOIN — Generating All Combinations

A CROSS JOIN produces the full Cartesian product with no join predicate. Its output has $|A| \times |B|$ rows. Practical uses include generating test data, date-dimension spine tables, or all (customer, product) pairs for recommendation scoring:

SQL
-- Every customer paired with every product (for a recommendation scaffold)
SELECT c.customer_id, p.product_id
FROM   customers c
CROSS JOIN products p;

Warn yourself before running CROSS JOINs on large tables — 10 000 customers × 50 000 products yields 500 million rows.

Self Join — Relating a Table to Itself

A self join joins a table to itself using two aliases. The canonical example is an employee-manager hierarchy, but in our schema we can find customers in the same country:

SQL
-- Pairs of customers who share the same country (excluding self-pairs)
SELECT a.name AS customer_a,
       b.name AS customer_b,
       a.country
FROM   customers a
JOIN   customers b
       ON  a.country = b.country
       AND a.customer_id < b.customer_id;  -- avoid (A,B) and (B,A) duplicates

The a.customer<em>id &lt; b.customer</em>id trick is idiomatic for generating unordered pairs without duplicates.

Code Examples

Schema setup and seed data (SQLite-compatible)

Creates and populates the chapter schema in SQLite for local experimentation.

SQL
-- Run in SQLite or any ANSI-SQL database
CREATE TABLE IF NOT EXISTS customers (
    customer_id INTEGER PRIMARY KEY,
    name        TEXT NOT NULL,
    email       TEXT UNIQUE NOT NULL,
    country     TEXT,
    joined_at   TEXT  -- ISO-8601 date string in SQLite
);

CREATE TABLE IF NOT EXISTS categories (
    category_id INTEGER PRIMARY KEY,
    name        TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS products (
    product_id  INTEGER PRIMARY KEY,
    name        TEXT NOT NULL,
    price       REAL NOT NULL,
    category_id INTEGER REFERENCES categories(category_id)
);

CREATE TABLE IF NOT EXISTS orders (
    order_id    INTEGER PRIMARY KEY,
    customer_id INTEGER REFERENCES customers(customer_id),
    placed_at   TEXT NOT NULL,
    status      TEXT
);

CREATE TABLE IF NOT EXISTS order_items (
    item_id     INTEGER PRIMARY KEY,
    order_id    INTEGER REFERENCES orders(order_id),
    product_id  INTEGER REFERENCES products(product_id),
    quantity    INTEGER NOT NULL,
    unit_price  REAL NOT NULL
);

-- Seed data
INSERT INTO customers VALUES
  (1,'Alice','alice@example.com','US','2023-01-15'),
  (2,'Bob',  'bob@example.com',  'CA','2023-03-22'),
  (3,'Carol','carol@example.com','US','2023-06-01'),
  (4,'Dave', 'dave@example.com', 'UK','2024-01-10'),
  (5,'Eve',  'eve@example.com',  'CA','2024-02-28');

INSERT INTO categories VALUES (1,'Electronics'),(2,'Books'),(3,'Apparel');

INSERT INTO products VALUES
  (1,'Laptop',   999.99, 1),
  (2,'Headphones',79.99, 1),
  (3,'SQL Textbook',49.99,2),
  (4,'T-Shirt',  19.99, 3),
  (5,'Tablet',  499.99, 1);

INSERT INTO orders VALUES
  (1,1,'2024-03-01','shipped'),
  (2,1,'2024-04-15','shipped'),
  (3,2,'2024-04-20','pending'),
  (4,3,'2024-05-01','cancelled'),
  (5,4,'2024-05-10','shipped');

INSERT INTO order_items VALUES
  (1,1,1,1,999.99),
  (2,1,3,2,49.99),
  (3,2,2,1,79.99),
  (4,3,4,3,19.99),
  (5,4,5,1,499.99),
  (6,5,2,2,79.99);
Output
-- No output; tables created and populated.

Running the schema in Python with sqlite3

Executes the schema creation and a join query entirely in Python using the built-in sqlite3 module.

PYTHON
import sqlite3
import textwrap

con = sqlite3.connect(":memory:")
con.row_factory = sqlite3.Row
cur = con.cursor()

# (Assume DDL and seed INSERTs from the SQL example above have been executed)
# ... [run DDL here] ...

cur.execute(textwrap.dedent("""
    SELECT o.order_id,
           c.name        AS customer_name,
           o.placed_at,
           o.status
    FROM   orders o
    JOIN   customers c ON c.customer_id = o.customer_id
    ORDER  BY o.placed_at
"""))

for row in cur.fetchall():
    print(dict(row))

con.close()
Output
{'order_id': 1, 'customer_name': 'Alice', 'placed_at': '2024-03-01', 'status': 'shipped'}
{'order_id': 2, 'customer_name': 'Alice', 'placed_at': '2024-04-15', 'status': 'shipped'}
{'order_id': 3, 'customer_name': 'Bob',   'placed_at': '2024-04-20', 'status': 'pending'}
{'order_id': 4, 'customer_name': 'Carol', 'placed_at': '2024-05-01', 'status': 'cancelled'}
{'order_id': 5, 'customer_name': 'Dave',  'placed_at': '2024-05-10', 'status': 'shipped'}

42.2 GROUP BY and Aggregate Functions Beginner

The Aggregation Pipeline

SQL query execution follows a logical order that every practitioner must internalize:

$$\text{FROM} \rightarrow \text{JOIN} \rightarrow \text{WHERE} \rightarrow \text{GROUP BY} \rightarrow \text{HAVING} \rightarrow \text{SELECT} \rightarrow \text{ORDER BY} \rightarrow \text{LIMIT}$$

GROUP BY partitions the rows that survive the WHERE filter into groups, one group per unique combination of the grouping columns. Aggregate functions then reduce each group to a single scalar value. The result set has one row per group.

Core Aggregate Functions

SQL provides five standard aggregate functions:

  • COUNT(expr) — count of non-NULL values; COUNT(<em>) counts all rows including NULLs.
  • SUM(expr) — arithmetic sum; returns NULL for an empty group.
  • AVG(expr) — arithmetic mean; NULLs are excluded from both numerator and denominator.
  • MIN(expr) / MAX(expr) — work on numbers, strings, and dates.

All five ignore NULL values in their input (except COUNT(</em>)). This is often surprising: AVG(discount) computes the average only among rows where discount is not NULL, not among all rows.

SQL
-- Revenue, order count, and average order value per customer
SELECT c.customer_id,
       c.name,
       COUNT(DISTINCT o.order_id)                     AS order_count,
       SUM(oi.quantity * oi.unit_price)               AS total_revenue,
       AVG(oi.quantity * oi.unit_price)               AS avg_item_revenue,
       MIN(o.placed_at)                               AS first_order,
       MAX(o.placed_at)                               AS last_order
FROM   customers c
LEFT JOIN orders o      ON o.customer_id = c.customer_id
LEFT JOIN order_items oi ON oi.order_id  = o.order_id
GROUP BY c.customer_id, c.name
ORDER BY total_revenue DESC NULLS LAST;

COUNT() vs COUNT(col) vs COUNT(DISTINCT col)

These three forms answer different questions:

  • COUNT(</em>) — how many rows are in this group?
  • COUNT(col) — how many rows have a non-NULL value for col?
  • COUNT(DISTINCT col) — how many distinct non-NULL values does col take?

In the query above, COUNT(DISTINCT o.order<em>id) is necessary because the JOIN with order</em>items produces one row per line item, so a single order with three items contributes three rows. Without DISTINCT, the count would reflect line items, not orders.

Grouping by Multiple Columns

You can group by any combination of columns. Every column in SELECT that is not inside an aggregate must appear in GROUP BY (this is enforced by standard SQL and PostgreSQL; MySQL historically allowed violations).

SQL
-- Revenue by country and order status
SELECT c.country,
       o.status,
       COUNT(DISTINCT o.order_id)       AS num_orders,
       SUM(oi.quantity * oi.unit_price) AS gross_revenue
FROM   customers c
JOIN   orders o       ON o.customer_id  = c.customer_id
JOIN   order_items oi ON oi.order_id    = o.order_id
GROUP BY c.country, o.status
ORDER BY c.country, o.status;

This query produces one row for every (country, status) pair that actually appears in the data — it does not generate rows for combinations with zero orders.

HAVING — Filtering on Aggregate Results

WHERE filters individual rows before grouping. HAVING filters groups after aggregation. Confusing the two is one of the most common SQL mistakes.

SQL
-- Customers with more than one shipped order
SELECT c.customer_id,
       c.name,
       COUNT(o.order_id) AS shipped_count
FROM   customers c
JOIN   orders o ON o.customer_id = c.customer_id
WHERE  o.status = 'shipped'         -- filter rows before grouping
GROUP BY c.customer_id, c.name
HAVING COUNT(o.order_id) > 1        -- filter groups after aggregation
ORDER BY shipped_count DESC;

Note that the WHERE clause reduces the input to shipped orders only, then GROUP BY groups by customer, then HAVING discards customers with only one shipped order.

Why not put everything in HAVING? Because WHERE is applied before GROUP BY, it dramatically reduces the number of rows the aggregation engine must process. Placing filterable predicates in HAVING forces the engine to aggregate all rows before discarding most of them — a significant performance cost on large tables.

Aggregating Expressions

Aggregate functions accept arbitrary expressions, not just column references:

SQL
-- Category-level revenue analytics
SELECT cat.name                                      AS category,
       COUNT(DISTINCT p.product_id)                  AS product_count,
       SUM(oi.quantity * oi.unit_price)              AS total_revenue,
       SUM(oi.quantity * oi.unit_price)
           / NULLIF(COUNT(DISTINCT o.order_id), 0)  AS rev_per_order
FROM   categories cat
JOIN   products p    ON p.category_id  = cat.category_id
JOIN   order_items oi ON oi.product_id = p.product_id
JOIN   orders o       ON o.order_id    = oi.order_id
WHERE  o.status <> 'cancelled'
GROUP BY cat.category_id, cat.name
ORDER BY total_revenue DESC;

NULLIF(expr, 0) guards against division-by-zero — it returns NULL when the denominator is 0, which propagates NULL through the division rather than raising an error.

GROUP BY ROLLUP and CUBE (Advanced Extensions)

PostgreSQL and most analytical databases support ROLLUP and CUBE modifiers that generate subtotal rows automatically:

SQL
-- Revenue with subtotals per country and a grand total
SELECT COALESCE(c.country, 'ALL') AS country,
       COALESCE(o.status, 'ALL')  AS status,
       SUM(oi.quantity * oi.unit_price) AS revenue
FROM   customers c
JOIN   orders o       ON o.customer_id = c.customer_id
JOIN   order_items oi ON oi.order_id   = o.order_id
GROUP BY ROLLUP(c.country, o.status)
ORDER BY c.country NULLS LAST, o.status NULLS LAST;

ROLLUP generates subtotals for each prefix of the grouping list: (country, status), (country), and (). CUBE generates subtotals for every subset.

Code Examples

Top-N customers by lifetime value

Computes lifetime value and ranks the top 3 customers, filtering out cancelled orders.

SQL
SELECT c.name,
       ROUND(SUM(oi.quantity * oi.unit_price), 2) AS ltv,
       COUNT(DISTINCT o.order_id)                 AS orders
FROM   customers c
JOIN   orders o       ON o.customer_id  = c.customer_id
JOIN   order_items oi ON oi.order_id    = o.order_id
WHERE  o.status <> 'cancelled'
GROUP BY c.customer_id, c.name
HAVING SUM(oi.quantity * oi.unit_price) > 0
ORDER BY ltv DESC
LIMIT  3;
Output
name   | ltv      | orders
-------|----------|-------
Alice  | 1129.97  | 2
Dave   |  159.98  | 1
Bob    |   59.97  | 1

GROUP BY results into a pandas DataFrame

Runs an aggregation query through sqlite3 and loads results directly into pandas for further analysis.

PYTHON
import sqlite3
import pandas as pd

con = sqlite3.connect(":memory:")
# ... [DDL and seed data here] ...

query = """
    SELECT c.country,
           COUNT(DISTINCT c.customer_id) AS customers,
           COUNT(DISTINCT o.order_id)    AS orders,
           ROUND(SUM(oi.quantity * oi.unit_price), 2) AS revenue
    FROM   customers c
    LEFT JOIN orders o       ON o.customer_id = c.customer_id
    LEFT JOIN order_items oi ON oi.order_id   = o.order_id
    GROUP BY c.country
    ORDER BY revenue DESC NULLS LAST
"""

df = pd.read_sql_query(query, con)
print(df.to_string(index=False))
con.close()
Output
 country  customers  orders  revenue
      US          2       3  1189.94
      UK          1       1   159.98
      CA          2       1    59.97

Products never sold (aggregate on outer join)

Uses a LEFT JOIN combined with HAVING to find products that appear in zero order_items rows.

SQL
SELECT p.product_id,
       p.name,
       COUNT(oi.item_id) AS times_sold
FROM   products p
LEFT JOIN order_items oi ON oi.product_id = p.product_id
GROUP BY p.product_id, p.name
HAVING COUNT(oi.item_id) = 0;
Output
product_id | name    | times_sold
-----------|---------|----------
4          | T-Shirt | 0

42.3 Subqueries: Scalar, Derived Tables, and Correlated Intermediate

What Is a Subquery?

A subquery (also called an inner query or nested query) is a SELECT statement nested inside another SQL statement. Subqueries appear in three positions:

  1. In the SELECT list — scalar subqueries that return a single value.
  2. In the FROM clause — derived tables (or inline views).
  3. In the WHERE / HAVING clause — for filtering with IN, EXISTS, or comparison operators.

Each position has distinct semantics and performance characteristics.

Scalar Subqueries in SELECT

A scalar subquery returns exactly one column and one row. When placed in the SELECT list, it computes a value for each row of the outer query:

SQL
-- Each customer's spend compared to the overall average
SELECT c.name,
       ROUND(SUM(oi.quantity * oi.unit_price), 2)      AS customer_spend,
       ROUND(
         (SELECT AVG(sub.total)
          FROM (
            SELECT SUM(oi2.quantity * oi2.unit_price) AS total
            FROM   orders o2
            JOIN   order_items oi2 ON oi2.order_id = o2.order_id
            WHERE  o2.status <> 'cancelled'
            GROUP BY o2.customer_id
          ) sub
         ), 2
       )                                                AS avg_customer_spend
FROM   customers c
JOIN   orders o       ON o.customer_id  = c.customer_id
JOIN   order_items oi ON oi.order_id    = o.order_id
WHERE  o.status <> 'cancelled'
GROUP BY c.customer_id, c.name
ORDER BY customer_spend DESC;

The inner query computes the average spend per customer across the whole dataset; this value is the same for every row. In practice, this pattern is often cleaner as a CTE (Common Table Expression), but the subquery form illustrates the principle.

Derived Tables in FROM

A subquery in the FROM clause is called a derived table or inline view. It behaves exactly like a regular table but exists only for the duration of the query. You must give it an alias:

SQL
-- Products ranked by revenue, keeping only those in the top 50%
SELECT ranked.name,
       ranked.revenue,
       ranked.revenue_rank
FROM (
    SELECT p.product_id,
           p.name,
           SUM(oi.quantity * oi.unit_price)  AS revenue,
           RANK() OVER (ORDER BY SUM(oi.quantity * oi.unit_price) DESC)
                                             AS revenue_rank
    FROM   products p
    JOIN   order_items oi ON oi.product_id = p.product_id
    GROUP BY p.product_id, p.name
) ranked
WHERE ranked.revenue_rank <= (
    SELECT CEIL(COUNT(DISTINCT product_id) * 0.5)
    FROM   order_items
);

Derived tables are essential when you need to filter on an aggregate or window function result — something you cannot do directly in HAVING because HAVING only sees the current query's aggregates.

Subqueries in WHERE: IN and NOT IN

The IN operator tests whether a value appears in the result set of a subquery:

SQL
-- Customers who placed at least one order in 2024
SELECT customer_id, name, country
FROM   customers
WHERE  customer_id IN (
    SELECT DISTINCT customer_id
    FROM   orders
    WHERE  placed_at >= '2024-01-01'
);

NOT IN pitfall: If the subquery returns any NULL, NOT IN returns FALSE (or UNKNOWN) for every row, silently eliminating the entire result. This is because x NOT IN (1, 2, NULL) evaluates as x &lt;&gt; 1 AND x &lt;&gt; 2 AND x &lt;&gt; NULL, and x &lt;&gt; NULL is always UNKNOWN. Always add WHERE col IS NOT NULL inside a NOT IN subquery, or prefer NOT EXISTS instead.

Correlated Subqueries

A correlated subquery references columns from the outer query. It is re-evaluated once per row of the outer query, making it logically similar to a nested loop:

SQL
-- Orders whose total value exceeds that customer's average order value
SELECT o.order_id,
       o.customer_id,
       SUM(oi.quantity * oi.unit_price) AS order_total
FROM   orders o
JOIN   order_items oi ON oi.order_id = o.order_id
GROUP BY o.order_id, o.customer_id
HAVING SUM(oi.quantity * oi.unit_price) > (
    SELECT AVG(sub.total)
    FROM (
        SELECT SUM(oi2.quantity * oi2.unit_price) AS total
        FROM   orders o2
        JOIN   order_items oi2 ON oi2.order_id = o2.order_id
        WHERE  o2.customer_id = o.customer_id  -- correlation reference
        GROUP BY o2.order_id
    ) sub
);

Correlated subqueries are expressive but can be slow on large tables because the inner query runs for every outer row. Modern optimizers often rewrite them as joins automatically, but understanding the semantics is essential for debugging cases where the optimizer cannot.

EXISTS and NOT EXISTS

EXISTS returns TRUE if the subquery produces at least one row, FALSE otherwise. It short-circuits as soon as the first matching row is found:

SQL
-- Customers who have never placed any order
SELECT customer_id, name
FROM   customers c
WHERE  NOT EXISTS (
    SELECT 1
    FROM   orders o
    WHERE  o.customer_id = c.customer_id
);

The SELECT 1 inside EXISTS is idiomatic — the actual projection is irrelevant; only row existence matters. NOT EXISTS is the preferred pattern for "has no related rows" queries because it handles NULLs correctly (unlike NOT IN).

CTEs as Named Subqueries

Common Table Expressions (WITH clauses) are syntactic sugar that name subqueries for reuse and readability. They do not change semantics but dramatically improve query maintainability:

SQL
WITH order_totals AS (
    SELECT order_id,
           customer_id,
           SUM(quantity * unit_price) AS total
    FROM   order_items
    JOIN   orders USING (order_id)
    WHERE  status <> 'cancelled'
    GROUP BY order_id, customer_id
),
customer_avg AS (
    SELECT customer_id,
           AVG(total) AS avg_order_value
    FROM   order_totals
    GROUP BY customer_id
)
SELECT ot.order_id,
       ot.customer_id,
       ot.total,
       ca.avg_order_value,
       ot.total - ca.avg_order_value AS deviation
FROM   order_totals ot
JOIN   customer_avg ca USING (customer_id)
ORDER  BY deviation DESC;

CTEs also enable recursive queries (hierarchical data, graph traversal), covered in the advanced chapter.

Code Examples

Customers who bought every product in the Electronics category

Uses relational division via NOT EXISTS to find customers who have purchased all Electronics products.

SQL
-- Relational division: customers who ordered ALL electronics products
SELECT c.customer_id, c.name
FROM   customers c
WHERE  NOT EXISTS (
    -- A product in Electronics that this customer has NOT bought
    SELECT p.product_id
    FROM   products p
    JOIN   categories cat ON cat.category_id = p.category_id
    WHERE  cat.name = 'Electronics'
    AND    NOT EXISTS (
        SELECT 1
        FROM   orders o
        JOIN   order_items oi ON oi.order_id = o.order_id
        WHERE  o.customer_id  = c.customer_id
        AND    oi.product_id  = p.product_id
    )
);
Output
-- Returns customers who have purchased every Electronics product.
-- With the seed data, no customer qualifies (no one bought all 3 Electronics products).

Derived table: month-over-month revenue change

Uses a derived table to compute revenue per month and then a self-join to compute month-over-month delta.

SQL
WITH monthly AS (
    SELECT strftime('%Y-%m', o.placed_at) AS month,
           SUM(oi.quantity * oi.unit_price) AS revenue
    FROM   orders o
    JOIN   order_items oi ON oi.order_id = o.order_id
    WHERE  o.status <> 'cancelled'
    GROUP BY 1
)
SELECT curr.month,
       ROUND(curr.revenue, 2)                            AS revenue,
       ROUND(curr.revenue - prev.revenue, 2)             AS mom_delta,
       ROUND(100.0 * (curr.revenue - prev.revenue)
             / NULLIF(prev.revenue, 0), 1)               AS mom_pct
FROM   monthly curr
LEFT JOIN monthly prev
       ON prev.month = strftime('%Y-%m',
            date(curr.month || '-01', '-1 month'))
ORDER BY curr.month;
Output
month   | revenue | mom_delta | mom_pct
--------|---------|-----------|--------
2024-03 | 1099.97 |      NULL |    NULL
2024-04 |  239.96 |   -860.01 |  -78.2
2024-05 |   79.99 |   -159.97 |  -66.7

Detecting NOT IN / NULL pitfall

Demonstrates the silent empty-result bug when NOT IN encounters NULLs, and shows the NOT EXISTS fix.

PYTHON
import sqlite3

con = sqlite3.connect(":memory:")
con.executescript("""
    CREATE TABLE a (id INTEGER);
    CREATE TABLE b (id INTEGER);
    INSERT INTO a VALUES (1),(2),(3);
    INSERT INTO b VALUES (2),(3),( NULL );  -- Note the NULL
""")

print("NOT IN with NULL in subquery (broken):")
result = con.execute(
    "SELECT id FROM a WHERE id NOT IN (SELECT id FROM b)"
).fetchall()
print(result)  # Empty! NULL poisons NOT IN

print("\nNOT IN with explicit NULL filter (fixed):")
result = con.execute(
    "SELECT id FROM a WHERE id NOT IN (SELECT id FROM b WHERE id IS NOT NULL)"
).fetchall()
print(result)  # [(1,)]

print("\nNOT EXISTS (always correct):")
result = con.execute(
    "SELECT id FROM a WHERE NOT EXISTS (SELECT 1 FROM b WHERE b.id = a.id)"
).fetchall()
print(result)  # [(1,)]

con.close()
Output
NOT IN with NULL in subquery (broken):
[]

NOT IN with explicit NULL filter (fixed):
[(1,)]

NOT EXISTS (always correct):
[(1,)]

42.4 Set Operations: UNION, INTERSECT, EXCEPT Intermediate

Combining Result Sets

Set operations stack query results vertically — they combine rows from two result sets rather than joining them horizontally. SQL provides three operations corresponding to the three classical set operations:

  • UNION — rows in either result (duplicates removed).
  • UNION ALL — rows in either result (duplicates kept).
  • INTERSECT — rows in both results.
  • EXCEPT (called MINUS in Oracle) — rows in the first result but not the second.

Requirements: Both queries must have the same number of columns, and corresponding columns must have compatible types. Column names in the final result come from the first query.

UNION vs UNION ALL

UNION deduplicates the combined result using a hash or sort, adding CPU and memory overhead. UNION ALL skips deduplication entirely. Whenever you know the two result sets are disjoint (or you explicitly want all rows including duplicates), always prefer UNION ALL for performance.

SQL
-- All people (customers + staff) in the system,
-- tagged by role. The two sub-tables are disjoint by design.
SELECT customer_id AS person_id,
       name,
       'customer' AS role
FROM   customers
UNION ALL
SELECT staff_id,
       name,
       'staff'
FROM   staff;  -- hypothetical table

A common analytical pattern is combining the same table at different aggregation levels:

SQL
-- Revenue by product, plus a grand-total row (poor-man's ROLLUP)
SELECT p.name        AS label,
       SUM(oi.quantity * oi.unit_price) AS revenue
FROM   products p
JOIN   order_items oi ON oi.product_id = p.product_id
GROUP BY p.name

UNION ALL

SELECT 'TOTAL',
       SUM(oi.quantity * oi.unit_price)
FROM   order_items oi
ORDER BY revenue DESC;

Note that the ORDER BY at the end applies to the entire union result.

INTERSECT — Finding Common Rows

INTERSECT returns only rows that appear in both result sets. It is logically equivalent to an INNER JOIN on all projected columns, but often cleaner for set-membership tests:

SQL
-- Customers who placed orders in BOTH March and April 2024
SELECT DISTINCT customer_id
FROM   orders
WHERE  placed_at >= '2024-03-01' AND placed_at < '2024-04-01'

INTERSECT

SELECT DISTINCT customer_id
FROM   orders
WHERE  placed_at >= '2024-04-01' AND placed_at < '2024-05-01';

With the seed data, Alice (customer 1) ordered in both March and April, so she is the only result.

EXCEPT — Finding Rows Unique to the First Set

EXCEPT returns rows from the first query that do not appear in the second. This is the set-theoretic difference $A \setminus B$:

SQL
-- Products sold in 2024 Q1 but NOT in Q2 (discontinued or out of stock)
SELECT DISTINCT oi.product_id
FROM   order_items oi
JOIN   orders o ON o.order_id = oi.order_id
WHERE  o.placed_at >= '2024-01-01' AND o.placed_at < '2024-04-01'

EXCEPT

SELECT DISTINCT oi.product_id
FROM   order_items oi
JOIN   orders o ON o.order_id = oi.order_id
WHERE  o.placed_at >= '2024-04-01' AND o.placed_at < '2024-07-01';

Practical: Multi-Source Report Consolidation

Set operations are invaluable when consolidating data from separate sources (legacy systems, partitioned tables, different regions) into a single analysis:

SQL
-- Combine orders from two regional databases (already present in one schema here,
-- but modeled as separate queries to illustrate the pattern)
WITH all_orders AS (
    SELECT order_id, customer_id, placed_at, 'US' AS region
    FROM   orders
    WHERE  customer_id IN (SELECT customer_id FROM customers WHERE country = 'US')

    UNION ALL

    SELECT order_id, customer_id, placed_at, 'INTL' AS region
    FROM   orders
    WHERE  customer_id IN (SELECT customer_id FROM customers WHERE country <> 'US')
)
SELECT region,
       COUNT(*) AS order_count,
       MIN(placed_at) AS earliest,
       MAX(placed_at) AS latest
FROM   all_orders
GROUP BY region;

Ordering and Limiting Set Operation Results

Only one ORDER BY is allowed and it must come after the final query in the set operation. You cannot ORDER BY inside individual member queries (except in databases that support it through parenthesization). LIMIT/FETCH applies to the final combined result:

SQL
(SELECT name, 'customer' AS role FROM customers)
UNION ALL
(SELECT name, 'staff' AS role FROM staff)
ORDER BY name
LIMIT 20;

Code Examples

Symmetric difference with EXCEPT and UNION ALL

Computes the symmetric difference (rows in either set but not both) using EXCEPT and UNION ALL.

SQL
-- Symmetric difference: product IDs sold in Q1 XOR Q2 (not in both)
SELECT product_id, 'Q1 only' AS presence
FROM (
    SELECT DISTINCT oi.product_id
    FROM order_items oi JOIN orders o USING (order_id)
    WHERE o.placed_at < '2024-04-01'
) q1
EXCEPT
    SELECT DISTINCT oi.product_id
    FROM order_items oi JOIN orders o USING (order_id)
    WHERE o.placed_at >= '2024-04-01'

UNION ALL

SELECT product_id, 'Q2 only' AS presence
FROM (
    SELECT DISTINCT oi.product_id
    FROM order_items oi JOIN orders o USING (order_id)
    WHERE o.placed_at >= '2024-04-01'
) q2
EXCEPT
    SELECT DISTINCT oi.product_id
    FROM order_items oi JOIN orders o USING (order_id)
    WHERE o.placed_at < '2024-04-01'
ORDER BY product_id;
Output
-- With seed data, product 1 (Laptop) was only sold in Q1
-- product_id | presence
-- 1          | Q1 only

Set operations with pandas for verification

Shows how INTERSECT and EXCEPT map to pandas operations, useful for cross-checking SQL results.

PYTHON
import pandas as pd

# Simulate two query result sets
q1_products = pd.DataFrame({'product_id': [1, 2, 3]})
q2_products = pd.DataFrame({'product_id': [2, 3, 4, 5]})

# UNION ALL
union_all = pd.concat([q1_products, q2_products], ignore_index=True)
print("UNION ALL:", sorted(union_all['product_id'].tolist()))

# UNION (deduplicated)
union = union_all.drop_duplicates()
print("UNION:    ", sorted(union['product_id'].tolist()))

# INTERSECT
q1_set = set(q1_products['product_id'])
q2_set = set(q2_products['product_id'])
intersect = pd.DataFrame({'product_id': sorted(q1_set & q2_set)})
print("INTERSECT:", sorted(intersect['product_id'].tolist()))

# EXCEPT (q1 minus q2)
except_ = pd.DataFrame({'product_id': sorted(q1_set - q2_set)})
print("EXCEPT:   ", sorted(except_['product_id'].tolist()))
Output
UNION ALL:  [1, 2, 2, 3, 3, 4, 5]
UNION:      [1, 2, 3, 4, 5]
INTERSECT:  [2, 3]
EXCEPT:     [1]

42.5 CASE Expressions: Conditional Logic in SQL Intermediate

CASE Is an Expression, Not a Statement

Unlike the CASE statement in procedural languages (which controls program flow), SQL's CASE is an expression — it evaluates to a value and can appear anywhere a value is valid: SELECT, WHERE, GROUP BY, ORDER BY, and as arguments to aggregate functions. This makes it extraordinarily versatile.

SQL provides two syntactic forms:

Simple CASE (equality comparison):

SQL
CASE status
    WHEN 'shipped'   THEN 'Fulfilled'
    WHEN 'pending'   THEN 'In Progress'
    WHEN 'cancelled' THEN 'Cancelled'
    ELSE                  'Unknown'
END

Searched CASE (arbitrary boolean conditions):

SQL
CASE
    WHEN total > 1000 THEN 'High Value'
    WHEN total > 100  THEN 'Medium Value'
    ELSE                   'Low Value'
END

The searched form is more general and is typically preferred because it supports any boolean expression, including BETWEEN, LIKE, IS NULL, and subqueries.

Labeling and Bucketing

The most common use of CASE is transforming raw values into human-readable labels or analytical buckets:

SQL
-- Order value tier for each order
SELECT o.order_id,
       c.name,
       ROUND(SUM(oi.quantity * oi.unit_price), 2) AS order_total,
       CASE
           WHEN SUM(oi.quantity * oi.unit_price) >= 500  THEN 'Platinum'
           WHEN SUM(oi.quantity * oi.unit_price) >= 100  THEN 'Gold'
           WHEN SUM(oi.quantity * oi.unit_price) >= 50   THEN 'Silver'
           ELSE                                               'Bronze'
       END                                           AS value_tier
FROM   orders o
JOIN   customers c    ON c.customer_id = o.customer_id
JOIN   order_items oi ON oi.order_id   = o.order_id
GROUP BY o.order_id, c.name
ORDER BY order_total DESC;

CASE Inside Aggregate Functions — Conditional Aggregation

Placing CASE inside an aggregate function enables conditional aggregation — computing multiple aggregates over different subsets of data in a single pass. This is far more efficient than running separate queries for each condition:

SQL
-- Revenue by status in a single query row per customer (pivot-like)
SELECT c.name,
       SUM(CASE WHEN o.status = 'shipped'
                THEN oi.quantity * oi.unit_price ELSE 0 END) AS shipped_rev,
       SUM(CASE WHEN o.status = 'pending'
                THEN oi.quantity * oi.unit_price ELSE 0 END) AS pending_rev,
       SUM(CASE WHEN o.status = 'cancelled'
                THEN oi.quantity * oi.unit_price ELSE 0 END) AS cancelled_rev,
       COUNT(CASE WHEN o.status = 'shipped'   THEN 1 END)    AS shipped_count,
       COUNT(CASE WHEN o.status = 'cancelled' THEN 1 END)    AS cancelled_count
FROM   customers c
JOIN   orders o       ON o.customer_id = c.customer_id
JOIN   order_items oi ON oi.order_id   = o.order_id
GROUP BY c.customer_id, c.name
ORDER BY shipped_rev DESC;

Notice the asymmetry: SUM(CASE ... ELSE 0 END) uses ELSE 0 so that non-matching rows contribute 0 to the sum. COUNT(CASE ... END) omits ELSE (implying ELSE NULL) because COUNT ignores NULLs — non-matching rows should not add to the count.

CASE in ORDER BY — Custom Sort Orders

CASE in ORDER BY lets you impose arbitrary sort sequences that alphabetical or numeric ordering cannot achieve:

SQL
-- Status-priority sort: shipped first, then pending, then cancelled
SELECT order_id, status, placed_at
FROM   orders
ORDER BY
    CASE status
        WHEN 'shipped'   THEN 1
        WHEN 'pending'   THEN 2
        WHEN 'cancelled' THEN 3
        ELSE                  4
    END,
    placed_at DESC;

CASE for NULL Handling

CASE elegantly handles NULL substitution (similar to COALESCE, but more expressive when the replacement value depends on other columns):

SQL
SELECT product_id,
       name,
       CASE
           WHEN price IS NULL   THEN 'Price TBD'
           WHEN price = 0       THEN 'Free'
           ELSE CAST(price AS TEXT)
       END AS price_display
FROM products;

Nested and Chained CASE

CASE expressions can be nested, though deeply nested CASE is often a sign that a lookup table or application logic would be cleaner:

SQL
-- Two-level classification: category then price tier
SELECT name,
       CASE
           WHEN category_id = 1 THEN  -- Electronics
               CASE WHEN price >= 500 THEN 'Premium Electronics'
                    ELSE 'Budget Electronics' END
           WHEN category_id = 2 THEN 'Reference Material'
           ELSE 'General Merchandise'
       END AS classification
FROM products;

Common Pitfalls

Evaluation order matters. CASE evaluates conditions from top to bottom and returns on the first TRUE. Overlapping conditions must be ordered from most specific to least specific.

Type consistency. All THEN and ELSE branches must return compatible types. Mixing TEXT and NUMERIC causes a type error in strict databases. Use explicit CAST when needed.

Missing ELSE. Without ELSE, CASE returns NULL for unmatched rows. In aggregate contexts this is often intentional (COUNT ignores NULLs); in display contexts you usually want an explicit ELSE fallback.

Code Examples

Conditional aggregation: retention cohort table

Builds a cohort table showing how many customers who joined in each quarter placed repeat orders.

SQL
SELECT
    CASE
        WHEN joined_at < '2023-04-01' THEN '2023-Q1'
        WHEN joined_at < '2023-07-01' THEN '2023-Q2'
        WHEN joined_at < '2023-10-01' THEN '2023-Q3'
        WHEN joined_at < '2024-01-01' THEN '2023-Q4'
        ELSE '2024+'
    END                                              AS join_cohort,
    COUNT(DISTINCT c.customer_id)                    AS cohort_size,
    COUNT(DISTINCT CASE WHEN o.order_id IS NOT NULL
                   THEN c.customer_id END)           AS ever_ordered,
    COUNT(DISTINCT CASE
        WHEN order_rank.num_orders >= 2
        THEN c.customer_id END)                      AS repeat_buyers
FROM customers c
LEFT JOIN (
    SELECT customer_id,
           COUNT(order_id) AS num_orders
    FROM   orders
    WHERE  status <> 'cancelled'
    GROUP BY customer_id
) order_rank ON order_rank.customer_id = c.customer_id
LEFT JOIN orders o ON o.customer_id = c.customer_id
                   AND o.status <> 'cancelled'
GROUP BY 1
ORDER BY 1;
Output
join_cohort | cohort_size | ever_ordered | repeat_buyers
------------|-------------|--------------|---------------
2023-Q1     | 1           | 1            | 1
2023-Q2     | 1           | 0            | 0
2024+       | 3           | 3            | 0

Manual pivot: product revenue by month using CASE

Pivots order_items into month columns without PIVOT syntax, which is unavailable in standard SQL.

SQL
SELECT
    p.name,
    ROUND(SUM(CASE WHEN strftime('%Y-%m', o.placed_at) = '2024-03'
                   THEN oi.quantity * oi.unit_price ELSE 0 END), 2) AS mar_2024,
    ROUND(SUM(CASE WHEN strftime('%Y-%m', o.placed_at) = '2024-04'
                   THEN oi.quantity * oi.unit_price ELSE 0 END), 2) AS apr_2024,
    ROUND(SUM(CASE WHEN strftime('%Y-%m', o.placed_at) = '2024-05'
                   THEN oi.quantity * oi.unit_price ELSE 0 END), 2) AS may_2024
FROM products p
JOIN order_items oi ON oi.product_id = p.product_id
JOIN orders o       ON o.order_id    = oi.order_id
WHERE o.status <> 'cancelled'
GROUP BY p.product_id, p.name
ORDER BY p.name;
Output
name        | mar_2024 | apr_2024 | may_2024
------------|----------|----------|---------
Headphones  | 0.00     | 79.99    | 159.98
Laptop      | 999.99   | 0.00     | 0.00
SQL Textbook| 99.98    | 0.00     | 0.00

Replicating CASE logic in pandas with np.select

Shows how SQL CASE maps to numpy's np.select in pandas, useful for validating SQL bucket logic.

PYTHON
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'order_id': [1, 2, 3, 4, 5],
    'total':    [1099.97, 79.99, 59.97, 499.99, 159.98]
})

# Equivalent to SQL CASE WHEN total >= 500 THEN ... ELSE ...
conditions = [
    df['total'] >= 500,
    df['total'] >= 100,
    df['total'] >= 50,
]
choices = ['Platinum', 'Gold', 'Silver']
df['value_tier'] = np.select(conditions, choices, default='Bronze')

print(df[['order_id', 'total', 'value_tier']].to_string(index=False))
Output
 order_id    total value_tier
        1  1099.97   Platinum
        2    79.99     Silver
        3    59.97     Silver
        4   499.99       Gold
        5   159.98       Gold

42.6 Multi-Join Analytical Queries: Putting It All Together Intermediate

Composing Complex Analytical Queries

Real analytical work rarely involves a single clause in isolation. A production-grade report query typically chains together: multiple joins, aggregation, HAVING, subqueries or CTEs, CASE expressions, and set operations. This section walks through increasingly complex examples using the full chapter schema, narrating the reasoning process as a working analyst would apply it.

Query Design Process

Before writing SQL, answer these questions:

  1. What is the grain of the result? One row per what — customer, order, product, (customer, month)?
  2. Which tables contain the needed data? Trace the foreign-key graph.
  3. What join type is needed? Should unmatched rows appear (outer join) or be silently dropped (inner join)?
  4. What aggregations are needed? At what granularity?
  5. Are there filter conditions on raw rows (WHERE) or aggregate results (HAVING)?
  6. Example 1: Customer Purchase Funnel Report

Goal: For each customer, show total orders, total line items, gross revenue, a value tier, and whether they are a repeat buyer.

SQL
WITH customer_metrics AS (
    SELECT
        c.customer_id,
        c.name,
        c.country,
        c.joined_at,
        COUNT(DISTINCT o.order_id)               AS total_orders,
        COUNT(oi.item_id)                        AS total_items,
        SUM(oi.quantity * oi.unit_price)         AS gross_revenue
    FROM   customers c
    LEFT JOIN orders o       ON o.customer_id = c.customer_id
                            AND o.status <> 'cancelled'
    LEFT JOIN order_items oi ON oi.order_id   = o.order_id
    GROUP BY c.customer_id, c.name, c.country, c.joined_at
)
SELECT
    customer_id,
    name,
    country,
    total_orders,
    total_items,
    COALESCE(ROUND(gross_revenue, 2), 0.00)   AS gross_revenue,
    CASE
        WHEN gross_revenue >= 1000 THEN 'Platinum'
        WHEN gross_revenue >= 200  THEN 'Gold'
        WHEN gross_revenue >= 50   THEN 'Silver'
        WHEN gross_revenue >  0    THEN 'Bronze'
        ELSE                            'Inactive'
    END                                       AS value_tier,
    CASE WHEN total_orders >= 2 THEN 'Yes' ELSE 'No' END AS repeat_buyer
FROM customer_metrics
ORDER BY gross_revenue DESC NULLS LAST;

Key design decisions: the LEFT JOINs preserve customers with no orders (they appear with 0s rather than being omitted). The status filter o.status &lt;&gt; &#039;cancelled&#039; is in the ON clause, not the WHERE clause — putting it in WHERE would convert the LEFT JOIN to an effective INNER JOIN by filtering out NULLs.

Example 2: Product Performance with Category Context

Goal: Revenue, quantity sold, order count, and average unit price per product, including products never sold, with their category name.

SQL
SELECT
    cat.name                                   AS category,
    p.product_id,
    p.name                                     AS product,
    p.price                                    AS list_price,
    COALESCE(COUNT(oi.item_id), 0)             AS times_ordered,
    COALESCE(SUM(oi.quantity), 0)              AS units_sold,
    COALESCE(COUNT(DISTINCT oi.order_id), 0)   AS distinct_orders,
    COALESCE(ROUND(SUM(oi.quantity * oi.unit_price), 2), 0) AS total_revenue,
    ROUND(AVG(oi.unit_price), 2)               AS avg_selling_price
FROM   categories cat
JOIN   products p    ON p.category_id  = cat.category_id
LEFT JOIN order_items oi ON oi.product_id = p.product_id
LEFT JOIN orders o       ON o.order_id   = oi.order_id
                         AND o.status <> 'cancelled'
GROUP BY cat.category_id, cat.name,
         p.product_id, p.name, p.price
ORDER BY cat.name, total_revenue DESC;

Note the JOIN categories → products (INNER) combined with LEFT JOIN order<em>items — this gives us all products regardless of sales history.

Example 3: Cohort Retention Query

Goal: For customers acquired in each half-year cohort, what fraction placed a second order within 90 days of their first?

SQL
WITH first_orders AS (
    SELECT customer_id,
           MIN(placed_at) AS first_order_date
    FROM   orders
    WHERE  status <> 'cancelled'
    GROUP BY customer_id
),
cohorted AS (
    SELECT
        c.customer_id,
        CASE
            WHEN fo.first_order_date < '2024-01-01' THEN 'Pre-2024'
            ELSE '2024+'
        END                        AS cohort,
        fo.first_order_date,
        MIN(o2.placed_at)          AS second_order_date
    FROM   customers c
    JOIN   first_orders fo ON fo.customer_id = c.customer_id
    LEFT JOIN orders o2
           ON  o2.customer_id = c.customer_id
           AND o2.status      <> 'cancelled'
           AND o2.placed_at   > fo.first_order_date  -- must be AFTER first order
    GROUP BY c.customer_id, cohort, fo.first_order_date
)
SELECT
    cohort,
    COUNT(*)                                              AS cohort_size,
    COUNT(CASE WHEN second_order_date IS NOT NULL
               AND julianday(second_order_date)
                   - julianday(first_order_date) <= 90
               THEN 1 END)                               AS returned_90d,
    ROUND(
        100.0 * COUNT(CASE
            WHEN second_order_date IS NOT NULL
            AND  julianday(second_order_date)
                 - julianday(first_order_date) <= 90
            THEN 1 END
        ) / COUNT(*), 1
    )                                                    AS retention_pct_90d
FROM   cohorted
GROUP BY cohort
ORDER BY cohort;

Common Multi-Join Pitfalls

Fan-out duplication. Joining a table on the "many" side without aggregating first inflates row counts. If you join orders (1 row each) to order</em>items (multiple rows per order) and then SUM(order revenue), you must SUM order<em>items.unit</em>price * quantity, not a column from orders.

Filter placement in outer joins. A WHERE predicate on a right-side column of a LEFT JOIN converts it to an inner join. Put such conditions in the ON clause if you still want left-side rows preserved.

Ambiguous column names. In queries with four or more tables, always qualify every column with a table alias. Ambiguous column references produce errors or — worse — silently pick the wrong column.

NULL propagation. COALESCE is your friend when converting NULLs from outer-join misses to zero or empty strings for reporting.

Code Examples

Full funnel: category to customer drill-down

A single query that joins all five tables to produce a per-category, per-customer revenue breakdown.

SQL
SELECT
    cat.name                                      AS category,
    c.name                                        AS customer,
    COUNT(DISTINCT o.order_id)                    AS orders,
    SUM(oi.quantity)                              AS units,
    ROUND(SUM(oi.quantity * oi.unit_price), 2)    AS revenue
FROM   categories cat
JOIN   products p     ON p.category_id  = cat.category_id
JOIN   order_items oi ON oi.product_id  = p.product_id
JOIN   orders o       ON o.order_id     = oi.order_id
JOIN   customers c    ON c.customer_id  = o.customer_id
WHERE  o.status <> 'cancelled'
GROUP BY cat.category_id, cat.name,
         c.customer_id, c.name
ORDER BY cat.name, revenue DESC;
Output
category     | customer | orders | units | revenue
-------------|----------|--------|-------|---------
Books        | Alice    | 1      | 2     | 99.98
Electronics  | Alice    | 2      | 2     | 1079.98
Electronics  | Dave     | 1      | 2     | 159.98

End-to-end: seed, query, and visualize with matplotlib

Seeds the SQLite database, runs the customer metrics CTE query, and plots revenue by tier.

PYTHON
import sqlite3
import pandas as pd
import matplotlib.pyplot as plt

con = sqlite3.connect(":memory:")
# [run full DDL + seed INSERT statements here]

query = """
    WITH m AS (
        SELECT c.name,
               COALESCE(SUM(oi.quantity * oi.unit_price), 0) AS rev
        FROM customers c
        LEFT JOIN orders o       ON o.customer_id = c.customer_id
                                AND o.status <> 'cancelled'
        LEFT JOIN order_items oi ON oi.order_id = o.order_id
        GROUP BY c.customer_id, c.name
    )
    SELECT name,
           ROUND(rev, 2) AS revenue,
           CASE
             WHEN rev >= 1000 THEN 'Platinum'
             WHEN rev >= 200  THEN 'Gold'
             WHEN rev >= 50   THEN 'Silver'
             WHEN rev >  0    THEN 'Bronze'
             ELSE 'Inactive'
           END AS tier
    FROM m
    ORDER BY revenue DESC
"""

df = pd.read_sql_query(query, con)
print(df.to_string(index=False))

fig, ax = plt.subplots()
ax.barh(df['name'], df['revenue'], color='steelblue')
ax.set_xlabel('Revenue ($)')
ax.set_title('Customer Lifetime Value')
plt.tight_layout()
plt.savefig('/tmp/customer_ltv.png', dpi=100)
print('Chart saved to /tmp/customer_ltv.png')
con.close()
Output
    name   revenue      tier
   Alice   1129.97  Platinum
    Dave    159.98      Gold
     Bob     59.97    Silver
   Carol      0.00  Inactive
     Eve      0.00  Inactive
Chart saved to /tmp/customer_ltv.png

Detecting data quality issues with a multi-join audit query

Uses LEFT JOINs and CASE to audit referential integrity and flag orphaned records.

SQL
-- Audit query: find order_items with missing parent records
SELECT
    oi.item_id,
    oi.order_id,
    oi.product_id,
    CASE WHEN o.order_id   IS NULL THEN 'MISSING ORDER'   ELSE 'OK' END AS order_status,
    CASE WHEN p.product_id IS NULL THEN 'MISSING PRODUCT' ELSE 'OK' END AS product_status
FROM   order_items oi
LEFT JOIN orders   o ON o.order_id   = oi.order_id
LEFT JOIN products p ON p.product_id = oi.product_id
WHERE  o.order_id IS NULL
    OR p.product_id IS NULL;
Output
-- With the seed data, all FK relationships are intact, so no rows returned.
-- In production, orphaned records would appear here.