---
title: "Distributed Data and Computation"
subtitle: "November 23 · Scale the measured bottleneck"
author: "MaDS Databases & SQL"
format:
  revealjs:
    theme: [default, mads-sql-reveal.scss]
    slide-number: c/t
    chalkboard: true
    code-line-numbers: true
    transition: fade
    footer: "Adapted from Alex Reinhart · MADS Computing"
---

## One question will organize today

::: {.question}
When is one well-designed database no longer enough—and what complexity would distribution buy us?
:::

## Course source and adaptation

::: {.source-note}
Today's sequence follows Alex Reinhart's [Distributed Data and Computation](https://www.refsmmat.com/courses/msp-computing/data-engineering/distributed-data.html). Hadoop/MapReduce ideas are treated as enduring design concepts; the project decision begins with measured Fall 2026 workloads.

Current reference: [Apache Spark cluster overview](https://spark.apache.org/docs/latest/cluster-overview.html).
:::

## By the end of class

You should be able to:

- distinguish scale-up from scale-out,
- identify data and computation boundaries,
- explain map, shuffle, and reduce,
- reason about partitioning and data movement,
- reject distribution when a simpler fix solves the bottleneck.

## Measure before distributing

Ask:

- Which step is slow?
- CPU, memory, disk, network, lock, or source API?
- How much data and how fast is it growing?
- Can an index, batch, column selection, or larger machine fix it?

Distribution is an architecture change, not a performance adjective.

## Scale up or scale out

**Scale up:** more CPU, RAM, or faster storage on one system.

**Scale out:** partition data/work across multiple systems.

Scale-out adds coordination, retries, skew, and partial failure.

## Data can be distributed without compute

Large immutable source files can live in object storage partitioned by source/date while
PostgreSQL holds curated facts and indexes.

```text
raw objects → parallel/staged transforms → curated PostgreSQL
```

This hybrid is often enough.

## Partition keys are workload decisions

Possible keys:

- source and acquisition date,
- observation/event year,
- geography,
- route or corridor.

A good key prunes most data for common work and avoids one giant hot partition.

## Checkpoint 1 · Find the bottleneck

::: {.checkpoint}
Choose the slowest inherited workflow. Record data size, elapsed time, rows read/written, and one plan or system clue. Name the simplest plausible fix.
:::

## Distributed computation moves work to partitions

Classic MapReduce:

```text
map each partition → emit key/value
        ↓
shuffle values by key
        ↓
reduce each key → final aggregate
```

The shuffle is often the expensive part.

## Example · Count events by state

**Map:** parse event rows and emit `(state, 1)`.

**Shuffle:** group all values for the same state.

**Reduce:** sum counts for each state.

If every worker needs every row, the partition plan failed.

## Joins expose data movement

Small lookup + huge fact table:

- broadcast the small lookup.

Two huge tables keyed alike:

- partition both by the join key.

Spatial or many-to-many join:

- expect expensive candidate generation; prefilter and index first.

## Skew creates stragglers

One state, year, route, or key may dominate the data.

Average partition size can look fine while one worker determines the finish time.

Inspect the distribution, not only total bytes.

## Checkpoint 2 · Map, shuffle, reduce

::: {.checkpoint}
Rewrite one client aggregation as map/shuffle/reduce. State the partition key, shuffled key, partial result, and likely skew.
:::

Then ask whether PostgreSQL already performs it adequately.

## Partial failure is normal

Distributed workflows need:

- deterministic tasks,
- retry-safe outputs,
- checkpoints/manifests,
- task-level logs,
- protection from publishing partial results.

The idempotency lesson returns at a larger scale.

## Consistency choices become visible

If partitions finish at different times, when is the dataset ready?

Use a manifest, atomic pointer, or publish step so clients do not mix old and new
partitions unknowingly.

## Spark improves the execution model, not physics

Modern engines can keep data in memory, build a directed execution plan, and optimize
many transformations beyond classic disk-heavy MapReduce.

They still pay for shuffles, skew, serialization, and coordination.

## Checkpoint 3 · Write a no/yes memo

::: {.checkpoint}
Recommend one of three actions: keep one database, add object-storage partitions, or distribute compute. Support it with the measured bottleneck and an acceptance threshold.
:::

Include the new failure mode your choice introduces.

## Project transfer

::: {.project-prompt}
Add a scaling boundary to the inherited architecture: current limit, measurement, next intervention, and trigger for reconsidering distribution.
:::

Do not add infrastructure the receiver cannot operate.

## Homework starts here

Homework 3 may use a scale design as its extension only when it includes a measured
bottleneck and a small executable analogue or benchmark.

## The pattern to keep

```text
measure the bottleneck
  → try the simplest fix
  → choose a useful partition
  → minimize data movement
  → plan for skew and partial failure
  → publish atomically
  → distribute only past a stated trigger
```

After Thanksgiving: package the workflow so the receiver installs a command, not a pile of files.