Distributed Data and Computation

November 23 · Scale the measured bottleneck

MaDS Databases & SQL

One question will organize today

When is one well-designed database no longer enough—and what complexity would distribution buy us?

Course source and adaptation

Today’s sequence follows Alex Reinhart’s Distributed Data and Computation. Hadoop/MapReduce ideas are treated as enduring design concepts; the project decision begins with measured Fall 2026 workloads.

Current reference: Apache Spark cluster overview.

By the end of class

You should be able to:

  • distinguish scale-up from scale-out,
  • identify data and computation boundaries,
  • explain map, shuffle, and reduce,
  • reason about partitioning and data movement,
  • reject distribution when a simpler fix solves the bottleneck.

Measure before distributing

Ask:

  • Which step is slow?
  • CPU, memory, disk, network, lock, or source API?
  • How much data and how fast is it growing?
  • Can an index, batch, column selection, or larger machine fix it?

Distribution is an architecture change, not a performance adjective.

Scale up or scale out

Scale up: more CPU, RAM, or faster storage on one system.

Scale out: partition data/work across multiple systems.

Scale-out adds coordination, retries, skew, and partial failure.

Data can be distributed without compute

Large immutable source files can live in object storage partitioned by source/date while PostgreSQL holds curated facts and indexes.

raw objects → parallel/staged transforms → curated PostgreSQL

This hybrid is often enough.

Partition keys are workload decisions

Possible keys:

  • source and acquisition date,
  • observation/event year,
  • geography,
  • route or corridor.

A good key prunes most data for common work and avoids one giant hot partition.

Checkpoint 1 · Find the bottleneck

Choose the slowest inherited workflow. Record data size, elapsed time, rows read/written, and one plan or system clue. Name the simplest plausible fix.

Distributed computation moves work to partitions

Classic MapReduce:

map each partition → emit key/value
        ↓
shuffle values by key
        ↓
reduce each key → final aggregate

The shuffle is often the expensive part.

Example · Count events by state

Map: parse event rows and emit (state, 1).

Shuffle: group all values for the same state.

Reduce: sum counts for each state.

If every worker needs every row, the partition plan failed.

Joins expose data movement

Small lookup + huge fact table:

  • broadcast the small lookup.

Two huge tables keyed alike:

  • partition both by the join key.

Spatial or many-to-many join:

  • expect expensive candidate generation; prefilter and index first.

Skew creates stragglers

One state, year, route, or key may dominate the data.

Average partition size can look fine while one worker determines the finish time.

Inspect the distribution, not only total bytes.

Checkpoint 2 · Map, shuffle, reduce

Rewrite one client aggregation as map/shuffle/reduce. State the partition key, shuffled key, partial result, and likely skew.

Then ask whether PostgreSQL already performs it adequately.

Partial failure is normal

Distributed workflows need:

  • deterministic tasks,
  • retry-safe outputs,
  • checkpoints/manifests,
  • task-level logs,
  • protection from publishing partial results.

The idempotency lesson returns at a larger scale.

Consistency choices become visible

If partitions finish at different times, when is the dataset ready?

Use a manifest, atomic pointer, or publish step so clients do not mix old and new partitions unknowingly.

Spark improves the execution model, not physics

Modern engines can keep data in memory, build a directed execution plan, and optimize many transformations beyond classic disk-heavy MapReduce.

They still pay for shuffles, skew, serialization, and coordination.

Checkpoint 3 · Write a no/yes memo

Recommend one of three actions: keep one database, add object-storage partitions, or distribute compute. Support it with the measured bottleneck and an acceptance threshold.

Include the new failure mode your choice introduces.

Project transfer

Add a scaling boundary to the inherited architecture: current limit, measurement, next intervention, and trigger for reconsidering distribution.

Do not add infrastructure the receiver cannot operate.

Homework starts here

Homework 3 may use a scale design as its extension only when it includes a measured bottleneck and a small executable analogue or benchmark.

The pattern to keep

measure the bottleneck
  → try the simplest fix
  → choose a useful partition
  → minimize data movement
  → plan for skew and partial failure
  → publish atomically
  → distribute only past a stated trigger

After Thanksgiving: package the workflow so the receiver installs a command, not a pile of files.