36-614 · Mini 2 · Oct 19–Dec 4

Data Engineering and Distributed Environments

PostgreSQL, SQL, data pipelines, and the practical work of handing a system to someone else.

Meetings: Mondays and Wednesdays, 9:30–10:50 a.m., in Baker Hall 140A. Database plan: students connect to one managed PostgreSQL server using individual credentials and work in assigned databases or schemas.

Quick access

Homework

Each editable QMD and ready-to-read PDF appears automatically on its scheduled release date.

Lectures + homework

Weekly calendar

Slides, source files, checkpoints, assignments, and due dates appear in the week when students need them.

Week 1 · Oct 19–23

Start asking the database

  • Before Monday: install the tools and connect to the course server. Setup primer
  • Mon Oct 19: Pipeline motivation; SELECT, WHERE, NULL, and sorting. Slides · Source · Checkpoints
  • Wed Oct 21: Aggregation, GROUP BY, HAVING, and honest denominators. Slides · Source · Checkpoints
  • HW1 · opens Sun Oct 25; due Tue Nov 3 at 11:59 p.m.: SQL foundations + schema audit, with one small AI query comparison.

Week 2 · Oct 26–30

Join tables and design the schema

Week 3 · Nov 2–6

Compose and diagnose queries

  • Mon Nov 2: CASE, subqueries, CTEs, and set operations. Slides · Source · Checkpoints
  • Tue Nov 3 at 11:59 p.m.: HW1 due. Files are available in the homework section.
  • Wed Nov 4: Windows, ranking, lag/lead, indexes, and EXPLAIN. Slides · Source · Checkpoints
  • HW2 · opens Sun Nov 1; due Thu Nov 12 at 11:59 p.m.: builder query library + handoff readiness, with one small AI documentation check.

Week 4 · Nov 9–13

Make the system transferable

  • Mon Nov 9: End-to-end flow, idempotency, validation, failure, and monitoring. Slides · Source · Checkpoints
  • Wed Nov 11: PostgreSQL from Python: parameters, transactions, batching, and failure handling. Slides · Source · Checkpoints
  • Thu Nov 12 at 11:59 p.m.: HW2 due. Files are available in the homework section.
  • Fri Nov 13: Formal knowledge handoff. Builders transfer one project; receivers begin a different, only loosely related system. Handoff criteria
  • HW3 · opens Sun Nov 15; due Tue Nov 24 at 11:59 p.m.: receiver cold start + operations audit.

Week 5 · Nov 16–20

Search and operate inherited data

Week 6 · Nov 23–27

Find the real scaling boundary

  • Mon Nov 23: Scale-up versus scale-out, partitions, object storage, shuffle, skew, and failure. Slides · Source · Checkpoints
  • Tue Nov 24 at 11:59 p.m.: HW3 due. Files are available in the homework section.
  • Wed Nov 25: No class — Thanksgiving break.
  • HW4 · opens Sun Nov 29; due Tue Dec 8 at 11:59 p.m.: receiver extension + acceptance test.

Week 7 · Nov 30–Dec 4

Package, test, and explain

  • Mon Nov 30: Modules, src/, pyproject.toml, installable commands, and clean installs. Slides · Source · Checkpoints
  • Tue Dec 1: Professor McGovern project Parts I and II due.
  • Wed Dec 2: Receiver acceptance test and project readout rehearsal. Slides · Source · Checkpoints
  • Tue Dec 8 at 11:59 p.m.: HW4 due. Files are available in the homework section.

Two projects · one formal handoff

Build one system, receive the other

The projects are intentionally distinct and only loosely related. Before November 13, each team builds its assigned client system. After the handoff, the team must connect to, verify, improve, and explain the other system from the receiver’s point of view.

Climate risk and P&C insurance

  • PostgreSQL
  • NOAA Storm Events, HURDAT2, GHCN-Daily, and NWS
  • Weather risk, exposure, and underwriting questions

EV fleet siting and grid context

  • PostgreSQL or DuckDB
  • EIA, NREL AFDC, FHWA, eGRID, ACS, and geographic context
  • Charging gaps, scheduling, and site-priority questions

Reference files

Course documents and setup

Find setup instructions, course expectations, project requirements, and server documentation here.

Setup primer

Install VS Code, add Microsoft’s PostgreSQL extension, and connect to the planned Azure server.

Open setup primer

Course outline

The full Reinhart alignment, learning goals, meeting rhythm, assignment arc, and grading plan.

Open course outline

Project requirements

Builder evidence, receiver evidence, knowledge-transfer expectations, and acceptance criteria.

Open project requirements

Homework index

Release dates, due dates, formats, and the assignment sequence.

Open homework index