36-614 · Data Engineering and Distributed Environments
This student-facing syllabus is the concise companion to the full course outline.
Instructor: Shannon Gallagher
(sgallagh@stat.cmu.edu)
Office: 129D Baker Hall
Office Hours: Mondays, 11:00 a.m.–noon, and Tuesdays,
1:00–2:00 p.m., in Baker Hall 129D
Course in one sentence
Build and query one client database, hand it to another team, then prove you can operate and improve a different inherited system.
Source and credit
The course follows Alex Reinhart's MADS Computing course. Reinhart's ten data-engineering chapters and exercises are the instructional backbone; the Fall 2026 sequence updates the tooling and applies the work to client databases. The two client briefs are from Professor McGovern's Data Engineering Assignment Instructions (July 6, 2026).
Meetings and preparation
We meet Mondays and Wednesdays, 9:30–10:50 a.m., in Baker Hall 140A, from October 19 through December 2. There is no class November 25. Preparation is capped at 10 minutes. Meetings contain two or three short checkpoints interleaved with instruction; checkpoint work can start the homework.
Tools
VS Code, Azure Database for PostgreSQL, Python with
psycopg, Git, and Quarto. Start with the VS Code and PostgreSQL setup
primer.
Schedule
| Date | Topic |
|---|---|
| Oct 19, 21, 26 | SQL Basics, split across three meetings |
| Oct 28 | Database Fundamentals |
| Nov 2, 4 | Advanced SQL, split across two meetings |
| Nov 3 | Homework 1 due at 11:59 p.m. |
| Nov 9 | The Data Pipeline |
| Nov 11 | Using SQL from Code |
| Nov 12 | Homework 2 due at 11:59 p.m. |
| Nov 13 | Formal project handoff |
| Nov 16 | Full Text Search |
| Nov 18 | Cloud Computing |
| Nov 23 | Distributed Data and Computation |
| Nov 24 | Homework 3 due at 11:59 p.m. |
| Nov 25 | No class |
| Nov 30 | Packaging Code |
| Dec 1 | Professor McGovern project Parts I and II due |
| Dec 2 | Project: Data Pipeline; receiver acceptance test |
| Dec 8 | Homework 4 due at 11:59 p.m. |
| Finals week | Presentations |
Projects
- Climate Risk Analytics & P&C Insurance Underwriting: PostgreSQL with NOAA Storm Events, HURDAT2, GHCN-Daily, NWS, and selected socioeconomic enrichments.
- Predictive EV Fleet Siting & Grid Capacity Optimization: PostgreSQL or DuckDB with EIA, NREL AFDC, FHWA freight, eGRID, ACS, and OSM/TIGER context.
Each team is first a builder and then a receiver. The two systems are distinct and only loosely related. Claims must match available evidence; proxy measures must be labeled.
Grading
| Component | Weight |
|---|---|
| In-class checkpoints and participation | 10% |
| Four assignments | 40% |
| Professor McGovern project Parts I and II | 35% |
| Knowledge transfer and receiver readout | 15% |
Participation means contributing visible work—a prediction, query, test, question, or explanation—not speaking fastest. Missed checkpoint work can be made up.
Working rules
- Do not commit passwords, tokens, or connection strings containing secrets.
- AI use begins as a small, explicit pilot. Use it only where an assignment invites it; submit the prompt and response, verify the result yourself, and never share credentials or restricted data with an AI system.
- State the grain, provenance, units, and exclusions behind every important result.
- Use parameters for values sent from Python to SQL.
- Test reruns and partial failures; a pipeline is not complete because it ran once.
- Write the handoff for a receiver who was not in the room.