36-614 · Data Engineering and Distributed Environments

October 19–December 2, 2026 · Mondays and Wednesdays · 9:30–10:50 a.m. · Baker Hall 140A
Tools: PostgreSQL on Azure, Python, psycopg, VS Code, Git, and Quarto (.qmd)
Projects: two distinct client systems, followed by a formal knowledge handoff and receiver sprint

Course source and credit. The course builds on Alex Reinhart's MADS Computing course, especially its ten-part data-engineering sequence. The Fall 2026 version updates the tooling, reorganizes the SQL material, and connects the exercises to two client projects supplied by Professor McGovern in Data Engineering Assignment Instructions (July 6, 2026).

Course goals

By the end of the mini-course, students should be able to:

• design a relational schema from a client question and messy source data;
• write tested SQL with joins, aggregations, CTEs, windows, text search, and indexes;
• call PostgreSQL safely from Python using parameters and transactions;
• build an idempotent ingestion workflow with validation, logging, and a rerun plan;
• explain when cloud or distributed infrastructure is—and is not—justified;
• package pipeline code so another team can install and run it; and
• hand off a system, then operate a different inherited system from its documentation.

Class meetings

Preparation takes no more than 10 minutes. In class, short explanations alternate with two or three checkpoints where students predict, write, run, and explain code using the current project database. Some homework work begins during these checkpoints.

Schedule

The two projects and the handoff

The class runs two loosely related client projects, not one shared weather database. Each team builds one project through November 11. On November 13, teams exchange systems. The second stint begins from the receiver's point of view: students must connect, verify, extend, and explain a system they did not design.

Project A · Climate Risk Analytics & P&C Insurance Underwriting

• Database: PostgreSQL.
• Core sources: NOAA Storm Events, HURDAT2, GHCN-Daily, and the NWS API.
• Possible enrichments: Census ACS, Zillow ZHVI, Census Geocoder, and xView2.
• Client questions: flood-damage frequency versus severity; seasonal hail exposure; billion-dollar event trends; and the relationship between reported damage and harm to people.

Project B · Predictive EV Fleet Siting & Grid Capacity Optimization

• Database: PostgreSQL or DuckDB, with PostgreSQL preferred for the shared server.
• Core sources: EIA bulk power data, NREL AFDC, and FHWA freight data.
• Possible enrichments: EPA eGRID, Census ACS, and OpenStreetMap/TIGER geography.
• Client questions: charging gaps along freight corridors; 24-hour charging schedules; upgrade priorities; and high-traffic locations with plausible site context.

Feasibility note: the listed public data do not establish transformer or substation capacity, eGRID is not a real-time grid feed, and OSM footprints are not parcel acreage. Unless the client supplies authoritative capacity data, teams should label proxy measures clearly and must not claim to have measured unavailable capacity.

Assignments

The homework exercises adapt Reinhart's problems to the active project system. Work begun at an in-class checkpoint may be reused.

1. SQL foundations and schema audit — opens Oct 25; due Nov 3. Six tested queries, a result-grain statement, and a short schema/source audit on the builder database, plus one small comparison with an AI-drafted join.
2. Builder query library and handoff readiness — opens Nov 1; due Nov 12. Four client-facing queries, one safe Python entry point, rerun and failure tests, the first complete handoff pack, and one small AI documentation check.
3. Receiver cold start and operations audit — opens Nov 15; due Nov 24. Reproduce one documented result from the different inherited system, test a text query, and audit its cloud operations story.
4. Receiver extension and acceptance test — opens Nov 29; due Dec 8. Make one measured improvement, run the system from a clean-start condition, and record the remaining risks. PDF

Receiver assignments are submitted as pointers into the Professor McGovern project deliverables, not as duplicated copies of the same work.

Detailed prompts are released through the course website homework section.

Project evidence

Builder pack

• working database, source inventory, and source-to-table map;
• ER diagram and data dictionary with units, grain, and provenance;
• idempotent ingestion/refresh command with validation and failure behavior;
• at least eight non-trivial, client-relevant queries;
• a parameterized Python report or query entry point;
• runbook, access instructions, known issues, and recovery notes; and
• a concise live knowledge-transfer session.

Receiver pack

• cold-start log: what worked from the documentation alone;
• reproduction of one builder result and an explanation of any discrepancy;
• one new client question answered with SQL;
• one tested improvement to the inherited system; and
• reflection on what the team will change in its own future handoffs.

Assessment

Checkpoint participation is credit for making the work visible: submitting a prediction, query, test result, question, or explanation. Students are not graded on being the first to answer, and an absence can be made up with the checkpoint file.

Submission conventions

• Keep SQL in .sql, Python in .py, and reports in Quarto .qmd.
• AI use is limited to the small exercises that explicitly request it. Preserve the prompt and response, verify generated work with an executable test, and do not share credentials or restricted data with an AI system.
• Never commit credentials. Use environment variables or the course-approved secret mechanism.
• Every result must identify its grain, data source, and important exclusions.
• Code and documentation must be sufficient for the receiving team to run without private oral instructions.