Project: Data Pipeline

December 2 · Prove the inherited system is operable

MaDS Databases & SQL

One question will organize today

What evidence proves this is a transferred system—not merely a successful builder demo?

Course source and adaptation

Today’s acceptance test follows Alex Reinhart’s Project: Data Pipeline: schema design, runnable loaders, transactions, invalid-row handling, automated reporting, documentation, and paired handoff. Project requirements come from Professor McGovern’s July 6, 2026 brief.

By the end of class

You should be able to:

• evaluate a pipeline from a receiver’s evidence,
• distinguish reproducibility from a live demo,
• test schema, load, query, and documentation contracts,
• state client conclusions and evidence boundaries,
• rehearse a concise builder/receiver readout.

Two projects; two roles

stint 1: build one client system
          ↓ November 13 knowledge transfer
stint 2: operate and extend the other system

Today, lead with what you verified as receivers.

The acceptance path

access → install → connect → inspect → refresh
       → validate → reproduce → extend → explain

One screenshot cannot cover this path.

Evidence beats narration

Strong evidence includes:

• command and exit status,
• version and configuration names,
• row counts and date coverage,
• validation output,
• SQL and result grain,
• before/after plan or behavior,
• receiver timestamped cold-start log.

Checkpoint 1 · Cold start

From the handoff entry point, perform the smallest successful operation. Record only what the documentation told you before contacting builders.

Stop at the first undocumented dependency and classify it.

Classify transfer failures

Failure	Example
Access	role cannot see schema
Environment	dependency or secret name missing
Documentation	command/order ambiguous
Data contract	grain/unit/source unclear
Code	command fails in documented state
Evidence	result has no reproducible validation

Classification leads to a useful fix.

Reproduce one builder result

Require:

• exact query/report version,
• source snapshot or freshness time,
• expected grain,
• row count and date coverage,
• one domain validation,
• explanation of any discrepancy.

“Looks similar” is not reproduction.

Checkpoint 2 · Reproduction duel

Receiver runs one documented client result. Builder predicts its validation values before the run. Compare and explain every difference.

Schema acceptance

Can the receiver answer:

• What does one row mean in each important table?
• Which keys define identity?
• Which values are raw, cleaned, or derived?
• What are the units and geographic/time coverage?
• Which constraints encode truth?

Pipeline acceptance

Can the receiver:

• run a small safe refresh,
• repeat it without duplicate damage,
• find rejected records,
• see step-level status,
• recover from an intentional failure,
• know when results are publishable?

Query/report acceptance

Does each important result state:

• client question,
• query parameters,
• grain and denominator,
• important exclusions,
• source freshness,
• uncertainty or proxy boundary?

Project-specific truth tests

Climate: distinguish frequency from severity; account for missing/encoded damage; avoid treating all event records as comparable exposure.

EV: do not infer measured transformer/substation capacity from public proxy data; label eGRID timing and site-context limits.

A useful extension changes behavior

Receiver extension should have:

client need → baseline → change → acceptance test → result → limitation

The second-half topic is a means, not the accomplishment.

Checkpoint 3 · Five-minute readout

Give a five-minute receiver readout: client decision, cold-start result, reproduced evidence, extension, and one unresolved boundary.

Listeners write the first claim they could not independently verify.

Final presentation · recommended shape

1. Client and decision · 1 minute
2. What we built · 2 minutes
3. How it transferred · 2 minutes
4. What we inherited and reproduced · 3 minutes
5. Receiver extension · 2 minutes
6. Limit and next decision · 1 minute

Keep backup slides for schema and technical detail.

Demo design

Pre-stage:

• known-safe environment,
• smallest-success command,
• validation output,
• one client query,
• static backup evidence.

A backup is professionalism, not surrender.

What not to hide

• source data gaps,
• unverifiable proxy claims,
• manual steps,
• transfer failures,
• queries that changed after handoff,
• improvements not yet tested.

The course rewards honest system boundaries.

Final submission checklist

• Professor McGovern Parts I and II
• repository and artifact version
• ER diagram and data dictionary
• ingestion and validation evidence
• eight-query library and Python entry point
• runbook and known issues
• KT evidence and cold-start log
• receiver extension and acceptance test
• final client readout

The pattern to keep

receive cold
  → reproduce exactly
  → classify transfer failures
  → extend for a client need
  → test changed behavior
  → state limits
  → hand off better next time

The final product is not only a database. It is a system another team can understand, operate, and question.