---
title: "Project: Data Pipeline"
subtitle: "December 2 · Prove the inherited system is operable"
author: "MaDS Databases & SQL"
format:
  revealjs:
    theme: [default, mads-sql-reveal.scss]
    slide-number: c/t
    chalkboard: true
    code-line-numbers: true
    transition: fade
    footer: "Adapted from Alex Reinhart · MADS Computing"
---

## One question will organize today

::: {.question}
What evidence proves this is a transferred system—not merely a successful builder demo?
:::

## Course source and adaptation

::: {.source-note}
Today's acceptance test follows Alex Reinhart's [Project: Data Pipeline](https://www.refsmmat.com/courses/msp-computing/data-engineering/project.html): schema design, runnable loaders, transactions, invalid-row handling, automated reporting, documentation, and paired handoff. Project requirements come from Professor McGovern's July 6, 2026 brief.
:::

## By the end of class

You should be able to:

- evaluate a pipeline from a receiver's evidence,
- distinguish reproducibility from a live demo,
- test schema, load, query, and documentation contracts,
- state client conclusions and evidence boundaries,
- rehearse a concise builder/receiver readout.

## Two projects; two roles

```text
stint 1: build one client system
          ↓ November 13 knowledge transfer
stint 2: operate and extend the other system
```

Today, lead with what you verified as receivers.

## The acceptance path

```text
access → install → connect → inspect → refresh
       → validate → reproduce → extend → explain
```

One screenshot cannot cover this path.

## Evidence beats narration

Strong evidence includes:

- command and exit status,
- version and configuration names,
- row counts and date coverage,
- validation output,
- SQL and result grain,
- before/after plan or behavior,
- receiver timestamped cold-start log.

## Checkpoint 1 · Cold start

::: {.checkpoint}
From the handoff entry point, perform the smallest successful operation. Record only what the documentation told you before contacting builders.
:::

Stop at the first undocumented dependency and classify it.

## Classify transfer failures

| Failure | Example |
|---|---|
| Access | role cannot see schema |
| Environment | dependency or secret name missing |
| Documentation | command/order ambiguous |
| Data contract | grain/unit/source unclear |
| Code | command fails in documented state |
| Evidence | result has no reproducible validation |

Classification leads to a useful fix.

## Reproduce one builder result

Require:

- exact query/report version,
- source snapshot or freshness time,
- expected grain,
- row count and date coverage,
- one domain validation,
- explanation of any discrepancy.

“Looks similar” is not reproduction.

## Checkpoint 2 · Reproduction duel

::: {.checkpoint}
Receiver runs one documented client result. Builder predicts its validation values before the run. Compare and explain every difference.
:::

## Schema acceptance

Can the receiver answer:

- What does one row mean in each important table?
- Which keys define identity?
- Which values are raw, cleaned, or derived?
- What are the units and geographic/time coverage?
- Which constraints encode truth?

## Pipeline acceptance

Can the receiver:

- run a small safe refresh,
- repeat it without duplicate damage,
- find rejected records,
- see step-level status,
- recover from an intentional failure,
- know when results are publishable?

## Query/report acceptance

Does each important result state:

- client question,
- query parameters,
- grain and denominator,
- important exclusions,
- source freshness,
- uncertainty or proxy boundary?

## Project-specific truth tests

**Climate:** distinguish frequency from severity; account for missing/encoded damage; avoid treating all event records as comparable exposure.

**EV:** do not infer measured transformer/substation capacity from public proxy data; label eGRID timing and site-context limits.

## A useful extension changes behavior

Receiver extension should have:

```text
client need → baseline → change → acceptance test → result → limitation
```

The second-half topic is a means, not the accomplishment.

## Checkpoint 3 · Five-minute readout

::: {.checkpoint}
Give a five-minute receiver readout: client decision, cold-start result, reproduced evidence, extension, and one unresolved boundary.
:::

Listeners write the first claim they could not independently verify.

## Final presentation · recommended shape

1. **Client and decision** · 1 minute
2. **What we built** · 2 minutes
3. **How it transferred** · 2 minutes
4. **What we inherited and reproduced** · 3 minutes
5. **Receiver extension** · 2 minutes
6. **Limit and next decision** · 1 minute

Keep backup slides for schema and technical detail.

## Demo design

Pre-stage:

- known-safe environment,
- smallest-success command,
- validation output,
- one client query,
- static backup evidence.

A backup is professionalism, not surrender.

## What not to hide

- source data gaps,
- unverifiable proxy claims,
- manual steps,
- transfer failures,
- queries that changed after handoff,
- improvements not yet tested.

The course rewards honest system boundaries.

## Final submission checklist {.smaller}

- Professor McGovern Parts I and II
- repository and artifact version
- ER diagram and data dictionary
- ingestion and validation evidence
- eight-query library and Python entry point
- runbook and known issues
- KT evidence and cold-start log
- receiver extension and acceptance test
- final client readout

## The pattern to keep

```text
receive cold
  → reproduce exactly
  → classify transfer failures
  → extend for a client need
  → test changed behavior
  → state limits
  → hand off better next time
```

The final product is not only a database. It is a system another team can understand, operate, and question.