Introduction to Data over Space and Time

Cosma Shalizi

25 August 2026, 36-740/620

\[ \newcommand{\Expect}[1]{\mathbb{E}\left[ #1 \right]} \newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]} \newcommand{\Cov}[1]{\mathrm{Cov}\left[ #1 \right]} \]

Data over Space and Time

Data over Space and Time

(From [https://www.cdc.gov/measles/data-research/index.html] on 2026-08-25)

Data over Space and Time

(From [https://www.cdc.gov/measles/data-research/index.html] on 2026-08-25)

Data over Space and Time

(From [https://www.cdc.gov/measles/data-research/index.html] on 2026-08-25)

Data over Space and Time

(From [https://www.cdc.gov/measles/data-research/index.html] on 2026-08-25)

Data over Space and Time

Special Statistical Issues

Course Mechanics

Office hours

Lectures

Assignments/Grading

  1. Weekly homework: 60%
  2. Oral final exam: 20%
  3. Scribing: 20% for 620, 10% for 740
  4. Developing a research question: 0% for 620, 10% for 740

Homework

Oral Final Exams

Scribing

Research question notes

Any questions?

What are the big issues?

Problems:

  1. Basic statistical theory is about independent, identically distributed (IID) data.
  2. But we (usually) only see one realization of a whole process.
  3. Every observation is dependent on every other observation.
  4. Basic statistical theory says that \(n=1\) and refuses to draw any inferences.

How are we going to deal with these issues?

Why is this worth knowing?

So what are we going to cover?

What we will not cover

To get started…

Cherry blossoms in Kyoto

Cherries at the Hirano shrine in Kyoto (David Montasco on flickr)

Flowering of cherry trees has been a central part of Japanese high art & culture for well over a millennium

Hanami

Kitao Shigemasa, Sangatsu, Asukayam Hanami = Third Lunar Month, Blossom Viewing at Asuka Hill, c. 1776, via Library of Congress

Notice the date in the title!

This is data!

Cherry blossoms track climate

Snow at the Hirano shrine (yopparainokobito on flickr)

A data set

A data set

Problems

Smoothed cherry blossoms

Both curves come from averaging observed values

More concrete problems

We’re going to need to build some concepts

By Thursday:

(Let me know if you haven’t gotten the invitations / links by 6 pm today)

Take-aways

Exercises (to think through, not hand in)

A first warm-up on statistical theory with dependence

Throughout, suppose \(\Expect{X(t)} = \mu\) for all \(t\), and \(\Var{X(t)} = \sigma^2 > 0\) for all \(t\). Define \(A_n = n^{-1} \sum_{t=1}^{n}{X(t)}\).

  1. (Baby’s first law of large numbers) \(\Cov{X(t), X(s)} = 0\) (unless \(t=s\)). Explain why each step of this proof is justified: \[\begin{eqnarray*} \Expect{A_n} & = & \frac{n\mu}{n} = \mu\\ \Var{A_n} & = & \frac{1}{n^2}\sum_{t=1}^{n}{\Var{X(t)}} = \frac{n \sigma^2}{n^2} = \frac{\sigma^2}{n}\\ \Expect{(A_n - \mu)^2} & = & \frac{\sigma^2}{n} \end{eqnarray*}\] We have shown that \(A_n \rightarrow \mu\); what is the sense of convergence?
  2. Suppose \(X(t) = \alpha + \beta X(t-1) + \epsilon(t)\), where \(\Expect{\epsilon(t)} = 0\), \(\Var{\epsilon(t)} = \rho^2\), and \(\Cov{\epsilon(t), \epsilon(s)} = 0\) (unless \(t=s\)). Prove the following:
    1. If \(\Expect{X(t)} = \mu\) for all \(t\), then \(\mu=\frac{\alpha}{1-\beta}\).
    2. If \(\Var{X(t)} = \sigma^2\) for all \(t\), then \(\sigma^2 = \frac{\rho^2}{1-\beta^2}\). Why does this require \(|\beta|<1\)?
    3. Assuming (2.1) and (2.2), \(\Cov{X(t), X(t+h)} = \sigma^2 \beta^h\).
  3. (Baby’s first ergodic theorem) Assume all the things from (2). Show that \(A_n \rightarrow \mu\) still, by showing that \(\Expect{(A_n - \mu)^2} \rightarrow \frac{\sigma^2}{n\tau}\), and find an explicit expression for \(\tau\)
    • For purists: \(\Expect{n(A_n-\mu)^2} \rightarrow \sigma^2/\tau\)

  1. Calvino’s Cosmicomics is a precious part of our common cultural heritage.↩

  2. The Pillow Book of Sei Shonagon is a precious part of our common cultural heritage.↩

  3. The traditional Japanese calendar system (from 645) didn’t have an accumulating count of years the way we do, but rather reckoned years by “name eras”, so a given year would be called something like “year \(k\) of the reign of Emperor So-and-so.” (Some emperors had more than one name era, and the name of the era was not the emperor’s name but one he chose, but that was the basic idea.) This is as though we called this year 2 of Trump II, called 2016 year 8 of Obama, etc. Part of the work of compiling data like this is to keep track of when, in our terms, each name era began. Japan in fact still has name eras for some official purposes (2026 is year 7 of the Reiwa era), but in 1873, as part of the Meiji Revolution, the government adopted the Gregorian calendar and the common era, the year-numbering scheme formerly known as AD/BC. (“Meiji” is itself an era name.) More-or-less similar schemes, where the count of years resets when the ruler changes, have been very common across the world; unending sequential year numbers seem to have been invented twice, by the Seleucid dynasty in what we now call the Middle East around 300 BCE, with remarkable consequences, and in Central America, in the form of the “long count” calendar used by the Mayans and other civilizations and reckoning days since 11 August 3114 BCE. (That calendar was certainly invented much more recently and we don’t know why that was their zero-day.)↩