Special Statistical Issues

The goal: learn how here and now relates to there and then
The two problems:
1. Everything depends on everything else $\Rightarrow$ basic theory doesn’t apply
2. We don’t get multiple samples of time or of space
- $\Rightarrow$ $n=1$, always
Nonetheless, we will see how to do statistical inference with dependent, spatio-temporal data

Course Mechanics

Syllabus: [http://www.stat.cmu.edu/~cshalizi/dst/20]
- All assignments will be posted there
- Information about readings and other course resources ditto
Canvas: turn in assignments, gradebook, some readings that can’t go on the public web
Piazza for question-answering

Homework

Every week, Thursdays at 6 pm (Pittsburgh)

After-Lecture Questions

Online through Canvas, the day after each class, short answers

Office hours

Me, W 2:00–3:00 by Zoom, Th 2:00–3:00 by Piazza
TAs, TBD

Textbooks

Gidon Eshel, Spatiotemporal Data Analysis

Required, but full text PDF through JSTOR

Textbooks

Peter Guttorp, Stochastic Modeling of Scientific Data

Required excerpts will be distributed (as PDFs) through Canvas
- It’s a good book and I recommend reading the whole thing, but $$$

Textbooks

Paul Teetor, The R Cookbook

Recommended; consult as needed

Lectures

Clarification, amplification, examples, alternatives
Complements to the readings
- $\therefore$ Do the readings ahead of time
Please ask questions
I will often ask you to solve some problem during lecture
- not graded, but
- students find the discussion of the solutions very helpful
- and more helpful if you’ve tried it yourself
Not recorded

Assignments/Grading

After-class review questions and exercises (10%)
Weekly homework (90%)

NO exams

After-class review questions and exercises

10% of your total grade
Specific questions about stuff we went over in class, and/or in the readings
Turn in through Canvas, the day after each class
- Due at 6 pm (Pittsburgh time) the next day so $> 24$ hr
Should take $\approx 10$ minutes
- But it’s untimed
Usually short-answer questions, may sometimes be multiple choice
Hand-written math OK as needed
- Scan or just take a picture with your phone and upload
- Dark ink on white paper works best
All equally weighted
Lowest 4 dropped, no questions asked

Homework

90% of your total grade
Data analysis & computing & a little theory
Must use R Markdown
Always turned in electronically via Canvas/Gradescope
Due at 6 pm on Thursday (Pittsburgh time) every week
- Except this first week (no homework)
- and Thanksgiving week (also no homework)
Lowest 3 dropped, no questions asked
- If you turn in all 13, and get at least 60% on each, lowest 4 dropped
NO LATE HOMEWORK FOR ANY REASON
- Turn in as many incomplete or rough versions as you like well ahead of the deadline

R Markdown

R Markdown is a straightforward way to combine text, math, code, and code output in one document
Enforces reproducibility: your numbers, figures, etc., are generated by the code, so results always match
Use is required for this class
- Your homework will lose points if you don’t use it
- You will lose more points as the semester goes on
- Resources for learning R Markdown on the syllabus page

Collaboration, Cheating & Plagiarism

Everything you turn in for a grade needs to be your own work, or an acknowledged borrowing from an approved source
Full policy for this class is part of the syllabus
Read that policy and the CMU policy on academic integrity
Turn in Homework 0 on these policies by Thursday
- Not graded, but you won’t get credit for any work until you complete it successfully

What are the big issues?

We see $X$ at time $t$ and point $r$, $X(r,t)$, and what to know how it relates to $Y$ at time $s$ and point $q$, $Y(q,s)$

Problems:

Basic statistical theory is about independent, identically distributed (IID) data.
But we (usually) only see one realization of a whole process.
Every observation is dependent on every other observation.
Basic statistical theory says that $n=1$ and refuses to draw any inferences.

How are we going to deal with these issues?

Methods for describing relationships and finding patterns in the data
Especially methods for predicting $X(r,t)$ from $Y(q,s)$
Incorporate dependence into statistical theory, so we can say when methods will work
Quantifying uncertain through modeling and simulation

Why is this worth knowing?

Italo Calvino, “All at One Point” (1963) imagines the world un-extended in time or space¹

Otherwise, every branch of science deals with data spread over space and time
Our examples will come from
- Geology
- Climatology
- Meteorology
- Ecology
- Epidemiology
- Demography
- Economics
Could (and might) add examples from neuroscience, physics, technology, policy, business, etc.

So what are we going to cover?

Exploratory methods for spatio-temporal data
Linear prediction
Inference with dependent data
Generative models
Simulations and simulation-based inference
Organized by method, skip around on subject matter

What we will not cover

ARIMA (etc.) models — take 36-618
Finance

To get started…

Something vivid, but more cheerful than Covid

Cherry blossoms in Kyoto

Cherries at the Hirano shrine in Kyoto (David Montasco on flickr)

Flowering of cherry trees has been a central part of Japanese high art & culture for well over a millennium

Hanami

Kitao Shigemasa, Sangatsu, Asukayam Hanami = Third Lunar Month, Blossom Viewing at Asuka Hill, c. 1776, via Library of Congress

Notice the date in the title!

This is data!

Ancient diaries², poetry, etc., and modern newspapers, record when cherry trees in Kyoto came into bloom
- Kyoto because it’s the ancient capital and has been a continuous seat of art & culture

Cherry blossoms track climate

Japan gets cold

Snow at the Hirano shrine (yopparainokobito on flickr)

Cherries only blossom when it gets warm enough
$\therefore$ The date when cherry trees are in full flower tells us about how warm the year was
- Date of first flowering is also informative, but less often recorded

A data set

Assembled by Prof. Yasuyuki Aono
- Data points going back to the early 800s
- Almost continuous for modern times
- Re-formatted version at [http://www.stat.cmu.edu/~cshalizi/dst/20/data/kyoto-2020.csv]
For each year, the day of the year (1–366) of full bloom
- April 1 = 91 or 92
- Aono had to search out the ancient records, poems, histories, etc.
- and convert dates before 1873 to our calendar³

Problems

How to fill in the data between the observations? (interpolation)
- When did the cherries bloom in 1015?
How to extend beyond the observations? (extrapolation)
- Make a guess for 2021, or for 800
How to remove measurement noise in the observations? (filtering)
- Poets can be sloppy about dates
How to separate year-to-year fluctuations from longer-term trends?
- How do we model fluctuations?
- How do we model trends?
How seriously should we treat the warming since about 1800? (inference)
- Could it be an artifact of denser observations?
- Does the climate just do stuff like this occasionally?
- How long do we need to wait to be confident?

Smoothed cherry blossoms

Both curves come from averaging observed values

We’re going to need to build some concepts

Stochastic process = collection of random variables over time or space or both, typically dependent, say $X(t)$
Trend = central tendency of the process, say $\mu(t) = \mathbb{E}[X(t)]$
Fluctuations = difference from the trend
How can we figure out the trend if we just see $X$ once?

By Thursday:

Make sure you’re on Canvas & Piazza for the course
- Let me know if not
Homework 0
- Read the course policy on collaboration, cheating and plagiarism
- Read the university policy on academic integrity
- Read the excerpt from Taurabian’s Manual for Writers (Canvas)
- Homework 0 isn’t graded but you need to do it to get a grade for anything else

Take-aways

Everything is statistically dependent on everything else
Dependence means what happens here and now gives us information about there and then, so we can predict
Dependence means the statistical theory we’ve learned needs to be fixed
For now, we will focus on describing dependence
Next time: EDA and graphics

Calvino’s Cosmicomics is a precious part of our common cultural heritage.↩
The Pillow Book of Sei Shonagon is a precious part of our common cultural heritage.↩
The traditional Japanese calendar system (from 645) didn’t have an accumulating count of years the way we do, but rather reckoned years by “name eras”, so a given year would be called something like “year $k$ of the reign of Emperor So-and-so.” (Some emperors had more than one name era, and the name of the era was not the emperor’s name but one he chose, but that was the basic idea.) This is as though we called this year 3 of Trump, called 2016 year 8 of Obama, etc. Part of the work of compiling data like this is to keep track of when, in our terms, each name era began. Japan in fact still has name eras for some official purposes (this is year 3 of the Reiwa era), but in 1873, as part of the Meiji Revolution, the government adopted the Gregorian calendar and the common era, the year-numbering scheme formerly known as AD/BC. (“Meiji” is itself an era name.) More-or-less similar schemes, where the count of years resets when the ruler changes, have been very common across the world; unending sequential year numbers seem to have been invented twice, by the Seleucid dynasty in what we now call the Middle East around 300 BCE, with remarkable consequences, and in Central America, in the form of the “long count” calendar used by the Mayans and other civilizations and reckoning days since 11 August 3114 BCE. (The calendar was certainly invented much more recently and we don’t know why that was their zero-day.)↩

Introduction to the Course

Data over Space and Time

Data over Space and Time

Data over Space and Time

Data over Space and Time

Data over Space and Time

Special Statistical Issues

Course Mechanics

Homework

After-Lecture Questions

Office hours

Textbooks

Textbooks

Textbooks

Lectures

Assignments/Grading

After-class review questions and exercises

Homework

R Markdown

Collaboration, Cheating & Plagiarism

Any questions?

What are the big issues?

How are we going to deal with these issues?

Why is this worth knowing?

So what are we going to cover?

What we will not cover

To get started…

Cherry blossoms in Kyoto

Hanami

This is data!

Cherry blossoms track climate

A data set

A data set

Problems

Smoothed cherry blossoms

More concrete problems

We’re going to need to build some concepts

By Thursday:

Take-aways