---
title: "Visualizations for 2D Categorical and 1D Quantitative Data"
subtitle: "Lecture 3"
author: "Shannon Gallagher"
date: "August 31, 2026"
date-format: long
format:
  revealjs:
    theme: ../mads-36613-assignments/lecture_drafts/theme.scss
    slide-number: c/t
    chalkboard: true
    code-fold: show
    code-line-numbers: true
    smaller: true
    linestretch: 1.25
    fig-width: 8
    fig-height: 4.5
    footer: "MaDS Data Visualization · Fall 2026"
execute:
  echo: true
  warning: false
  message: false
---

```{r setup}
#| include: false
source("_setup.R")
```

## Last time and today

**Last time:**

- map variables to aesthetics in `ggplot2`,
- build and interpret plots for one categorical variable.

**Today:**

- choose the denominator when comparing two categorical variables,
- use residuals to find where an association occurs,
- distinguish discrete and continuous quantitative variables,
- make a first honest display of a skewed quantity.

## 2D categorical basics: marginal / conditional distribution

For a table of event type by state:

- row percentages ask about the mix **within an event type**,
- column percentages ask about the mix **within a state**,
- joint percentages ask where all records occur.

There is no context-free “percent.”

## Connecting distributions to visualizations

```{r denominators}
state_type <- events |>
  filter(state %in% c("Texas", "Florida", "California", "Pennsylvania"), event_type %in% top_types) |>
  count(state, event_type)

state_type |>
  group_by(state) |>
  mutate(within_state = n / sum(n)) |>
  arrange(state, desc(within_state)) |>
  slice_head(n = 2)
```

## Stacked bar charts—a bar chart of spine charts

```{r stacked-state-type, fig.alt="Stacked bar chart of common NOAA event types in four states."}
state_type |>
  ggplot(aes(state, n, fill = event_type)) +
  geom_col() +
  labs(title = "Totals and composition are visible together", x = NULL, y = "Event records", fill = "Event type")
```

The total bar height shows the marginal distribution of state. Segments show the joint
counts, but only the bottom segment has a common baseline.

## Stacked bar charts with conditional proportions

```{r proportional-state-type, fig.alt="Proportional stacked bar chart of event types within four states."}
state_type |>
  ggplot(aes(state, n, fill = event_type)) +
  geom_col(position = "fill") +
  scale_y_continuous(labels = percent) +
  labs(title = "Now each bar answers: within this state, what is the mix?", x = NULL, y = "Within-state share", fill = "Event type")
```

## Side-by-side bar charts

```{r dodged-state-type, fig.alt="Grouped bar chart comparing event type counts across four states."}
state_type |>
  ggplot(aes(event_type, n, fill = state)) +
  geom_col(position = position_dodge(preserve = "single")) +
  coord_flip() +
  labs(title = "A common baseline helps compare states within an event type", x = NULL, y = "Event records", fill = "State")
```

## [Complete](https://tidyr.tidyverse.org/reference/complete.html) missing values to preserve location

```{r complete-state-type}
state_type |>
  complete(state, event_type, fill = list(n = 0)) |>
  arrange(state, event_type) |>
  slice_head(n = 12)
```

An absent row and a true zero are not automatically the same. Complete the grid only
after deciding what a missing combination means.

## Visualize independence tests with mosaic plots

```{r mosaic-state-type, fig.alt="Mosaic plot of state and event type shaded by Pearson residuals."}
#| echo: false
#| fig-width: 10
#| fig-height: 5.8
#| out-width: "100%"
tab <- state_type |>
  pivot_wider(names_from = event_type, values_from = n, values_fill = 0) |>
  tibble::column_to_rownames("state") |>
  as.matrix()

mosaicplot(
  tab,
  shade = TRUE,
  las = 2,
  main = "State and event type: area shows counts; shading shows residuals"
)
```

## Checkpoint 1 · Name the denominator

::: {.checkpoint}
You are comparing Pennsylvania with Texas. Write one question that needs counts and one
that needs within-state percentages. What misleading conclusion could arise from using
the wrong denominator?
:::

## A chi-square test compares observed and expected counts

The null hypothesis says the two categorical variables are independent.

For cell $(i,j)$, independence predicts

$$
E_{ij}=\frac{(\text{row }i\text{ total})(\text{column }j\text{ total})}{\text{grand total}}.
$$

## Pearson residuals are the pieces of chi-square

The **Pearson residual** standardizes the observed-minus-expected discrepancy for one
cell:

$$
r_{ij}=\frac{O_{ij}-E_{ij}}{\sqrt{E_{ij}}}.
$$

The chi-square statistic adds their squared values:

$$
X^2=\sum_i\sum_j r_{ij}^2
=\sum_i\sum_j\frac{(O_{ij}-E_{ij})^2}{E_{ij}}.
$$

The sign shows whether a cell is above or below expectation; the magnitude shows how
strongly that cell contributes to the overall result.

## A chi-square result is only a starting point

It does not tell us:

- which cells drive the result,
- whether the difference matters,
- why the association exists,
- whether the records represent the underlying hazard process fairly.

## Shade by *Pearson residuals*

```{r categorical-residuals}
tab <- state_type |>
  pivot_wider(names_from = event_type, values_from = n, values_fill = 0) |>
  tibble::column_to_rownames("state") |>
  as.matrix()

round(chisq.test(tab)$residuals[, 1:4], 1)
```

Large positive residuals mean more records than independence predicts; large negative
residuals mean fewer.

## 1D Quantitative Data

Quantitative variables encode amounts.

- **Discrete:** counts such as injuries or deaths.
- **Continuous:** duration, hail size, wind speed, or damage amount.

The stored type in R is not enough. Meaning comes from the data-generating process.

## NOAA damage is not analysis-ready

```{r damage-fields}
events |>
  select(damage_property_raw, prop_usd, prop_usd_zero) |>
  filter(!is.na(damage_property_raw)) |>
  distinct() |>
  slice_head(n = 8)
```

Missing damage and zero damage are different claims.

## A summary can hide the distribution

```{r damage-summary}
events |>
  filter(!is.na(prop_usd)) |>
  summarise(
    observed_reports = n(),
    zero_reports = sum(prop_usd == 0),
    zero_share = percent(mean(prop_usd == 0), accuracy = 0.1),
    median_all = median(prop_usd),
    median_positive = median(prop_usd[prop_usd > 0], na.rm = TRUE),
    largest = max(prop_usd, na.rm = TRUE)
  )
```

## A boxplot alone hides the distribution

```{r damage-boxplot, fig.alt="Sampled non-missing damage reports, including zeros, shown as jittered points with a boxplot calculated from all non-missing damage reports."}
#| echo: false
#| fig-height: 5.4
#| out-width: "100%"
observed_damage <- events |>
  filter(!is.na(prop_usd))

set.seed(36613)
damage_point_sample <- observed_damage |>
  slice_sample(n = min(600, nrow(observed_damage)))

observed_damage |>
  ggplot(aes(x = log10(prop_usd + 1), y = "")) +
  geom_boxplot(
    width = 0.22,
    outlier.shape = NA,
    fill = NA,
    color = deep_gold,
    linewidth = 1
  ) +
  geom_jitter(
    data = damage_point_sample,
    height = 0.12,
    width = 0,
    alpha = 0.18,
    size = 1.5,
    color = blue_gray
  ) +
  scale_x_continuous(
    breaks = c(0, 3, 6, 9),
    labels = c("$0", "$1K", "$1M", "$1B")
  ) +
  labs(
    title = "Observed zeros dominate the distribution",
    subtitle = "600 points shown; boxplot calculated from all non-missing reports",
    x = "Property damage, displayed as log10(dollars + 1)",
    y = NULL
  )
```

## An ECDF answers threshold questions

The empirical cumulative distribution function at value $x$ is

$$
\widehat{F}(x) = \frac{\#\{X_i \le x\}}{n}.
$$

Read it as: **the share of observations at or below $x$.**

Unlike a histogram, an ECDF has no bins to choose.

## The jump at $0 is a feature, not a nuisance

```{r damage-ecdf, fig.alt="Empirical cumulative distribution of non-missing NOAA property damage, including zeros, on a log10 damage-plus-one scale."}
#| echo: false
#| fig-height: 6
#| out-width: "100%"
events |>
  filter(!is.na(prop_usd)) |>
  ggplot(aes(log10(prop_usd + 1))) +
  stat_ecdf(color = deep_gold, linewidth = 1) +
  scale_x_continuous(
    breaks = c(0, 3, 6, 9),
    labels = c("$0", "$1K", "$1M", "$1B")
  ) +
  scale_y_continuous(labels = percent) +
  labs(
    subtitle = "Observed zero values retained; missing damage excluded",
    x = "Property damage, displayed as log10(dollars + 1)",
    y = "Cumulative share of reports"
  )
```

## What about comparing to theoretical distributions?

A one-sample Kolmogorov–Smirnov statistic measures the largest vertical distance
between an empirical cumulative distribution and a specified theoretical CDF:

$$
D_n = \sup_x |F_n(x)-F_0(x)|.
$$

The test is about a fully specified distribution, not merely whether a histogram looks
roughly bell-shaped.

## Checkpoint 2 · Predict the shape

::: {.checkpoint}
Before plotting, sketch the distribution of non-missing property damage. What feature
should appear at $0? Where will the mean fall relative to the median? What will adding
one and taking a log reveal?
:::

## Histograms display 1D continuous distributions

```{r first-damage-hist, fig.alt="Histogram of non-missing property damage, including zeros, after a log10 damage-plus-one transformation."}
#| echo: false
#| fig-height: 5.5
#| out-width: "100%"
events |>
  filter(!is.na(prop_usd)) |>
  ggplot(aes(log10(prop_usd + 1))) +
  geom_histogram(binwidth = 0.25, boundary = 0, fill = gold, color = "white") +
  scale_x_continuous(
    breaks = c(0, 3, 6, 9),
    labels = c("$0", "$1K", "$1M", "$1B")
  ) +
  labs(
    title = "Zero damage is the dominant feature",
    subtitle = "Observed zeros retained; only missing damage values excluded",
    x = "Property damage, displayed as log10(dollars + 1)",
    y = "Event records"
  )
```

## Why transform?

A $\log_{10}(\text{damage}+1)$ transform keeps zero visible while making multiplicative
differences readable:

- 0 represents $0,
- 3 is approximately $1,000,
- 6 is approximately $1,000,000,
- 9 is approximately $1,000,000,000.

The transform changes the visual scale, not the archived values. Missing damage remains
missing rather than being recoded to zero.

## Checkpoint 3 · Write the caption

::: {.checkpoint}
Write a two-sentence caption for the histogram: one sentence describing the main pattern
and one sentence explaining how zero and missing values were handled. Avoid saying the data are
“normal.”
:::

## Recap and next steps

1. Percentages require a named denominator.
2. A test detects association; residuals help locate it.
3. A quantitative summary is not a distribution.
4. Cleaning and scale choices are part of the claim.

Next: choosing and tuning distribution displays, then comparing distributions across groups.