Visualizations for Text Data

Lecture 11

Shannon Gallagher

September 30, 2026

source("_setup.R")
library(tidytext)

narratives <- events |>
  filter(!is.na(event_narrative), event_narrative != "") |>
  select(event_id, event_type, event_narrative)

Reminders, previously, and today…

• define a document and tokenization unit,
• build and critique a bag-of-words representation,
• compare groups with term frequency and TF-IDF,
• understand why sentiment dictionaries are not automatic insight.

Working with raw text data

For NOAA Storm Events, a document might be:

• one event narrative,
• all narratives for an event type,
• all narratives for a state-month,
• one episode narrative.

The choice determines what “frequent” and “distinctive” mean.

Tokenize text into long format

tokens <- narratives |>
  unnest_tokens(word, event_narrative) |>
  anti_join(stop_words, by = "word") |>
  filter(str_detect(word, "^[a-z]+$"), str_length(word) > 2)

tokens |> slice_head(n = 8)

# A tibble: 8 × 3
  event_id event_type word     
     <dbl> <chr>      <chr>    
1  1150291 Drought    drought  
2  1150291 Drought    severity 
3  1150291 Drought    decreased
4  1150291 Drought    severe   
5  1150291 Drought    moderate 
6  1150291 Drought    late     
7  1150291 Drought    january  
8  1150292 Drought    drought  

Now one row is one token occurrence, not one storm event.

Bag of Words representation of text

A document-term matrix has one row per document, one column per term, and a count or weight in each cell.

tokens |>
  filter(event_type %in% c("Flash Flood", "Tornado", "Hail")) |>
  count(event_type, word) |>
  group_by(event_type) |>
  slice_max(n, n = 5, with_ties = FALSE) |>
  ungroup() |>
  pivot_wider(names_from = word, values_from = n, values_fill = 0)

# A tibble: 3 × 14
  event_type  flooding  road water reported   due  hail  size report quarter
  <chr>          <int> <int> <int>    <int> <int> <int> <int>  <int>   <int>
1 Flash Flood     2616  2321  2080     1783  1134     0     0      0       0
2 Hail               0     0     0     2155     0  4991  2194   1569    1547
3 Tornado            0  2760     0        0     0     0     0      0       0
# ℹ 4 more variables: tornado <int>, damage <int>, county <int>, trees <int>

The representation is useful for comparison but cannot recover word order or syntax.

Remove stop words

Common function words often overwhelm a frequency table, so analysts remove a stop-word list. But domain-specific meaning can be lost:

• negation words such as “not”;
• timing words such as “before” and “after”;
• warning language such as “under” or “until.”

Inspect the removed terms rather than treating the list as universally correct.

Apply stemming

Stemming reduces related forms to a common root, such as damage, damaged, and damaging. This can improve counting, but the stem may not be a real word and unrelated terms can be merged.

Lemmatization uses vocabulary and grammar to return dictionary forms, but requires more language modeling.

Checkpoint 1 · Name the lost information

After converting narratives to individual words, list three kinds of information that have been discarded. Which loss matters most for interpreting warnings or impacts?

Create a word cloud using term frequencies

tokens |>
  count(word, sort = TRUE) |>
  slice_head(n = 15) |>
  mutate(word = fct_reorder(word, n)) |>
  ggplot(aes(n, word)) +
  geom_col(fill = gold) +
  scale_x_continuous(labels = comma) +
  labs(title = "Common words mostly describe generic event mechanics", x = "Token occurrences", y = NULL)

Frequency is not importance

Frequent terms can be:

• boilerplate,
• documentation conventions,
• location language,
• common meteorological vocabulary.

Counts describe the corpus; interpretation requires context.

Comparison clouds

Word clouds make exact frequency and group comparison difficult. Font size is imprecise, placement is arbitrary, and long words attract attention.

A sorted bar chart is usually easier to audit.

A word cloud encodes frequency with area

if (requireNamespace("wordcloud", quietly = TRUE)) {
  cloud_terms <- tokens |>
    count(word, sort = TRUE) |>
    slice_head(n = 100)
  wordcloud::wordcloud(cloud_terms$word, cloud_terms$n, max.words = 100,
                       colors = c(deep_gold, blue_gray, charcoal), random.order = FALSE)
} else {
  plot.new()
  text(0.5, 0.5, "Install wordcloud to render this example")
}

Use it as an exploratory overview, not as the primary evidence for a ranked comparison.

Comparison needs a common scale

tokens |>
  filter(event_type %in% c("Tornado", "Flash Flood")) |>
  count(event_type, word) |>
  group_by(event_type) |>
  mutate(rate = n / sum(n)) |>
  slice_max(rate, n = 10, with_ties = FALSE) |>
  ungroup() |>
  mutate(word = reorder_within(word, rate, event_type)) |>
  ggplot(aes(rate, word, fill = event_type)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~ event_type, scales = "free_y") +
  scale_y_reordered() +
  scale_x_continuous(labels = percent) +
  labs(title = "Within-group rates support a direct vocabulary comparison", x = "Share of tokens", y = NULL)

TF-IDF weighting

For term \(t\) and document/group \(d\):

\[ \mathrm{tfidf}(t,d)=\mathrm{tf}(t,d)\log\left(\frac{N}{\mathrm{df}(t)}\right) \]

Terms score highly when common in one group but uncommon across groups.

Top words by TF-IDF for each NOAA event type

tfidf_terms <- tokens |>
  filter(event_type %in% c("Flash Flood", "Tornado", "Hail")) |>
  count(event_type, word) |>
  bind_tf_idf(word, event_type, n) |>
  group_by(event_type) |>
  slice_max(tf_idf, n = 8, with_ties = FALSE) |>
  ungroup()

tfidf_terms |>
  mutate(word = reorder_within(word, tf_idf, event_type)) |>
  ggplot(aes(tf_idf, word, fill = event_type)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~event_type, scales = "free") +
  scale_y_reordered() +
  labs(title = "TF-IDF surfaces language distinctive to each event type", x = "TF-IDF", y = NULL)

Checkpoint 2 · Audit a distinctive term

Choose one high-TF-IDF word. Read three source narratives containing it. Does the word mean what the chart seems to imply? Report one context the token display hid.

Sentiment Analysis

A lexicon maps words to labels or scores. It does not understand:

• negation,
• technical language,
• irony,
• narrative structure,
• the severity of the actual event.

“Fatal” may be informative, but a sentiment score is not a measure of harm.

NOAA narrative sentiment analysis

sentiment_terms <- tokens |>
  filter(event_type %in% c("Tornado", "Flash Flood", "Hail")) |>
  inner_join(get_sentiments("bing"), by = "word") |>
  count(event_type, sentiment)

sentiment_terms |>
  group_by(event_type) |>
  mutate(share = n / sum(n)) |>
  ggplot(aes(event_type, share, fill = sentiment)) +
  geom_col(position = "dodge") +
  scale_y_continuous(labels = percent) +
  labs(title = "Lexicon matches describe words, not event severity", x = NULL, y = "Share of matched tokens", fill = NULL)

Other functions of text

narratives |>
  unnest_tokens(bigram, event_narrative, token = "ngrams", n = 2) |>
  separate_wider_delim(bigram, delim = " ", names = c("word1", "word2")) |>
  filter(!word1 %in% stop_words$word, !word2 %in% stop_words$word) |>
  count(word1, word2, sort = TRUE) |>
  slice_head(n = 10)

# A tibble: 10 × 3
   word1        word2       n
   <chr>        <chr>   <int>
 1 wind         gust     5679
 2 wind         gusts    3183
 3 power        lines    2004
 4 heat         index    1721
 5 size         hail     1638
 6 60           mph      1613
 7 mesonet      station  1582
 8 peak         wind     1562
 9 thunderstorm winds    1488
10 tree         damage   1434

Bigrams preserve adjacent word pairs, helping distinguish phrases such as “flash flood” or “power lines” that single tokens separate.

Better text questions

Instead of “Are narratives positive or negative?”, ask:

• Which consequences are described for different hazards?
• Which warning or response terms co-occur with injuries?
• How does documentation language vary by office or event type?
• Which narratives need qualitative review?

Checkpoint 3 · Design a text audit

Choose one text-based claim. Define the document, token/feature, comparison group, and a manual validation sample that would make the claim credible.

Recap and next steps

• Tokenization creates a new observational unit.
• Frequency, distinctiveness, and sentiment are different quantities.
• Text graphics need source-text validation.
• Simple, sortable comparisons beat decorative word clouds.

Next: animation, interaction, and choosing the right medium.