Environment

It would be impractical to run this notebook without GPU acceleration. If you do not have access to a GPU, opt for a 15-20% subset of the original validation set prior to creating the short and medium compressed datasets later in the post. Alternatively, check out Google Colab, which provides a limited GPU runtime for their managed notebook instances free of charge.

To match the environment used in throughout this notebook, install the dependencies specified in the following requirements.txt. Note that transformers>=5.0 and torch>2.5 are required:

!python -m pip install -r https://gist.githubusercontent.com/jtrive84/e313afbf2def24687e3c3247aa836fe9/raw/f2493005d291920dc1fcfd54274b9dfa42004ebe/requirements.txt --quiet

If an NVIDIA GPU is available, install the CUDA-enabled PyTorch wheel instead of the default CPU build. This is only necessary if installing the dependencies on your own machine. GPU instances in Sagemaker, Colab and the NVIDIA RAPIDS Docker images already have CUDA-enabled PyTorch available:

!python -m pip install -r https://gist.githubusercontent.com/jtrive84/bc9dad844ab927357f0c35ea71f33963/raw/01d84194f9ad59f3b265e45d7ff39cefe1386673/requirements-gpu.txt --quiet

If the CUDA-enabled PyTorch wheel was installed, the torch version number should have a +cu*** suffix:

import torch

print(f"torch version: {torch.__version__}")
print(f"cuda version : {torch.version.cuda}")

torch version: 2.6.0+cu124
cuda version : 12.4

To keep this notebook as self-contained as possible, the next few cells walk through retrieving and reproducing the train–validation splits used in Parts I and II of the series.

import os
import pandas as pd
import requests
from bs4 import BeautifulSoup

base_url = "https://www.ncei.noaa.gov/pub/data/swdi/stormevents/csvfiles/"


def get_latest_details_file(year):
    """
    Return the filename of the latest details file for the given year.
    Sample filename: StormEvents_details-ftp_v1.0_d2022_c20250721.csv.gz
    """
    html = requests.get(base_url).text
    soup = BeautifulSoup(html, "html.parser")
    candidates = [
        a["href"]
        for a in soup.find_all("a", href=True)
        if f"StormEvents_details-ftp_v1.0_d{year}" in a["href"]
    ]

    return sorted(candidates)[-1]


# Check if dataset exists locally prior to hitting NOAA servers.
if not os.path.exists("noaa-events-2008-2025.parquet"):
    
    print("Fetching latest filenames from NOAA servers...")
    # Get latest filenames for years 2008-2025.
    latest_filenames = [get_latest_details_file(year) for year in range(2008, 2026)]

    # Load each file into a DataFrame and concatenate.
    dfall = pd.concat(
        [pd.read_csv(f"{base_url + fname}", compression="gzip") for fname in latest_filenames],
        ignore_index=True,
    )
    dfall.to_parquet("noaa-events-2008-2025.parquet")
else:
    print("Loading dataset locally...")
    dfall = pd.read_parquet("noaa-events-2008-2025.parquet")

print(f"Total records loaded: {dfall.shape[0]:,}")
dfall.head()

Loading dataset locally...
Total records loaded: 1,161,550

Pre-processing steps:

• Retain events from 2008 to present.
• Keep only records with EVENT_TYPE = “Tornado”.
• Drop records having TOR_F_SCALE “EFU”, “EF0”, “EF1”,“F0”, “F1”.
• CLASS = 0 for EF2 events, CLASS = 1 for (EF3, EF4 and EF5) events.
• Create a DatasetDict, with training set including events from 2008-2022 and test set events from 2023-2025.

import warnings

from datasets import Dataset, DatasetDict
import matplotlib as mpl
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import torch

np.set_printoptions(suppress=True, precision=5, linewidth=1000)
pd.options.mode.chained_assignment = None
pd.set_option('display.max_columns', None)
pd.set_option('display.width', None)
pd.set_option("display.precision", 5)
warnings.filterwarnings("ignore")


# Filter for tornadoes from 2008 onward with significant damage ratings.
df = (
    dfall[
        (dfall.YEAR >= 2008) & 
        (dfall.EVENT_TYPE == "Tornado") & 
        (~dfall.TOR_F_SCALE.isin(["EFU", "EF0", "EF1" ,"F0", "F1"]))
    ]
    .dropna(subset=["EVENT_NARRATIVE"])
    .drop_duplicates(subset=["EVENT_NARRATIVE"])
    .reset_index(drop=True)
)

# Strip whitespace from EVENT_NARRATIVE.
df["EVENT_NARRATIVE"] = df["EVENT_NARRATIVE"].str.replace(r"\s+", " ", regex=True).str.strip()

# Create target class based on TOR_F_SCALE.
df["CLASS"] = np.where(df.TOR_F_SCALE.isin(["EF2"]), 0, 1)

keep_columns = [
    "EVENT_ID",
    "EVENT_NARRATIVE",
    "TOR_F_SCALE",
    "BEGIN_LAT",
    "BEGIN_LON",
    "CLASS"
]

# Create train-test splits. 
dftrain = df[df["YEAR"] <= 2022][keep_columns].reset_index(drop=True)
dfvalid = df[df["YEAR"] >  2022][keep_columns].reset_index(drop=True)

# Create Dataset objects.
ds_train = Dataset.from_pandas(dftrain)
ds_valid = Dataset.from_pandas(dfvalid)
ds = DatasetDict({"train": ds_train, "valid": ds_valid})

print(f"Train size: {len(dftrain):,}")
print(f"Valid size: {len(dfvalid):,}\n")
print(f"Training sample:\n{ds_train[0]}")

c:\Users\jtriv\miniforge3\envs\gnlp\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Train size: 2,621
Valid size: 642

Training sample:
{'EVENT_ID': 105611, 'EVENT_NARRATIVE': 'A tornado damaged numerous trees, including large trees uprooted, blew windows out of a home, destroyed a metal shed, blew two windows and part of a wall out of a metal building, damaged at least three grain bins, destroyed or damaged numerous outbuildings and small sheds, blew down or snapped off at least 15 power poles, bent a metal light pole, tipped one wagon and blew the top off another, blew down a barb wire fence and pushed fence posts almost to the ground, destroyed a hog barn, and flattened corn stubble, before crossing the county line into Lyon County. Contents inside several damaged or destroyed buildings and sheds were also damaged, especially on one farm where damaged buildings housed a farm and trucking business.', 'TOR_F_SCALE': 'EF2', 'BEGIN_LAT': 43.1421, 'BEGIN_LON': -96.3, 'CLASS': 0}

Fine-tune DistilBERT on Full Text (baseline model)

The compression experiment needs a single reference classifier. We train the model once on full text, then evaluate it on compressed holdout narratives. This largely reproduces the work from Part II.

import random

import numpy as np
import pandas as pd
import torch

from sklearn.metrics import (
    accuracy_score,
    roc_auc_score,
    average_precision_score
)

from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
    TrainingArguments,
    Trainer,
    DataCollatorWithPadding
)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")


# Set random seed for reproducibility.
SEED = 516
random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(SEED)

print(f"Device: {device}")


def tokenize_batch(batch):
    return tokenizer(
        batch["EVENT_NARRATIVE"],
        truncation=True,
        max_length=512
    )

def softmax(x):
    x = x - x.max(axis=1, keepdims=True)
    exp_x = np.exp(x)
    return exp_x / exp_x.sum(axis=1, keepdims=True)


def compute_metrics(eval_pred):
    logits, labels = eval_pred
    probs = softmax(logits)[:, 1]
    preds = (probs >= 0.5).astype(int)
    return {
        "accuracy": accuracy_score(labels, preds),
        "roc_auc": roc_auc_score(labels, preds),
        "precision": average_precision_score(labels, probs),
    }
 

Device: cuda

model_name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)

ds_tokenized = ds.map(tokenize_batch, batched=True)
ds_tokenized = ds_tokenized.remove_columns(["EVENT_NARRATIVE", "TOR_F_SCALE", "EVENT_ID", "BEGIN_LAT", "BEGIN_LON"])
ds_tokenized = ds_tokenized.rename_column("CLASS", "labels")
ds_tokenized.set_format("torch")
collator = DataCollatorWithPadding(tokenizer=tokenizer)
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2).to(device)

# Training arguments. 
args = TrainingArguments(
    output_dir="distilbert-noaa-ef-part3",
    eval_strategy="epoch",
    save_strategy="epoch",
    logging_strategy="steps",
    logging_steps=50,
    learning_rate=2e-5,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=16,
    num_train_epochs=10,
    weight_decay=0.01,
    load_best_model_at_end=False,
    metric_for_best_model="roc_auc",
    greater_is_better=True,
    seed=SEED
)

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=ds_tokenized["train"],
    eval_dataset=ds_tokenized["valid"],
    data_collator=collator,
    compute_metrics=compute_metrics
)

trainer.train()

Map: 100%|██████████| 2621/2621 [00:01<00:00, 2312.68 examples/s]
Map: 100%|██████████| 642/642 [00:00<00:00, 1815.11 examples/s]
Loading weights: 100%|██████████| 100/100 [00:00<00:00, 218.74it/s, Materializing param=distilbert.transformer.layer.5.sa_layer_norm.weight]   
DistilBertForSequenceClassification LOAD REPORT from: distilbert-base-uncased
Key                     | Status     | 
------------------------+------------+-
vocab_projector.bias    | UNEXPECTED | 
vocab_transform.bias    | UNEXPECTED | 
vocab_layer_norm.weight | UNEXPECTED | 
vocab_transform.weight  | UNEXPECTED | 
vocab_layer_norm.bias   | UNEXPECTED | 
pre_classifier.weight   | MISSING    | 
classifier.weight       | MISSING    | 
pre_classifier.bias     | MISSING    | 
classifier.bias         | MISSING    | 

Notes:
- UNEXPECTED    :can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING   :those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.

Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  2.00it/s]
Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  2.08it/s]
Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  1.68it/s]
Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  1.22it/s]
Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  3.29it/s]
Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  4.18it/s]
Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  4.09it/s]
Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  3.89it/s]
Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  4.04it/s]
Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  4.10it/s]

TrainOutput(global_step=3280, training_loss=0.14509652031631004, metrics={'train_runtime': 4959.9542, 'train_samples_per_second': 5.284, 'train_steps_per_second': 0.661, 'total_flos': 2612640273795948.0, 'train_loss': 0.14509652031631004, 'epoch': 10.0})

Compressing Tornado Event Narratives

Abstractive summarization is the process of using a summarization model to compress a narrative into fewer tokens. There are many summarization models to choose from, but we will focus on sshleifer/distilbart-cnn-12-6, which is a distilled version of the BART sequence-to-sequence transformer designed for text summarization. It was trained on the CNN/DailyMail news summarization dataset learning to generate summaries of long news articles. Because it is distilled (compressed) from the full BART model, it is smaller and faster at inference while retaining much of the original model’s summarization quality.

I also evaluated facebook/bart-large-cnn, but the generated summaries were very similar. Exploring alternative models is a topic that may warrant further examination.

from transformers import AutoModelForSeq2SeqLM
from sklearn.metrics import precision_recall_fscore_support


comp_model_id = "sshleifer/distilbart-cnn-12-6"
comp_tokenizer = AutoTokenizer.from_pretrained(comp_model_id)
summarizer = AutoModelForSeq2SeqLM.from_pretrained(comp_model_id).to(device)

# Settings for short and medium compressed narratives. 
summary_presets = {
    "short": {
        "max_new_tokens": 48,
        "min_new_tokens": 20,
        "num_beams": 4,
    },
    "medium": {
        "max_new_tokens": 96,
        "min_new_tokens": 40,
        "num_beams": 4,
    },
}


def token_len_distilbert(text):
    """
    Length of tokenized text.
    """
    return len(tokenizer(text, truncation=True, max_length=512)["input_ids"])


def summarize_text_fixed(text, summary_cfg):
    """
    Generate a summary using a fixed summarization configuration.
    """
    inputs = comp_tokenizer(
        text,
        return_tensors="pt",
        truncation=True,
        max_length=1024
    ).to(device)

    with torch.no_grad():
        summary_ids = summarizer.generate(
            **inputs,
            max_new_tokens=summary_cfg["max_new_tokens"],
            min_new_tokens=summary_cfg["min_new_tokens"],
            num_beams=summary_cfg["num_beams"],
            do_sample=False,
            early_stopping=True
        )

    return comp_tokenizer.decode(summary_ids[0], skip_special_tokens=True)


def summarize_then_fit_budget(text, keep_tokens, summary_cfg):
    """
    Summarize first, then enforce the final DistilBERT token budget so the
    downstream classifier sees the same maximum length as the truncation 
    baseline.
    """
    summary_text = summarize_text_fixed(text, summary_cfg)
    return truncate_to_token_budget(summary_text, keep_tokens)


def truncate_to_token_budget(text, keep_tokens):
    """
    Truncate original text to retain at most keep_tokens.
    """
    enc = tokenizer(text, truncation=True, max_length=512, add_special_tokens=True)
    ids = enc["input_ids"]

    # keep_tokens includes special tokens in this simple implementation.
    # Enforces a minimum length of 8 tokens.
    nbr_tokens = max(8, min(keep_tokens, len(ids)))
    ids_keep = ids[:nbr_tokens]

    # decode back to text.
    return nbr_tokens, tokenizer.decode(ids_keep, skip_special_tokens=True)

• summarize_text_fixed turns a the original event narrative into a generated summary using the pre-specified model in either short or medium mode.

• truncate_to_token_budget ensures the compressed narrative is constrained by the same maximum length as the truncation baseline.

• summarize_then_fit_budget wraps summarize_text_fixed and truncate_to_token_budget.

The data flow can be envisioned as follows:

original narrative text
      ↓
summarizer tokenizer
      ↓
summarizer token IDs
      ↓
DistilBART generate summary
      ↓
summary token IDs
      ↓
constrain number of tokens
      ↓
decode back to summarized text

The next cell provides an example using summarizer. We define a helper function, generate_compressed_narrative, to generate compressed text. generate_compressed_narrative is only used to show the summarizer in action, and isn’t used beyond the next cell.

from IPython.display import display, Markdown


def generate_compressed_narrative(txt, max_new_tokens, min_new_tokens):
    """
    Generate a compressed version of the input text using the summarization model.
    """
    inputs = comp_tokenizer(
        txt,
        return_tensors="pt",
        truncation=True,
        max_length=1024
    ).to(device)

    with torch.no_grad():
        summary_ids = summarizer.generate(
            **inputs,
            min_new_tokens=min_new_tokens,
            max_new_tokens=max_new_tokens,
            num_beams=4,
            do_sample=False,
            early_stopping=True
        )

    return comp_tokenizer.decode(summary_ids[0], skip_special_tokens=True)

# Select a random event narrative.
txt_full = ds_train["EVENT_NARRATIVE"][10] # 655
nbr_tokens_orig = len(comp_tokenizer(txt_full, add_special_tokens=True)["input_ids"])

txt_medium = generate_compressed_narrative(
    txt_full, 
    max_new_tokens=summary_presets["medium"]["max_new_tokens"],
    min_new_tokens=summary_presets["medium"]["max_new_tokens"]
)
nbr_tokens_medium = len(comp_tokenizer(txt_medium, add_special_tokens=True)["input_ids"])

txt_short = generate_compressed_narrative(
    txt_full, 
    max_new_tokens=summary_presets["short"]["max_new_tokens"],
    min_new_tokens=summary_presets["short"]["max_new_tokens"]
)
nbr_tokens_short = len(comp_tokenizer(txt_short, add_special_tokens=True)["input_ids"])


display(Markdown(f"**Original event narrative ({nbr_tokens_orig} tokens):**"))
display(Markdown(txt_full))
display(Markdown(f"**Medium summary ({nbr_tokens_medium} tokens):**"))
display(Markdown(txt_medium))
display(Markdown(f"**Short summary ({nbr_tokens_short} tokens):**"))
display(Markdown(txt_short))

build_validation_set constructs a validation dataset in order to validate our classification model under different input length strategies. Starting from the original event narratives and class labels, build_validation_setcan return either the full text, a truncated version limited to a specified token budget, or a summarized version that is subsequently constrained to that same budget. For each record, it tracks the original DistilBERT token length and the number of tokens allowed for the classifier, enabling later analysis of how compression affects model performance. The output is converted to a Hugging Face Dataset containing the processed narratives, labels, token statistics, and a method indicator identifying whether the input text was left unchanged, truncated, or summarized.

def build_validation_set(
    ds_valid, method="truncate", keep_tokens=512, summary_cfg=None, method_label=None
):
    """
    Build full, compressed or truncated datasets. 

    Parameters
    ----------
    ds_valid : Dataset
        Validation dataset containing EVENT_NARRATIVE and CLASS columns.

    keep_tokens : int
        Maximum number of DistilBERT tokens allowed for classifier input.

    method : str
        Either "full", "truncate" or "summarize".

    summary_cfg : dict or None
        Required when method="summarize". Controls summarization generation.

    """
    texts = ds_valid["EVENT_NARRATIVE"]
    ys = ds_valid["CLASS"]
    orig_nbr_tokens = [token_len_distilbert(txt) for txt in texts]

    if method == "full":

        df = pd.DataFrame({
            "EVENT_NARRATIVE": texts,
            "labels": ys,
            "orig_tokens": orig_nbr_tokens,
            "kept_tokens": orig_nbr_tokens,
            "method": ["full_text"] * len(texts)
        })

    else:
        out_texts, out_orig_tokens, out_nbr_tokens = [], [], []

        for txt in texts:
            orig_nbr_tokens = token_len_distilbert(txt)

            if method == "truncate":
                nbr_tokens, txt_c = truncate_to_token_budget(txt, keep_tokens)

            elif method == "summarize":
                nbr_tokens, txt_c = summarize_then_fit_budget(txt, keep_tokens, summary_cfg)

            else:
                raise ValueError("method must be 'full', 'truncate' or 'summarize'")

            out_texts.append(txt_c)
            out_orig_tokens.append(orig_nbr_tokens)
            out_nbr_tokens.append(nbr_tokens)

        df = pd.DataFrame({
            "EVENT_NARRATIVE": out_texts,
            "labels": ys,
            "orig_tokens": out_orig_tokens,
            "kept_tokens": out_nbr_tokens,
            "method": [(method_label or method)] * len(out_texts)
        })

    return Dataset.from_pandas(df)


def tokenize_for_eval(ds):
    ds2 = ds.map(tokenize_batch, batched=True)
    keep_cols = ["input_ids", "attention_mask", "labels", "orig_tokens", "keep_tokens", "method"]
    remove_cols = [c for c in ds2.column_names if c not in keep_cols]
    ds2 = ds2.remove_columns(remove_cols)
    ds2.set_format("torch")
    return ds2


def compute_metrics_from_predictions(yhat, yact):

    precision, recall, f1, _ = precision_recall_fscore_support(
        yhat,
        yact,
        average="weighted",
        zero_division=0
    )

    return {
        "accuracy": accuracy_score(yhat, yact),
        "precision_weighted": precision,
        "recall_weighted": recall,
        "f1_weighted": f1,
    }


def evaluate_dataset(ds):
    ds_tokenized = tokenize_for_eval(ds)
    pred_output = trainer.predict(ds_tokenized)
    yhat = np.argmax(pred_output.predictions, axis=1)
    yact = pred_output.label_ids
    metrics = compute_metrics_from_predictions(yhat, yact)
    avg_orig_tokens = float(np.mean(ds["orig_tokens"]))
    avg_keep_tokens = float(np.mean(ds["kept_tokens"]))
    
    return {
        **metrics,
        "avg_orig_tokens": avg_orig_tokens,
        "avg_nbr_tokens": avg_keep_tokens
    }

In the next cell each of the four validation sets are created and keep_tokens is set to 128. The event narratives are typically 200-300 DistilBERT tokens on average. A value around 128 compresses the average narrative by roughly 50% while still allowing multiple sentences, and is a common sequence length for BERT-style classifiers in practice. Note that the 128-token limit is treated as a maximum budget, not a target length. If a summarized narrative is already shorter than 128 tokens, it is passed to the classifier unchanged. Truncation is only applied when the compressed text exceeds this limit.

keep_tokens = 128

# Original event narratives. 
ds_valid_full = build_validation_set(
    ds_valid, 
    method="full"
)

# Event narratives truncated to 128 tokens. 
ds_valid_trunc = build_validation_set(
    ds_valid,
    keep_tokens=keep_tokens,
    method="truncate"
)

# Moderate compression.
ds_valid_sum_medium = build_validation_set(
    ds_valid,
    keep_tokens=keep_tokens,
    method="summarize",
    summary_cfg=summary_presets["medium"],
    method_label="summarize_medium"
)

# Aggressive compression.
ds_valid_sum_short = build_validation_set(
    ds_valid,
    keep_tokens=keep_tokens,
    method="summarize",
    summary_cfg=summary_presets["short"],
    method_label="summarize_short"
)

With the datasets prepared, we evaluate the classifier across the four validation sets and capture the resulting performance metrics:

from time import perf_counter

results = []

for name, ds_curr in [
    ("full_text", ds_valid_full),
    ("truncate", ds_valid_trunc),
    ("summarize_medium", ds_valid_sum_medium),
    ("summarize_short", ds_valid_sum_short)
]:
    t_i = perf_counter()
    metrics = evaluate_dataset(ds_curr)
    t_total = perf_counter() - t_i
    results.append({
        "condition": name,
        "runtime": t_total,
        **metrics
    })

dfsumm = pd.DataFrame(results)

dfsumm.head(5)

Map: 100%|██████████| 642/642 [00:00<00:00, 3081.44 examples/s]

Map: 100%|██████████| 642/642 [00:00<00:00, 4806.41 examples/s]

Map: 100%|██████████| 642/642 [00:00<00:00, 5358.20 examples/s]

Map: 100%|██████████| 642/642 [00:00<00:00, 2741.43 examples/s]

Our results show a clear trade-off between predictive performance and input length. Using full narratives resulted in the best classification performance (weighted recall of ~93%), but it is also the slowest condition because the classifier is required to process the entire text sequence for each sample. Surprisingly, truncation provides the best compromise among the three compression methods: although recall drops a bit, it retains roughly half the original tokens while cutting runtime by ~75%. The trade off is that truncation blindly discards the latter portion of the narrative, which may occasionally remove useful information.

The summarization approaches compress the narratives more aggressively, but the additional compression comes with a further decline in predictive performance. The results suggest that most of the useful signal for classification appears early in the narrative.

Whether the degradation in performance is acceptable is application dependent. For full text, the model’s inference rate was about 52 samples per second. For truncation, it was closer to 187 samples per second. The full-text approach would be more desirable in research settings where accuracy is the primary concern. If analysts are retrospectively studying severe weather events to improve risk models or validate damage classifications, runtime is far less important than ensuring the model captures every useful detail in the narrative. In a high-volume workflow, the ability to process records quickly and cheaply can matter more than extracting every last bit of predictive signal from the narrative.

Finally, an avenue for future exploration would be to carry out a truncation sweep, conceptually sticking to the original idea motivating this post but focusing on different truncation thresholds instead of constrained abstractive summarization.

Fine-Tuned Transformer Classifier (DistilBERT end-to-end)

Our objective is to train DistilBERT end-to-end by attaching a classification head and updating all weights. This is the simplest all-transformer setup, where text is provided as input and class probabilities are produced as output.

from transformers import (
    AutoTokenizer,
    AutoModel,
    AutoModelForSequenceClassification,
    DataCollatorWithPadding,
    TrainingArguments,
    Trainer
)

from sklearn.metrics import (
    accuracy_score,
    classification_report,
    roc_auc_score,
    average_precision_score,

)

model_name = "distilbert-base-uncased"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained(model_name)

print(f"\nDevice: {device}")

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

Device: cpu

As before we begin with tokenization, this time using the Hugging Face dataset mapping API. This keeps tokenization tied to the dataset object and avoids manual tensor management until training time. ds_tokenized.set_format("torch") ensures all features are automatically converted to torch.Tensor objects.

When fine-tuning a transformer, each training batch contains sequences of different lengths. While we already specified a maximum sequence length during tokenization, we do not want to pad every example in the dataset to that maximum ahead of time. Doing so would waste memory and slow training, especially on CPU.

DataCollatorWithPadding solves this by performing dynamic padding at batch time. For each batch, it looks at the longest sequence in that batch and pads all other sequences just enough to match it. The tokenizer is passed into the collator so it knows which padding token to use and how to construct the corresponding attention masks.

def tokenize_batch(batch):
    return tokenizer(
        batch["EVENT_NARRATIVE"],
        truncation=True,
        max_length=512
    )

ds_tokenized = ds.map(tokenize_batch, batched=True)
ds_tokenized = ds_tokenized.remove_columns(["EVENT_NARRATIVE", "TOR_F_SCALE", "EVENT_ID", "BEGIN_LAT", "BEGIN_LON"])
ds_tokenized = ds_tokenized.rename_column("CLASS", "labels")
ds_tokenized.set_format("torch")

collator = DataCollatorWithPadding(tokenizer=tokenizer)

Map: 100%|██████████| 2620/2620 [00:00<00:00, 8733.79 examples/s]
Map: 100%|██████████| 641/641 [00:00<00:00, 5777.82 examples/s]

DistilBERT with 2 classes produces logits of shape (N, 2). We convert logits to probabilities using softmax, then take P(class=1). compute_metrics is passed into the Trainer object initialized in a few cells, which provides us with validation metrics at each epoch.

def softmax(x):
    x = x - x.max(axis=1, keepdims=True)
    exp_x = np.exp(x)
    return exp_x / exp_x.sum(axis=1, keepdims=True)

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    probs = softmax(logits)[:, 1]
    preds = (probs >= 0.5).astype(int)
    return {
        "accuracy": accuracy_score(labels, preds),
        "roc_auc": roc_auc_score(labels, preds),
        "precision": average_precision_score(labels, probs),
    }
    

In the next cell, when AutoModelForSequenceClassification.from_pretrained is called, three things happen at once:

1. The pretrained DistilBERT encoder weights are loaded from Hugging Face. This is the same transformer we used in Part I as a frozen feature extractor.

2. A fully-connected neural network head is added on top of the encoder. For sequence classification, this is a linear layer that takes the model’s internal representation of the input sequence and maps it to num_labels outputs. For our dataset, the head outputs two logits (CLASS=0 and CLASS=1).

3. The model is configured end-to-end so that both the encoder and the classification head are trainable. During fine-tuning, gradients flow from the classification loss back through the head and into DistilBERT’s layers. This is what allows the model to adapt to NOAA-specific language and severity cues.

model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    num_labels=2
).to(device)

Loading weights: 100%|██████████| 100/100 [00:00<00:00, 964.04it/s, Materializing param=distilbert.transformer.layer.5.sa_layer_norm.weight]    
DistilBertForSequenceClassification LOAD REPORT from: distilbert-base-uncased
Key                     | Status     | 
------------------------+------------+-
vocab_layer_norm.bias   | UNEXPECTED | 
vocab_transform.weight  | UNEXPECTED | 
vocab_projector.bias    | UNEXPECTED | 
vocab_layer_norm.weight | UNEXPECTED | 
vocab_transform.bias    | UNEXPECTED | 
classifier.weight       | MISSING    | 
pre_classifier.bias     | MISSING    | 
pre_classifier.weight   | MISSING    | 
classifier.bias         | MISSING    | 

Notes:
- UNEXPECTED    :can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING   :those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.

TrainingArguments is the configuration object that tells the Hugging Face Trainer how to run training. More on available options can be found here.

args = TrainingArguments(
    output_dir="distilbert-noaa",
    eval_strategy="epoch",
    save_strategy="epoch",
    logging_strategy="steps",
    logging_steps=50,
    learning_rate=2e-5,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=16,
    num_train_epochs=5,
    weight_decay=0.01,
    load_best_model_at_end=True,
    metric_for_best_model="roc_auc",
    greater_is_better=True
)

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=ds_tokenized["train"],
    eval_dataset=ds_tokenized["valid"],
    data_collator=collator,
    compute_metrics=compute_metrics
)

All that’s left to do is train and evaluate. I kept num_train_epochs low assuming CPU training. Feel free to increase this value if GPU acceleration is available. On my machine, training via CPU for 5 epochs took ~1.5 hours:

trainer.train()
eval_out = trainer.evaluate()

print("Fine-tuned DistilBERT validation metrics:")
print(eval_out)

Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  4.00it/s]
Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  3.63it/s]
Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  2.02it/s]
Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  3.34it/s]
Writing model shards: 100%|██████████| 1/1 [00:00<00:00,  3.17it/s]
There were missing keys in the checkpoint model loaded: ['distilbert.embeddings.LayerNorm.weight', 'distilbert.embeddings.LayerNorm.bias'].
There were unexpected keys in the checkpoint model loaded: ['distilbert.embeddings.LayerNorm.beta', 'distilbert.embeddings.LayerNorm.gamma'].

Fine-tuned DistilBERT validation metrics:
{'eval_loss': 0.23589347302913666, 'eval_accuracy': 0.9453978159126365, 'eval_roc_auc': 0.92359900373599, 'eval_precision': 0.923015586437536, 'eval_runtime': 98.0304, 'eval_samples_per_second': 6.539, 'eval_steps_per_second': 0.418, 'epoch': 5.0}

Generate the classification report:

pred_out = trainer.predict(ds_tokenized["valid"])
logits = pred_out.predictions
p1 = softmax(logits)[:, 1] 
y_valid = pred_out.label_ids

# Assume 0.50 threshold for time being.
y_pred = (p1 >= 0.5).astype(int)

print(classification_report(y_valid, y_pred, digits=5))

              precision    recall  f1-score   support

           0    0.96559   0.96364   0.96461       495
           1    0.87755   0.88356   0.88055       146

    accuracy                        0.94540       641
   macro avg    0.92157   0.92360   0.92258       641
weighted avg    0.94554   0.94540   0.94546       641

We are now in a position to compare results from Part I. Recall the classification summary from our frozen embeddings model:

              precision    recall  f1-score   support

           0    0.89447   0.71919   0.79731       495
           1    0.42798   0.71233   0.53470       146

    accuracy                        0.71763       641
   macro avg    0.66123   0.71576   0.66601       641
weighted avg    0.78822   0.71763   0.73750       641

The big takeaway is that fine-tuning delivers a step-change improvement. Accuracy, precision and recall improved across the board, but what really stands out is the increase in CLASS=1 precision. In Part 1, CLASS=1 precision of ~0.43 meant that more than half of the events flagged as severe were actually EF2. It might catch severe events, but it would generate a lot of false positives. After fine-tuning, CLASS=1 precision jumped to ~0.84. Now, when the model says “this looks like an EF3+ event”, it’s usually correct. We’re no longer trading false positives for recall: the model is genuinely better at recognizing severe events from event narratives. From an insurance perspective, that’s the difference between an exploratory model and something that could potentially support triage or prioritization.

Conclusion

Fine-tuning DistilBERT end-to-end on tornado narratives led to a dramatic improvement in classification performance compared to using frozen embeddings. Precision and recall for EF3+ events both increased substantially, which indicates that the model has learned severity-specific language patterns that embeddings alone weren’t able to fully capture. However, the improvement fine-tuning provided came at the cost of 15x longer training time, greater computational overhead, and a more complex modeling pipeline. Whether that tradeoff is worthwhile depends on how much narrative detail is truly necessary for the task in question.

In Part III, we explore that question directly by evaluating whether compressed versions of the original text can achieve comparable performance, and how much information can be removed before predictive power begins to degrade.

1. Pre-Processing

The text descriptions are sourced from the the NOAA Severe Weather Events Database. It contains detailed records on hazards such as tornadoes, hail, floods, hurricanes and winter storms. Each entry provides structured data and narrative descriptions. Each file contains two columns with text descriptions of significant weather events:

• EVENT_NARRATIVE describes the specific, individual event. For example, if there is a tornado in one county, the EVENT NARRATIVE contains details about that tornado (its path, damage, casualties, and circumstances).

• EPISODE_NARRATIVE describes the broader weather episode that groups multiple related events. For instance, if a line of severe thunderstorms produced several tornadoes and hail reports across multiple counties, the EPISODE_NARRATIVE provides the larger context of the storm system, while each tornado or hail report has its own EVENT NARRATIVE.

Think of EPISODE NARRATIVE as the big picture story of the weather system and EVENT NARRATIVE as the details of each individual occurrence. We will focus on EVENT_NARRATIVE for our analysis, but a follow-up analysis will assess whether EPISODE_NARRATIVE combined with EVENT_NARRATIVE offers improvement vs. EVENT_NARRATIVE alone.

The dataset is filtered to retain events from 2008 to present and records with EVENT_TYPE = ‘Tornado’. We also drop records having TOR_F_SCALE “EFU”, “EF0”, “EF1”,“F0”, “F1”.

Since our goal is to train a classifier that can distinguish between low severity (EF2) and high severity (EF3, EF4 and EF5) events based on the event narrative, a CLASS column is created and used as the target for the downstream classifier. CLASS takes on the value 0 for EF2 events and 1 for EF3, EF4 and EF5 events.

To ensure the code provided is this series is reproducible into the future, it is strongly encouraged that a new virtual environment be created with the following dependencies:

# requirements.txt
numpy==1.26.4
pandas==2.2.1
scikit-learn==1.4.2
scipy==1.12.0
torch==2.4.1

# Hugging Face stack (Trainer-capable)
transformers==5.1.0
accelerate==1.3.0
datasets==2.18.0
evaluate==0.4.1

# Must match Transformers v5 requirements
tokenizers
huggingface-hub

# Utilities
tqdm==4.66.2
pyarrow==15.0.2
sentencepiece==0.2.0

# Additional
matplotlib==3.8.3
wordcloud==1.9.6
bs4 
requests 
contextily
geopandas

These can be installed directly from GitHub by executing the next cell:

!python -m pip install -r https://gist.githubusercontent.com/jtrive84/e313afbf2def24687e3c3247aa836fe9/raw/f2493005d291920dc1fcfd54274b9dfa42004ebe/requirements.txt

NOAA’s Storm Events Database can be pulled directly over HTTP, so we don’t need use a special API client. The next cell programmatically discovers the latest file for each year and reads each into a single DataFrame. We first check whether “noaa-events-2008-2025.parquet” exists in the current working directory. If not, the files are downloaded and saved locally. Subsequent executions of the next cell will read your local file instead of unnecessarily burdening NOAA’s file server:

import os
import pandas as pd
import requests
from bs4 import BeautifulSoup

base_url = "https://www.ncei.noaa.gov/pub/data/swdi/stormevents/csvfiles/"


def get_latest_details_file(year):
    """
    Return the filename of the latest details file for the given year.
    Sample filename: StormEvents_details-ftp_v1.0_d2022_c20250721.csv.gz
    """
    html = requests.get(base_url).text
    soup = BeautifulSoup(html, "html.parser")
    candidates = [
        a["href"]
        for a in soup.find_all("a", href=True)
        if f"StormEvents_details-ftp_v1.0_d{year}" in a["href"]
    ]

    return sorted(candidates)[-1]


# Check if dataset exists locally prior to hitting NOAA servers.
if not os.path.exists("noaa-events-2008-2025.parquet"):
    
    print("Fetching latest filenames from NOAA servers...")
    # Get latest filenames for years 2008-2025.
    latest_filenames = [get_latest_details_file(year) for year in range(2008, 2026)]

    # Load each file into a DataFrame and concatenate.
    dfall = pd.concat(
        [pd.read_csv(f"{base_url + fname}", compression="gzip") for fname in latest_filenames],
        ignore_index=True,
    )
    dfall.to_parquet("noaa-events-2008-2025.parquet")
else:
    print("Loading dataset locally...")
    dfall = pd.read_parquet("noaa-events-2008-2025.parquet")

print(f"Total records loaded: {dfall.shape[0]:,}")
dfall.head()

Fetching latest filenames from NOAA servers...
Total records loaded: 1,158,836

import warnings

import matplotlib as mpl
import numpy as np
import matplotlib.pyplot as plt
import torch

np.set_printoptions(suppress=True, precision=5, linewidth=1000)
pd.options.mode.chained_assignment = None
pd.set_option('display.max_columns', None)
pd.set_option('display.width', None)
pd.set_option("display.precision", 5)
warnings.filterwarnings("ignore")


# Event files by year available @ https://www.ncei.noaa.gov/stormevents/ftp.jsp.
# events_path = "C:\\Users\\jtriv\\repos\\blog-posts-in-progress\\practical-nlp-for-risk-modeling-part-2\\noaa-events-2008-2025.parquet"
# dfall = pd.read_parquet(events_path)

# Filter for tornadoes from 2008 onward with significant damage ratings.
df = (
    dfall[
        (dfall.YEAR >= 2008) & 
        (dfall.EVENT_TYPE == "Tornado") & 
        (~dfall.TOR_F_SCALE.isin(["EFU", "EF0", "EF1" ,"F0", "F1"]))
    ]
    .dropna(subset=["EVENT_NARRATIVE"])
    .drop_duplicates(subset=["EVENT_NARRATIVE"])
    .reset_index(drop=True)
)

# Strip whitespace from EVENT_NARRATIVE.
df["EVENT_NARRATIVE"] = df["EVENT_NARRATIVE"].str.replace(r"\s+", " ", regex=True).str.strip()

# Create target class based on TOR_F_SCALE.
df["CLASS"] = np.where(df.TOR_F_SCALE.isin(["EF2"]), 0, 1)

print(f"Total records after filtering: {df.shape[0]:,}")

df.head()

Total records after filtering: 3,261

Let’s get an idea of the distribution of events by TOR_F_SCALE as well as CLASS:

from IPython.display import display_html

# Distribution over TOR_F_SCALE.
tbl1a = df.TOR_F_SCALE.value_counts().reset_index(drop=False)
tbl1b = df.TOR_F_SCALE.value_counts(normalize=True).reset_index(drop=False)
tbl1 = tbl1a.merge(tbl1b, on="TOR_F_SCALE", how="left")

# Distribution over class.
tbl2a = df.CLASS.value_counts().reset_index(drop=False)
tbl2b = df.CLASS.value_counts(normalize=True).reset_index(drop=False)
tbl2 = tbl2a.merge(tbl2b, on="CLASS", how="left")

tbl1_style = tbl1.style.set_table_attributes("style='display:inline'").set_caption('TOR_F_SCALE').hide(axis='index')
tbl2_style = tbl2.style.set_table_attributes("style='display:inline'").set_caption('CLASS').hide(axis='index')
display_html(tbl1_style._repr_html_()+tbl2_style._repr_html_(), raw=True)

~75% of events fall in CLASS = 0, and ~25% in CLASS = 1. The dataset is mildly imbalanced, but a 75/25 split is common in real-world problems and usually workable with standard classifiers. Most algorithms can still learn meaningful decision boundaries without special tricks.

DAMAGE_PROPERTY captures the estimated dollar value of property damage associated with each event. This includes losses to homes, buildings, infrastructure, vehicles, and other physical assets. While we will not use DAMAGE_PROPERTY as a feature in our classifier in Part I, it is still informative to understand how average property damage varies across EF classes. Because this field is stored in a non-numeric format within the NOAA database, we first convert it to a numeric type and then compute average damage by TOR_F_SCALE:

def convert_to_numeric(damage_str):
    """
    Convert damage string with K, M, B suffix to numeric value.
    """
    if pd.isna(damage_str):
        return 0  
    damage_str = str(damage_str).upper()
    if "K" in damage_str:
        return float(damage_str.replace("K", "")) * 1000
    elif "M" in damage_str:
        return float(damage_str.replace("M", "")) * 1000000
    elif "B" in damage_str: 
        return float(damage_str.replace("B", "")) * 1000000000

# Apply function to each row of dft. 
df["DAMAGE_PROPERTY_NUMERIC"] = df["DAMAGE_PROPERTY"].apply(convert_to_numeric).fillna(0)

# Compute average property damage by TOR_F_SCALE.
avg_damage = df.groupby("TOR_F_SCALE")["DAMAGE_PROPERTY_NUMERIC"].mean()
print("Average Property Damage by TOR_F_SCALE:")
avg_damage = avg_damage.apply(lambda x: f"${x:,.2f}").reset_index(drop=False)

avg_damage

Average Property Damage by TOR_F_SCALE:

This aligns with our expectations. EF classes are based on wind speed, and the damage from EF5 events will be far greater on average than the other classes:

A word cloud can be used to quickly visualize the most frequent or prominent words in a body of text, with more important words appearing larger. Its purpose is to give an at-a-glance sense of dominant themes in the text.

Two separate word clouds are created based on EVENT_NARRATIVE (one for CLASS = 0 the other for CLASS = 1). Output is limited to the top 50 words:

from wordcloud import WordCloud

# Limit to top 50 words for word clouds.
num_words = 50

text0 = df.loc[df["CLASS"]==0, "EVENT_NARRATIVE"].str.cat(sep=" ")
text1 = df.loc[df["CLASS"]==1, "EVENT_NARRATIVE"].str.cat(sep=" ")
wc0 = WordCloud(max_words=num_words, background_color="white").generate(text0)
wc1 = WordCloud(max_words=num_words, background_color="white").generate(text1)

fig, ax = plt.subplots(1, 2, figsize=(12, 12), dpi=150, tight_layout=True)
ax[0].set_title("CLASS=0 (EF2)", fontsize=10, weight="bold", color="red")
ax[0].imshow(wc0)
ax[0].axis("off")
ax[1].set_title("CLASS=1 (EF3, EF4, EF5)", fontsize=10, weight="bold", color="red")
ax[1].imshow(wc1)
ax[1].axis("off")
plt.show()

At first glance, the CLASS=0 and CLASS=1 word clouds do not appear meaningfully different.

2. Using DistilBERT as a Feature Extractor

Hugging Face is an open-source AI platform best known for making modern NLP and foundation models easy to use and share. It provides pre-trained models, standardized model APIs, and datasets so you don’t have to build everything from scratch. Training is more convenient because common tasks—tokenization, batching, fine-tuning, and evaluation—are handled by well-designed libraries with sensible defaults.

The Python transformers library is Hugging Face’s core toolkit for working with models like BERT, GPT, T5, and vision transformers. It offers simple APIs to load pre-trained models, tokenize data and fine-tune on your own datasets. In this section, we carry out the following steps:

0. Create Dataset object from event narratives.
1. Tokenize event narratives with DistilBERT tokenizer.
   
2. Run frozen DistilBERT forward pass.
3. Pool token embeddings into one vector per event.

Instead of creating a random train-test split, events are split based on time. The training set will include all events from 2008-2022, the test set 2022-present. We do this because language evolves over time and reporting practices change. It can also be viewed as a way to assess how well the model generalizes to future event descriptions.

In the next cell train and validation Dataset objects are created from the original DataFrame. We create the DatasetDict instance manually: Had we opted for a random split, this would have been created automatically when calling .train_test_split on the original DataFrame, i.e., Dataset.from_pandas(df).train_test_split(test_size=0.20). Dataset objects integrate tightly with the Transformers training APIs. They represent data in a standard format that supports lazy loading and fast preprocessing.

from datasets import Dataset, DatasetDict

keep_columns = [
    "EVENT_ID",
    "EVENT_NARRATIVE",
    "TOR_F_SCALE",
    "CLASS"
]

# Create train-test splits. 
dftrain = df[df["YEAR"] <= 2022][keep_columns].reset_index(drop=True)
dfvalid = df[df["YEAR"] >  2022][keep_columns].reset_index(drop=True)

# Create Dataset objects.
ds_train = Dataset.from_pandas(dftrain)
ds_valid = Dataset.from_pandas(dfvalid)
ds = DatasetDict({"train": ds_train, "valid": ds_valid})

# Inspect first training sample.
ds["train"][0]

{'EVENT_ID': 105611,
 'EVENT_NARRATIVE': 'A tornado damaged numerous trees, including large trees uprooted, blew windows out of a home, destroyed a metal shed, blew two windows and part of a wall out of a metal building, damaged at least three grain bins, destroyed or damaged numerous outbuildings and small sheds, blew down or snapped off at least 15 power poles, bent a metal light pole, tipped one wagon and blew the top off another, blew down a barb wire fence and pushed fence posts almost to the ground, destroyed a hog barn, and flattened corn stubble, before crossing the county line into Lyon County. Contents inside several damaged or destroyed buildings and sheds were also damaged, especially on one farm where damaged buildings housed a farm and trucking business.',
 'TOR_F_SCALE': 'EF2',
 'CLASS': 0}

Inspecting the output of the first training sample, we see it is nothing more than a dictionary representation of the first row in dftrain with column names as keys and values set to the original column values.

The next step it to tokenize event narratives with the DistilBERT tokenizer. DistilBERT is a smaller, faster, and lighter version of BERT created using knowledge distillation. Essentially a smaller student model learns to mimic a larger teacher BERT model. It retains most of BERT’s language understanding performance while using fewer parameters and requiring less compute.

There are two variants we can choose from:

• distilbert-base-uncased: Does not make a difference between english and English.
• distilbert-base-cased: Does make a difference between english and English.

For the remainder of the post, the focus will be on distilbert-base-uncased, but it is very easy to swap and assess different models when working with the transformers library, in many cases it is as simple as updating the model name.

from transformers import AutoTokenizer, AutoModel

# Model name.
model_name = "distilbert-base-uncased"

# Initialize tokenizer and model.
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

# Run model on GPU if available.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Send model to device. 
model.to(device)

# Set model to evaluation mode.
model.eval()

# Print number of parameters in model.
total_params = sum(p.numel() for p in model.parameters())
print(f"Total parameters in {model_name}: {total_params:,}")
print(f"Tokenizer vocab_size: {tokenizer.vocab_size:,}")
print(f"Tokenizer model_max_length (maximum context size): {tokenizer.model_max_length}")

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Loading weights: 100%|██████████| 100/100 [00:00<00:00, 979.85it/s, Materializing param=transformer.layer.5.sa_layer_norm.weight]    
DistilBertModel LOAD REPORT from: distilbert-base-uncased
Key                     | Status     |  | 
------------------------+------------+--+-
vocab_layer_norm.weight | UNEXPECTED |  | 
vocab_transform.bias    | UNEXPECTED |  | 
vocab_transform.weight  | UNEXPECTED |  | 
vocab_layer_norm.bias   | UNEXPECTED |  | 
vocab_projector.bias    | UNEXPECTED |  | 

Notes:
- UNEXPECTED    :can be ignored when loading from different task/architecture; not ok if you expect identical arch.

Total parameters in distilbert-base-uncased: 66,362,880
Tokenizer vocab_size: 30,522
Tokenizer model_max_length (maximum context size): 512

Tokenization bridges the gap between free-form narrative text and the numeric tensors required by transformer models. Transformers do not operate on strings. Instead, they consume token IDs which are integers representing sub-word units from a learned vocabulary. DistilBERT uses a subword tokenizer, meaning words can be split into smaller pieces. This allows the model to handle rare words, misspellings, and morphological variations while keeping the vocabulary size manageable.

The tokenizer also produces an attention mask indicating which positions correspond to real tokens versus padding. This is critical when batching variable-length narratives, because the model needs to ignore padded positions when computing internal representations.

distilbert-base-uncased has a vocabulary of size 30,522 and a maximum context size of 512. The maximum context size means there can be at most 512 tokens (subword pieces, not characters or words) in a single input sequence. If an event narrative is longer than this, it must be truncated or split into chunks before being passed to the model.

The next cell provides an example of how we go from raw text to an encoded sequence of embedding IDs.

# Stanley Brothers, "The Fields have Turned Brown."
text = "The price I have paid, to live and to learn"

# Encoded text. 
enc_input = tokenizer(text, return_tensors='pt')

print(f"enc_input.input_ids.shape : {enc_input.input_ids.shape}")  # (batch_size, sequence_length)

print(f"enc_input:\n{enc_input}")
      

enc_input.input_ids.shape : torch.Size([1, 13])
enc_input:
{'input_ids': tensor([[ 101, 1996, 3976, 1045, 2031, 3825, 1010, 2000, 2444, 1998, 2000, 4553,
          102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}

len(enc_input["input_ids"][0])

13

enc_input is a dict-like object with input_ids representing the embedding ids, and attention_mask a tensor containing 1s for real tokens and 0s for padding tokens added to reach the desired sequence length.

The complete mapping of words/word parts to tokens can be obtained via tokenizer.get_vocab(). For example, we can check which id “learn” maps to:

vocab = tokenizer.get_vocab()

print(f"id for learn: {vocab['learn']}")

id for learn: 4553

Referring back to enc_input, we see that 4553 is the id at index 11.

To obtain features from the encoded input, enc_input is passed into model:

# Obtain text features. 
output = model(**enc_input)

print(f"type(output) : {type(output)}\n")
print(f"output.last_hidden_state.shape : {output.last_hidden_state.size()}")  # (batch_size, sequence_length, hidden_size)
print(f"\noutput.last_hidden_state:\n\n{output.last_hidden_state}")

type(output) : <class 'transformers.modeling_outputs.BaseModelOutput'>

output.last_hidden_state.shape : torch.Size([1, 13, 768])

output.last_hidden_state:

tensor([[[-0.0182,  0.0976, -0.0949,  ...,  0.0370,  0.2696,  0.1983],
         [ 0.2047, -0.0677, -0.3011,  ...,  0.0800,  0.4613, -0.0775],
         [ 0.7276, -0.3357, -0.0353,  ..., -0.0306,  0.2783, -0.2500],
         ...,
         [ 0.4852,  0.1291, -0.0054,  ...,  0.0494,  0.4393,  0.2179],
         [ 0.0979,  0.3544, -0.0155,  ..., -0.0864,  0.0971, -0.4329],
         [ 1.0055,  0.4603, -0.2136,  ..., -0.0508, -0.5924, -0.4584]]],
       grad_fn=<NativeLayerNormBackward0>)

In Hugging Face’s BERT models, last_hidden_state is the sequence of contextualized embeddings for every token in the input after the final transformer layer. They represent features that will be used to train our classifier. For our Stanley Brothers excerpt, this has dimension (1, 13, 768) = (batch_size, sequence_length, hidden_size):

• batch_size: The number of samples passed into the model.
• sequence_length: The number of tokens used to represent text.
• hidden_size: The dimensionality of each token’s final embedding after all transformer layers. Every sub-word token is represented as a 768-dimensional vector.

We wrap tokenizer inside a small function rather than calling it inline to ensure that truncation, padding, and maximum length are applied identically across training, validation, and inference. The same function can be reused later when scoring new narratives, which helps to avoid mismatches between training and prediction pipelines:

MAX_LEN = 512

def tokenize(texts):
    return tokenizer(
        texts,
        truncation=True,
        padding=True,
        max_length=MAX_LEN,
        return_tensors="pt"
    )

When we pass tokenized text through DistilBERT, the model does not return a single vector for the entire narrative. Instead, as we saw with our excerpt from The Fields Have Turned Brown, a vector is returned for each token in the sequence (the dimension of the output was 1 x 13 x 768, not 1 x 768). Conceptually, every token receives its own 768-dimensional embedding that reflects the token itself and surrounding context.

Classical machine learning models expect a fixed-length feature vector per observation. This creates a mismatch: we have many token-level vectors for a single narrative, but we need exactly one vector representing the entire event.

Mean pooling address this issue by averaging the token embeddings across the sequence, resulting in a single 768-dimensional vector per narrative (the desired 1 x 768 representation). The attention mask is used so that padded tokens do not contribute to the average. The result can be interpreted as a global summary of the narrative’s semantic content.

Pooling is a necessary step whenever we use a transformer as a frozen feature extractor. Without pooling, there is no straightforward way to pass transformer outputs into a traditional classifier.

This is handled in mean_pool below:

@torch.no_grad()
def mean_pool(last_hidden_state, attention_mask):
    mask = attention_mask.unsqueeze(-1).float()
    summed = (last_hidden_state * mask).sum(dim=1)
    counts = mask.sum(dim=1).clamp(min=1e-9)
    return summed / counts

At this stage, we combine everything built so far into a single operation that converts raw narrative text into numeric feature vectors. The embed_texts function in the next cell takes a list of narratives, tokenizes them, passes them through DistilBERT, and applies mean pooling to produce one fixed-length embedding per event.

Transformers operate on batches of tokenized text and return high-dimensional tensors, not directly usable feature matrices. Classical machine learning models expect a two-dimensional array where each row corresponds to an observation and each column corresponds to a feature. The embedding step bridges that gap by turning free-form text into a matrix of shape (number_of_events, 768).

Once this step is complete, the narratives will have been transformed into numerical features that can be handled in exactly the same way as any other tabular predictor. This is what makes the overall approach so practical: after embedding, the rest of the workflow is indistinguishable from a standard classification problem.

@torch.no_grad()
def embed_texts(texts, batch_size=32):

    all_vecs = []

    for i in range(0, len(texts), batch_size):
        batch = texts[i:(i + batch_size)]
        enc = tokenize(batch)
        enc = {k: v.to(device) for k, v in enc.items()}
        out = model(**enc)
        emb = mean_pool(out.last_hidden_state, enc["attention_mask"])
        all_vecs.append(emb.cpu().numpy())

    return np.vstack(all_vecs)

embed_texts is then used to generate features from the training and validation sets. The next cell may take 5-10 minutes to run if using CPU:

X_train = embed_texts(ds["train"]["EVENT_NARRATIVE"], batch_size=32)
X_valid = embed_texts(ds["valid"]["EVENT_NARRATIVE"], batch_size=32)
y_train = np.array(ds["train"]["CLASS"])
y_valid = np.array(ds["valid"]["CLASS"])

print(f"X_train shape: {X_train.shape}; y_train.shape: {y_train.shape}")
print(f"X_valid shape: {X_valid.shape}; y_valid.shape: {y_valid.shape}")

X_train shape: (2620, 768); y_train.shape: (2620,)
X_valid shape: (641, 768); y_valid.shape: (641,)

3. Classification

At this point, we can use all the familiar scikit-learn tooling to fit a classifier, because the problem has been fully reduced to standard tabular modeling. Each tornado event is now represented by a fixed-length numeric vector, and the target is a simple binary label indicating EF2 versus EF3+.

When working with transformer embeddings, simple linear models often perform surprisingly well, since much of the complexity has already been absorbed by the language model. Logistic Regression is a natural first choice in this setting. It is fast to train, handles high-dimensional feature spaces well, and provides well-behaved probability estimates. The embeddings produced by DistilBERT have 768 dimensions, which makes the problem inherently high-dimensional. Linear models with regularization are well-suited to this regime, because regularization helps control overfitting by shrinking less informative coefficients toward zero.

Nothing about this step is specific to text anymore. We could swap Logistic Regression for a Random Forest, Gradient Boosting model, or any other scikit-learn classifier without changing the upstream pipeline. This separation between representation learning and downstream modeling is one of the major advantages of using pretrained transformers as feature extractors.

In the next cell, we fit a Logistic Regression model with Elastic Net penalty. A grid search is performed over l1_ratio and C to identify the regularization regime which maximizes f1-score.

from scipy.stats import uniform, loguniform
from sklearn.model_selection import RandomizedSearchCV
from sklearn.linear_model import LogisticRegression

random_state = 516
verbosity = 3
n_iter = 50
scoring = "recall"
cv = 3

param_grid = {
    "l1_ratio": uniform(loc=0, scale=1), 
    "C": loguniform(1e-4, 1e4)
    }

clf = RandomizedSearchCV(
    LogisticRegression(solver="saga", class_weight="balanced", max_iter=500), 
    param_grid, 
    scoring=scoring, 
    cv=cv, 
    verbose=verbosity, 
    random_state=random_state, 
    n_iter=n_iter,
    n_jobs=-1
)

clf.fit(X_train, y_train)

print(f"\nOptimal parameters: {clf.best_params_}")

Fitting 3 folds for each of 50 candidates, totalling 150 fits

Optimal parameters: {'C': 0.10443596935926203, 'l1_ratio': 0.4360994501308848}

Assess model performance on holdout data:

from sklearn.metrics import classification_report

print(classification_report(y_valid, clf.predict(X_valid), digits=5))

              precision    recall  f1-score   support

           0    0.89447   0.71919   0.79731       495
           1    0.42798   0.71233   0.53470       146

    accuracy                        0.71763       641
   macro avg    0.66123   0.71576   0.66601       641
weighted avg    0.78822   0.71763   0.73750       641

This is a very solid result, given that we used a frozen encoder and text-only features. In particular, what matters most for our use case is CLASS=1 recall = 0.71, which means roughly 7 out of every 10 truly severe tornadoes are being correctly identified. Precision = 0.43, so only about half of the tornadoes flagged as EF3+ truly are EF3+. This is the expected tradeoff: recall increased, and we paid for it with more false positives.

Conclusion

The big-picture interpretation is that the pipeline works and the model has learned meaningful signal from the text on a real-world dataset. It’s worth emphasizing that the goal of this post was not to squeeze out the last basis point of classification performance, but to build intuition around how a pretrained transformer like DistilBERT can be used as a general-purpose feature extractor for downstream modeling. Even with a relatively simple Logistic Regression head, we were able to convert unstructured event narratives into dense numerical representations with only a few lines of code and obtain better than expected results. There are many obvious avenues for improvement:

• Experiment with alternative pooling strategies.
• Trying different pretrained backbones.
• Moving beyond linear models to more expressive classifiers.
• Incorporating additional features (property damage, spatial variables, etc.)

These are left as exercises for the reader.

In Part 2, we’ll take the next step and fine-tune the transformer itself for the same task, allowing the language model to adapt its internal representations directly to the classification objective, and demonstrate why end-to-end fine-tuning often delivers significant performance improvement over fixed embeddings.

The FireStationHandler Class

In the next cell the FireStationHandler class definition is provided, which is a custom osmium.SimpleHandler subclass used to extract information about fire stations from OSM data. Before describing FireStationHandler’s methods, a little clarification on OSM vernacular, specifically nodes, ways and relations:

• A node is a single point defined by a longitude and latitude. Nodes are used to represent standalone features such as trees, fire hydrants, or POIs like restaurants. They also serve as building blocks for more complex features.

• A way is an ordered list of nodes that forms a polyline or polygon. Ways can represent linear features like roads, rivers, and railways or area features like building footprints, parks, or lakes when the way forms a closed loop. Each way refers to multiple nodes, and the shape and geometry of the way are defined by the sequence of those nodes.

• A relation is a higher-level structure that groups together multiple nodes, ways, or even other relations into a single logical feature. Relations are used when a feature cannot be represented by a single way. For example, a multipolygon building with holes or a bus route made up of many road segments. Each member in a relation can have a role (like “outer” or “inner” in a multipolygon), and the relation itself carries tags that describe the whole feature.

The next cell defines the FireStationHandler class with node and way methods (I chose to skip the relation method since it wasn’t required for this exercise):

import folium
import geopandas as gpd
import numpy as np
import osmium
import pandas as pd
from shapely.geometry import Point, LineString, Polygon

np.set_printoptions(suppress=True, precision=8, linewidth=1000)
pd.options.mode.chained_assignment = None
pd.set_option("display.max_columns", None)
pd.set_option("display.width", None)


class FireStationHandler(osmium.SimpleHandler):
    """
    Collects OSM elements with amenity=fire_station.
    - Nodes: Point
    - Ways: Polygon if closed (>=4 nodes and first==last), else LineString
    """
    def __init__(self):
        super().__init__()
        self.features = []

    def node(self, n):
        if n.tags.get("amenity") == "fire_station" and n.location and n.location.valid():
            self.features.append({
                "osm_id": n.id,
                "osm_type": "node",
                "name": n.tags.get("name"),
                "lon": float(n.location.lon),
                "lat": float(n.location.lat),
                "tags": {t.k: t.v for t in n.tags},
            })

    def way(self, w):
        if w.tags.get("amenity") != "fire_station":
            return

        coords = []
        for nd in w.nodes:
            if not nd.location or not nd.location.valid():
                coords = []
                break
            coords.append((float(nd.location.lon), float(nd.location.lat)))

        if not coords:
            return

        lon = sum(c[0] for c in coords) / len(coords)
        lat = sum(c[1] for c in coords) / len(coords)
        
        self.features.append({
            "osm_id": w.id,
            "osm_type": "way",
            "name": w.tags.get("name"),
            "lon": lon,
            "lat": lat,
            "tags": {t.k: t.v for t in w.tags},
        })

Within FireStationHandler, the node method is automatically called for each node in the extract. It checks whether the node has a tag “amenity” having value “fire_station”. If so, it collects relevant data from the node including its ID, type (marked as “node”), longitude, latitude, name and any additional tags. All tags associated with the node are contained within a dictionary. The information is then appended to the self.features list.

Within the way method, it also checks whether the node has a tag “amenity” having value “fire_station”. The handler attempts to compute a representative point by averaging the longitudes and latitudes of its nodes (remember that ways are lists of nodes). This is a simple way to reduce a building footprint to a single coordinate pair.

A FireStationHandler instance is created, and the inherited apply_file method is called. Note that locations=True is included: This tells Osmium to build an in-memory index of node locations. When the handler processes a way, it uses this index to look up and attach the coordinate data to each node within that way. Without locations=True, the nd.location attribute in the way method would be invalid, and you would not be able to get the coordinates associated with that feature.

# Path to the OSM PBF file.
pbf_path = "iowa-latest.osm.pbf"

# Apply the handler to the PBF file.
handler = FireStationHandler()
handler.apply_file(pbf_path, locations=True)  

# Create DataFrame from the extracted data.
df = pd.DataFrame(handler.features)

# Convert DataFrame to a GeoDataFrame.
df = gpd.GeoDataFrame(
    df, 
    geometry=gpd.points_from_xy(df.lon, df.lat), 
    crs="EPSG:4326"
)

print(f"\nExtracted {len(df)} fire stations from {pbf_path}.")

df.head()

Extracted 626 fire stations from iowa-latest.osm.pbf.

An independent source puts the total number of fire stations in Iowa at 872, so our initial attempt captured ~70% of known fire stations.

The current FireStationHandler only extracts features where the tag amenity=“fire_station” is present on a node or way. However, in OSM, fire stations may be identified using different tags, for example office=“fire_service” or building=“fire_station”, sometimes without the amenity tag. In the next cell, we define is_fire_station, which aims to broaden the search beyond just the amenity tag. FireStationHandlerV2 updates the node and way methods by calling is_fire_station as the initial filtering step:

def is_fire_station(tags):
    """
    Determines if the given OSM tags indicate a fire station.
    """
    g = lambda k: (tags.get(k) or "").lower()
    return (
        g("amenity") == "fire_station"
        or g("office") == "fire_service"
        or g("emergency") in {"fire_service", "rescue_station"}
        or g("building") == "fire_station"
    )


class FireStationHandlerV2(osmium.SimpleHandler):
    """
    Collects OSM elements that are fire stations based on multiple tags.
    """
    def __init__(self):
        super().__init__()
        self.features = []

    def node(self, n):
        """
        Processes OSM nodes to identify fire stations.
        """
        if is_fire_station(n.tags) and n.location and n.location.valid():
            self.features.append({
                "osm_id": n.id,
                "osm_type": "node",
                "name": n.tags.get("name"),
                "lon": float(n.location.lon),
                "lat": float(n.location.lat),
                "tags": {t.k: t.v for t in n.tags},
            })

    def way(self, w):
        """
        Processes OSM ways to identify fire stations.
        """
        if not is_fire_station(w.tags): 
            return
        coords = []
        for nd in w.nodes:
            if not nd.location or not nd.location.valid():
                coords = []
                break
            coords.append((float(nd.location.lon), float(nd.location.lat)))
        if not coords: 
            return
        
        lon = sum(c[0] for c in coords) / len(coords)
        lat = sum(c[1] for c in coords) / len(coords)
        
        self.features.append({
            "osm_id": w.id,
            "osm_type": "way",
            "name": w.tags.get("name"),
            "lon": lon,
            "lat": lat,
            "tags": {t.k: t.v for t in w.tags},
        })

# Path to the OSM PBF file.
pbf_path = "iowa-latest.osm.pbf"

# Apply the handler to the PBF file.
handler = FireStationHandlerV2()
handler.apply_file(pbf_path, locations=True)  

# Create DataFrame from the extracted data.
df = pd.DataFrame(handler.features)

# Convert DataFrame to a GeoDataFrame.
df = gpd.GeoDataFrame(
    df, 
    geometry=gpd.points_from_xy(df.lon, df.lat), 
    crs="EPSG:4326"
)

print(f"\nExtracted {len(df)} fire stations from {pbf_path}.")

df.head()

Extracted 641 fire stations from iowa-latest.osm.pbf.

By broadening the search beyond the amenity tag, we captured 15 additional fire stations, for ~72% coverage.

As noted in the introduction, OpenStreetMap does not provide full coverage of U.S. fire stations. Because it’s community-contributed, completeness varies by region and over time. For context, the U.S. Fire Administration reports roughly 27,000 registered fire departments, while OSM lists fewer than 20,000 fire stations, evidence of uneven coverage. Still, OSM is an excellent starting layer for geospatial analysis. For example, when estimating the distance from a property to the nearest fire station, OSM-based distances are an upper bound: adding missing stations can only shorten the distance or leave it unchanged.

Using osmium-tool

Osmium-tool is a fast command-line utility used for processing OpenStreetMap data. We can use it to pre-filter pdf extracts, allowing us to work with much smaller files containing only the features we are interested in. To start, install the Osmium command line tool:

$ sudo apt update
$ sudo apt install osmium-tool

Then pass the path to the pbf file, along with the tags of interest:

$ osmium tags-filter iowa-latest.osm.pbf \
    nwr/amenity=fire_station,building=fire_station,emergency=fire_service \
    -o iowa-firestations-filtered.osm.pbf \
    --overwrite

nwr represents nodes, ways and relations. Running the above command will create a new, smaller .pbf file iowa-firestations-filtered.osm.pbf, consisting of those elements meeting the specified filter criteria (amenity=fire_station,building=fire_station,emergency=fire_service). We can load the extracted locations using FireStationHandlerv2 as before, but this time it should run much quicker (this approach returns only the original 626 fire stations. It would require additional research to figure out how to identify the full 641 using osmium-tool):

# Path to the OSM PBF file.
pbf_path = "iowa-firestations-filtered.osm.pbf"

# Apply the handler to the PBF file.
handler = FireStationHandlerV2()
handler.apply_file(pbf_path, locations=True)  

# Create DataFrame from the extracted data.
df = pd.DataFrame(handler.features)

# Convert DataFrame to a GeoDataFrame.
df = gpd.GeoDataFrame(
    df, 
    geometry=gpd.points_from_xy(df.lon, df.lat), 
    crs="EPSG:4326"
)

print(f"\nExtracted {len(df)} fire stations from {pbf_path}.")

df.head()

Extracted 626 fire stations from iowa-firestations-filtered.osm.pbf.

Once the desired amenities have been extracted, we can visualize the distribution across the state using folium:

# Unpack the coordinates and names.
lats, lons, names = df.lat.tolist(), df.lon.tolist(), df.name.tolist()

# Center map on central Iowa. 
mid_lon, mid_lat = -93.0977, 41.8780
 
m = folium.Map(
    location=[mid_lat, mid_lon], 
    zoom_start=8
)

for name, lon, lat in zip(names, lons, lats):

    folium.CircleMarker(
        location=[lat, lon], 
        radius=3, 
        color="red", 
        fill_color="red", 
        popup=name,
        fill=True,
        fill_opacity=1
        ).add_to(m)

m

OpenAddresses

OpenAddresses is an open data project that provides free, worldwide address data collected from government sources, open datasets, and community contributions. It includes individual address records containing street names, cities, ZIP Codes, longitude and latitude and sometimes administrative areas. The data is organized by country and region and is regularly updated. It is especially useful for geocoding, reverse geocoding, and location enrichment tasks where access to millions of structured address points is required.

For US locations, the data are divided into regions. For this demonstration, our goal is to assign a single zip code (not ZCTA) to 10 longitude latitude pairs falling somewhere in Iowa, so we’ll download us_midwest.zip, which is ~4GB compressed.

Within the zip archive, we are interested in the statewide-addresses-state.geojson files. For Iowa, the exact path would be something like ./../us/ia/statewide-addresses-state.geojson. We read this file into a GeoDataFrame and inspect the first few records:

%load_ext watermark

%watermark --python --conda --hostname --machine --iversions

Python implementation: CPython
Python version       : 3.11.11
IPython version      : 8.31.0

conda environment: module3

Compiler    : MSC v.1942 64 bit (AMD64)
OS          : Windows
Release     : 10
Machine     : AMD64
Processor   : Intel64 Family 6 Model 170 Stepping 4, GenuineIntel
CPU cores   : 22
Architecture: 64bit

Hostname: JTRIZPC11

numpy    : 1.26.3
folium   : 0.16.0
pandas   : 2.2.0
geopandas: 0.14.3

import warnings

import numpy as np
import pandas as pd
import geopandas as gpd
from shapely.geometry import Point

np.set_printoptions(suppress=True, precision=8, linewidth=1000)
pd.options.mode.chained_assignment = None
pd.set_option("display.max_columns", None)
pd.set_option("display.width", None)
warnings.filterwarnings("ignore")

# Example Iowa statewide file. Data available at https://results.openaddresses.io/
oa = gpd.read_file(
    "./ia/statewide-addresses-state.geojson"
)

print(f"oa.shape: {oa.shape}")
oa.head(10)

oa.shape: (1467745, 10)

Ultimately a nearest-neighbor lookup will be used to assign a 5-digit ZIP Code to the 10 sample points using coordinates associated with known addresses, therefore we drop records with missing geometry or postcode and perform a bit of additional clean-up. In the last step:

# Clean and filter.
oa = oa.dropna(subset=["postcode", "geometry"])
oa["postcode"] = oa["postcode"].astype(str).str.strip()
oa = oa[oa.postcode!=""].reset_index(drop=True)

# Retain ZIP5 only (drop the plus four code).
oa["postcode"] = oa["postcode"].astype(str).str.zfill(5)
oa["zip5"] = oa["postcode"].str.split("-", n=1, expand=True)[0]
oa = oa[["zip5", "geometry"]]

print(f"Iowa addresses in OpenAddresses dataset: {len(oa):,}")

oa.head()

Iowa addresses in OpenAddresses dataset: 1,314,249

We create a sample of 10 coordinate pairs that fall within the Iowa state boundary:

# Sample points of interest. All fall within Iowa state boundary.
pois = pd.DataFrame({
    "id": np.arange(10),
    "lon": [-91.4239, -92.5043, -90.6057, -93.6110, -93.1085,
            -90.6103, -95.8953, -92.4293, -93.5519, -95.0369],
    "lat": [41.9252, 40.6608, 41.5667, 42.0368, 41.3333, 
            41.5861, 41.2213, 42.4960, 41.6058, 40.7294]
    }
)

# Create GeoDataFrame.
pois = gpd.GeoDataFrame(
    pois,
    geometry=gpd.points_from_xy(pois.lon, pois.lat),
    crs="EPSG:4326"
)

pois.head(10)

Let’s visualize the 10 points to get an idea of their distribution across the state:

import folium



lats, lons = pois.lat.tolist(), pois.lon.tolist()
coords = list(zip(lats, lons))

# Center map on central Iowa. 
mid_lon, mid_lat = -93.0977, 41.8780
 
m = folium.Map(
    location=[mid_lat, mid_lon], 
    zoom_start=8
)

for lat, lon in coords:

    folium.CircleMarker(
        location=[lat, lon], 
        radius=5, 
        color="red", 
        fill_color="red", 
        fill=True,
        fill_opacity=1
        ).add_to(m)

m

Within GeoPandas, sjoin_nearest performs a spatial join that matches each geometry in one GeoDataFrame to the nearest geometry in another GeoDataFrame based on geometric distance. Instead of checking for intersection or containment like a standard spatial join, sjoin_nearest finds the closest match in terms of Euclidean distance (in projected coordinates) or great-circle distance if using geographic coordinates.

In our example, each of the 10 points will get assigned the ZIP Code associated with the nearest known address point based on great-circle distance:

# Perform nearest neighbor lookup to known lon-lat zip code.
pois = gpd.sjoin_nearest(
    pois,
    oa[["zip5", "geometry"]],
    how="left",
    distance_col="dist_meters"
)

pois.head()

In the resulting join, index_right specifies which row index from the OpenAddresses dataset served as the nearest match to the sample point.

Since our sample data is represented using a geographic coordinate reference system, dist_meters represents the distance to the nearest known address in terms of decimal degrees, not meters (even if we named the column "dist_meters"). If accurate distance measurements to the nearest known address are required, it is necessary to re-project the data to a projected CRS, then re-execute the spatial join:

# Using a projected CRS.
pois = pd.DataFrame({
    "id": np.arange(10),
    "lon": [-91.4239, -92.5043, -90.6057, -93.6110, -93.1085,
            -90.6103, -95.8953, -92.4293, -93.5519, -95.0369],
    "lat": [41.9252, 40.6608, 41.5667, 42.0368, 41.3333, 
            41.5861, 41.2213, 42.4960, 41.6058, 40.7294]
    }
)

# Create GeoDataFrame.
pois = gpd.GeoDataFrame(
    pois,
    geometry=gpd.points_from_xy(pois.lon, pois.lat),
    crs="EPSG:4326"
)

# Re-project to a projected CRS for accurate distance calculations.
pois = pois.to_crs("EPSG:3857")

# Ensure same CRS for OpenAddresses data. Unpack lat and lon for next section.
oa["lon"] = oa.geometry.x
oa["lat"] = oa.geometry.y
oa = oa.to_crs("EPSG:3857")

pois = gpd.sjoin_nearest(
    pois, 
    oa[["zip5", "geometry"]], 
    distance_col="dist_meters"
)

pois.head(10)

For the first sample (id = 0), dist_meters indicates the nearest known address is ~20 meters away.

Finding k-Nearest ZIP Codes with BallTree

sjoin_nearest performs a spatial join by finding the nearest geometry in one GeoDataFrame for each geometry in another using a spatial index. When sjoin_nearest is called, it uses the spatial index to find candidate nearest geometries efficiently, then refines the match using actual geometric distance.

One limitation of using sjoin_nearest is that only a single ZIP Code will be assigned to each sample. There may be cases where knowing the ZIP Codes of the five or ten nearest addresses might be of interest, possibly as an indicator of the homogeneity of ZIP Codes in a given region: If the 10 nearest addresses all share the same ZIP Code, it is likely that the sample point shares that ZIP Code (assuming a relatively small distance to each address).

We can identify and retrieve ZIP Codes associated with the k-nearest addresses and corresponding distances using a BallTree. Within scikit-learn, BallTree implements a fast nearest-neighbor search using a space-partitioning tree. It is not geospatially aware and does not work directly with geometry objects, and instead works with raw coordinate arrays. In the next cell, we create a BallTree instance, and identify the 10 nearest ZIP Codes for each sample in pois. Note that BallTree with haversine distance expects input in radians, not degrees:

from sklearn.neighbors import BallTree

# Convert lon/lat to radians for BallTree when using haversine 
# metric.
# OpenAddresses data.

oa_coords = np.deg2rad(oa[["lat", "lon"]].values)

# 10 sample data points. 
poi_coords = np.deg2rad(pois[["lat", "lon"]].values)

# Create BallTree instance.
bt = BallTree(oa_coords, metric="haversine")

# Query top 10 neighbors.
dists, indices = bt.query(poi_coords, k=10)

print(f"dists:\n{dists}\n")

print(f"indices:\n{indices}\n")

dists:
[[0.00000078 0.00000128 0.00000155 0.0000019  0.00000203 0.00000257 0.00000312 0.00000329 0.0000034  0.00000356]
 [0.00000043 0.00000734 0.00000983 0.0000122  0.00001249 0.00001339 0.000014   0.00001426 0.00001839 0.00002441]
 [0.00000014 0.00000024 0.00000029 0.00000048 0.00000056 0.0000006  0.00000061 0.00000093 0.00000093 0.00000094]
 [0.00000003 0.0000005  0.0000005  0.0000005  0.00000062 0.00000064 0.00000076 0.00000085 0.00000114 0.00000131]
 [0.00000287 0.00000313 0.00000334 0.00000416 0.00000419 0.00000437 0.00000449 0.00000452 0.00000454 0.00000457]
 [0.00000067 0.00000076 0.00000078 0.00000083 0.0000009  0.00000114 0.00000118 0.00000121 0.00000123 0.00000125]
 [0.00000158 0.00000411 0.00000447 0.00000459 0.00000587 0.00000624 0.0000063  0.00000662 0.00000686 0.00000708]
 [0.00000245 0.00000253 0.00000458 0.00000494 0.00000522 0.0000057  0.00000577 0.00000604 0.00000629 0.00000641]
 [0.00000059 0.00000117 0.00000119 0.00000134 0.00000143 0.00000153 0.00000159 0.00000168 0.00000172 0.00000194]
 [0.0000043  0.00000476 0.00000553 0.00000741 0.00001087 0.00001098 0.0000111  0.00001117 0.00001213 0.00001238]]

indices:
[[ 547709  457711  131556  510091   13847  856680  896565  192896 1064575  158127]
 [ 684538 1093125 1130773  540050  784185 1083753  600773  614572   58343  334934]
 [ 659368  286207  432808  627125  391029 1006928  338206  390721  575752  267197]
 [1261954 1280697  443539 1027393  567643 1038369  116769  799895  331017   70198]
 [ 312170  797161  506549   92729  392748  322733  532624  969750 1307702   79626]
 [ 241231  519778  282477  333948 1158412  575133  486008 1200591  238422 1134008]
 [ 395965  605176  682735  450244 1282644  889027 1276471  355442 1138247  620641]
 [ 766037  732231 1011916  550407 1161263 1088782  351109 1042692   25833 1034179]
 [ 187076 1143818 1077490  343559  836727 1266959 1041256   48610  677851  868357]
 [1245842  165862  471164  764378  276749 1284297  358873  362510 1020793  984672]]

indices is a 10 x 10 array representing the indices into the original OpenAddresses GeoDataFrame corresponding to the 10 nearest addresses. We can lookup the associated ZIP Codes as follows:

# Lookup ZIP Codes for first sample (row 0)

idx = indices[0]

oa.loc[idx, "zip5"].tolist()

['52314',
 '52314',
 '52314',
 '52314',
 '52314',
 '52314',
 '52314',
 '52314',
 '52314',
 '52314']

For the first point of interest, the 10 nearest ZIP Codes are the same, indicating a degree of homegeneity for that region.

dists is also a 10 x 10 array representing the distances to each of the 10 nearest addresses. When using the haversine metric, the distances will be returned in radians. To get distance in meters, multiply by 6,371,000 (assumes an Earth radius of 6,371km):

# Convert radians to meters.
dists*= 6_371_000  

dists.round(2)

array([[  4.99,   8.14,   9.9 ,  12.13,  12.96,  16.39,  19.89,  20.93,  21.68,  22.66],
       [  2.74,  46.77,  62.65,  77.71,  79.54,  85.31,  89.2 ,  90.82, 117.18, 155.54],
       [  0.9 ,   1.5 ,   1.82,   3.03,   3.58,   3.81,   3.91,   5.94,   5.95,   5.98],
       [  0.21,   3.17,   3.18,   3.21,   3.95,   4.09,   4.86,   5.44,   7.28,   8.36],
       [ 18.31,  19.97,  21.31,  26.53,  26.72,  27.86,  28.58,  28.78,  28.91,  29.13],
       [  4.24,   4.87,   4.99,   5.28,   5.74,   7.29,   7.53,   7.68,   7.82,   7.98],
       [ 10.08,  26.16,  28.49,  29.22,  37.38,  39.73,  40.13,  42.16,  43.71,  45.09],
       [ 15.6 ,  16.12,  29.2 ,  31.48,  33.29,  36.34,  36.75,  38.48,  40.05,  40.83],
       [  3.77,   7.46,   7.61,   8.55,   9.09,   9.74,  10.13,  10.73,  10.98,  12.35],
       [ 27.4 ,  30.32,  35.26,  47.18,  69.28,  69.94,  70.69,  71.14,  77.29,  78.86]])

Notice the distances increase from left to right. The nearest neighboring address will be at index 0 for each row, the furthest at the last index.

Let’s attach the ZIP Codes and distances to the original pois GeoDataFrame:

# Add ZIP Codes and distances to pois GeoDataFrame.
pois["zip5_bt"] = [oa.loc[ii, "zip5"].tolist() for ii in indices]
pois["dist_bt"] = [dists[ii].tolist() for ii in range(len(dists))]

pois.head(10)

We confirm that the ZIP Code identified using sjoin_nearest (ZIP5) matches the nearest address ZIP Code for each of the 10 points of interest.

As final diagnostic, we can visualize all the points from the OpenAddresses dataset that map to one of our POIs nearest ZIP Codes, along with the convex hull enclosing those points. This will provide additional context missing from a purely distance based diagnostic. We focus on the point with id = 1, and perform the following steps:

• Filter oa to addresses having ZIP Code = 52537.
• Combine resulting geometries using .unary_union.
• Call .convex_hull to compute the convex hull polygon enclosing the result from the previous step.
• Visualize address points, convex hull and original point of interest.

# Filter oa to addresses having ZIP Code = 52537.
# Repoject to WGS84 for visualization.
oa52537 = oa[oa.zip5=="52537"].to_crs("EPSG:4326").reset_index(drop=True)

print(f"Addresses in ZIP Code 52537: {len(oa52537):,}")

oa52537.head(10)

Addresses in ZIP Code 52537: 1,830

# Get convex hull for ZIP Code = 52537.
cvh = oa52537.unary_union.convex_hull

# Get POI lat and lon.
poi_y = pois[pois.id==1]["lat"].item()
poi_x = pois[pois.id==1]["lon"].item()

# Initialize map.
f = folium.Figure(width=800, height=525)
m = folium.Map(location=[poi_y, poi_x], zoom_start=10).add_to(f)


folium.Marker(
    location=[poi_y, poi_x],
    icon=folium.Icon(color="red"),
    popup=f"id=1: {poi_x} {poi_y}"
    ).add_to(m)

# Add address points to map.
for point in oa52537.geometry:
    folium.CircleMarker(
        location=[point.y, point.x],
        radius=3.5,
        color="#4f9de4",
        fill=True,
        weight=1,
        fill_opacity=0.35
    ).add_to(m)


# Add the convex hull polygon to the map.
folium.Polygon(
    locations=[[lat, lon] for lon, lat in cvh.exterior.coords],
    color="#184771",
    fill=False
).add_to(m)


m

The POI (red marker) is surrounded by many addresses with the same ZIP Code. This provides additional confidence that the assignment is reasonable and relatively accurate.

Conclusion

• ZCTAs <> ZIP Codes! They are not one-to-one. Some ZIP Codes don’t have ZCTA equivalents. ZCTAs are statistical areas and do not map exactly to USPS ZIP Codes. ZIp Codes represent collections of delivery routes (not enclosed polygons).
• ZCTAs are updated only with each decennial census, while USPS updates ZIP Codes constantly.
• ZIP Codes that span multiple counties or regions may be misrepresented in ZCTA form.
• Even though the OpenAddresses dataset contains over 500 million addresses, coverage is variable, and is generally better in urban/suburban areas than in rural areas. Some large metropolitan areas may be missing address data altogether.
• If you’re building something regulatory or production-facing, you should disclose that the USPS does not publish ZIP Code boundaries, and this method uses nearest inferred ZIP Codes from publicly available data.

Install Dependencies

Ensure Docker, the Docker Compose plugin and NGINX are installed. First install and update the EPEL repository:

$ sudo yum install epel-release
$ sudo yum update

Install NGINX:

$ sudo yum install nginx

Verify the installation:

$ sudo nginx -v
nginx version: nginx/1.20.1

To install Docker, follow the instructions here.

Finally, install the Docker Compose plugin:

$ sudo yum install docker-compose-plugin

app01: Simple Dash Application

Python Dash is a framework for building interactive web applications using Python. It’s ideal for data visualization and dashboards. Dash combines Flask on the backend with Plotly for charts and React.js for components, all without requiring knowledge of HTML, CSS, or JavaScript.

Referring to the repository linked above, the app01 sub-directory contains:

• app.py: Dash application code.
• wsgi.py: Defines the WSGI application object that Gunicorn uses to run and serve the Python web app.
• requirements.txt: Dependencies for Dash application.
• Dockerfile: Defines how to build a Docker image by specifying instructions, dependencies and configurations.
  docker-compose.yml: A configuration file that defines and manages Docker applications specifying services. Builds the Docker image and runs the container in single command.

The Web Server Gateway Interface (WSGI) is a Python standard that allows web servers to communicate with Python web applications. It acts as a bridge, enabling servers like Gunicorn to run applications built with frameworks like Flask and Django in a standardized, interoperable way. Gunicorn is included as a dependency in requirements.txt.

Our sample Dash app looks like the following:

The contents of app01/app.py:

# app01/app.py
import dash
from dash import dcc, html, Dash, Input, Output
import dash_bootstrap_components as dbc
import plotly.express as px
import pandas as pd

# Sample dataset.
df = pd.DataFrame({
    'Fruit': ['Apples', 'Bananas', 'Cherries', 'Dates'],
    'Sales Q1': [30, 45, 10, 20],
    'Sales Q2': [40, 30, 15, 25]
})

app = Dash(
    __name__, 
    external_stylesheets=[dbc.themes.BOOTSTRAP],
    url_base_pathname="/app01/"
)

app.layout = dbc.Container([
    html.H3("Fruit Sales by Quarter", className="my-3"),
    dcc.Dropdown(
        id='quarter-dropdown',
        options=[{'label': col, 'value': col} for col in df.columns if col != 'Fruit'],
        value='Sales Q1',
        clearable=False,
        className="mb-4"
    ),
    dcc.Graph(id='sales-chart')
], className="p-4")

@app.callback(
    Output('sales-chart', 'figure'),
    Input('quarter-dropdown', 'value')
)
def update_chart(selected_col):
    fig = px.bar(df, x='Fruit', y=selected_col, title=f"{selected_col} per Fruit", labels={'y': 'Sales'})
    return fig

server = app.server

if __name__ == '__main__':
    app.run(debug=True)

• In the Dash constructor, we include url_base_pathname="/app01/" to indicate the application should be served at a location other than root. This also needs to be specified in the NGINX configuration file.

• Just before if __name__ == '__main__', we have server = app.server. I’m not sure what this actually does, but the app will not be render correctly unless it is included.

The contents of app01/Dockerfile:

# app01/Dockerfile
FROM python:3.12

WORKDIR /app

RUN pip install --upgrade pip
COPY requirements.txt .
RUN pip install --no-cache-dir --upgrade -r requirements.txt

COPY . . 

CMD ["gunicorn", "--bind", "0.0.0.0:8050", "wsgi:server"]

• FROM python:3.12: Specifies the base image, which is the official Python 3.12 image from Docker Hub. This sets up a Python environment for app01.

• WORKDIR /app: Sets the working directory inside the container to /app. All subsequent commands will run relative to this directory.

• RUN pip install --upgrade pip: Upgrades pip to the latest version inside the app01 container.

• COPY requirements.txt .: Copies requirements.txt file from the local machine into the /app directory of the app01 container.

• RUN pip install --no-cache-dir --upgrade -r requirements.txt: Installs app01 dependencies listed in requirements.txt.

• COPY . .: Copies all files from your current local directory to the /app directory in the app01 container.

• CMD ["gunicorn", "--bind", "0.0.0.0:8050", "wsgi:server"]: Specifies the default command to run when the container starts. It launches the Gunicorn WSGI server, binding to port 8050 and using wsgi:server (i.e., server object from wsgi.py).