Lab — clustering end to end

Lecture 13 · room 204

Prof. Niccolò Salvini

HEMA · Università Cattolica del Sacro Cuore, Rome

Friday 13 November 2026

Last time

Lecture 12 · Hierarchical clustering · Mon 9 Nov

  • Agglomerative clustering merges the two closest groups, over and over — the record is the dendrogram
  • Linkage is the choice that changes everything: single chains, complete and Ward make compact groups
  • Height is distance. The tree is not the answer; the cut is

Today

Time Min
09:00 Last time · Sets 10–11, the In class questions 20’
09:20 Step 1 · load and look 25’
09:45 Step 2 · prepare: transform, then scale 20’
10:05 Step 3 · explore with a tree on a sample 25’
10:30 Break 15’
10:45 Step 4 · decide k, deploy with k-means 35’
11:20 Step 5 · profile, name, hand over 30’
11:50 The deliverable 10’

One table, one deliverable: a segmentation a marketing director could act on Monday. No new theory.

The table

customers.csv — on Blackboard, folder Prof. Salvini, and linked from the schedule

Column Meaning
id customer
recency days since last purchase
frequency purchases in the last 12 months
monetary total spend, last 12 months (€)
tenure months since first purchase
returns items returned, last 12 months
channel_online share of purchases made online

6 000 rows. Save it in your project’s data/ folder, then open the project.

Section 1 Load and look

Step 1 — read it, then read it

library(tidyverse); library(cluster); library(factoextra); library(here)

cust <- read_csv(here("data", "customers.csv"))

glimpse(cust)               # types right? id is not a feature

summary(cust)               # ranges, NAs, and the first surprise

cust |> select(-id) |> pivot_longer(everything()) |>
  ggplot(aes(value)) + geom_histogram(bins = 40) + facet_wrap(~name, scales = "free")

Before any model: what does monetary look like? Almost certainly skewed — a few customers spend a hundred times the median. k-means will build a cluster around them alone.

What to write down before step 2

Answer on paper, one line each:

  • Which columns are behaviour and which are identity? (id leaves now)
  • Which variable would dominate an unscaled distance? Why?
  • Are there rows that are not customers — test accounts, staff, monetary = 0 with frequency = 12?

Attention

Ten minutes here save an afternoon of explaining a segment that is “the three people who bought a car”.

Section 2 Prepare

Step 2 — transform, then scale

feat <- cust |>
  select(-id) |>
  mutate(monetary  = log1p(monetary),        # tame the tail
         frequency = log1p(frequency))

X <- scale(feat)                              # mean 0, sd 1, every column
attr(X, "scaled:center")                      # keep these: you need them to profile

colMeans(X) |> round(3)                       # all zero. Good.
  • log1p because log(0) is not a number and some customers spent nothing
  • scale() because distance in euros² + distance in days² is not a distance

Section 3 Explore

Step 3 — a tree on a sample

Six thousand rows is 18 million pairwise distances — fine. Sixty thousand would not be. The habit: explore the structure on a sample, deploy on everything.

set.seed(26)
idx <- sample(nrow(X), 1500)

hc <- hclust(dist(X[idx, ]), method = "ward.D2")
fviz_dend(hc, k = 5, rect = TRUE, show_labels = FALSE)

fviz_nbclust(X[idx, ], hcut, method = "silhouette", k.max = 8)

Look at the tree. Where is the big vertical gap? Two groups? Four? Six? Write down your k before you run the silhouette, then compare.

Section 4 Decide and deploy

Step 4 — k-means on everything

fviz_nbclust(X, kmeans, method = "wss", k.max = 8)        # elbow, full data
fviz_nbclust(X, kmeans, method = "silhouette", k.max = 8)

k <- 4                                        # your decision, with a reason

set.seed(26)
km <- kmeans(X, centers = k, nstart = 50)

Then, the test that matters more than any index:

set.seed(27); km2 <- kmeans(X, centers = k, nstart = 50)
table(km$cluster, km2$cluster)               # same groups, different seed?

If the table is not close to a permutation matrix, the structure is not stable at this k. Try k - 1.

Section 5 Profile and hand over

Step 5 — back to the units people understand

cust$segment <- factor(km$cluster)

profile <- cust |>
  group_by(segment) |>
  summarise(n = n(), share = n() / nrow(cust),
            across(c(recency, frequency, monetary, tenure, returns, channel_online),
                   ~ round(median(.x), 1)))
profile

fviz_cluster(km, data = X, geom = "point", ellipse.type = "norm")

Medians, not means — you logged monetary for a reason, and the mean is back under the tail’s control.

The deliverable

One table, four rows, in the language of the person who will use it:

Segment Name Who they are What to do
1 Lapsed high-value high monetary, recency > 180 d win-back, personal, now
2 Online regulars frequent, small, 90% online app push, bundle offers
3 New and undecided tenure < 6 mo, moderate onboarding journey; watch churn
4 Store loyalists frequent, in-store, long tenure protect, do not discount

Note

Names from the numbers, not from imagination. Every adjective in the table must point at a column in profile.

A segmentation is finished when someone else can act on it without you in the room.

Hand in — optional, ungraded

Push to the homework/ set for this week, or just keep it:

  • your profile table
  • your k, and the two sentences that justify it
  • the four names

We open Monday’s lecture with three of yours on the screen.

Reading

  • ISLR §12.4.3 — practical issues in clustering: scaling, k, and validation. Short, and everything today came from it.
  • Provost & Fawcett chapter 6, Clustering — reread the last section after doing this lab

Statistics & Big Data · HEMA, Università Cattolica del Sacro Cuore