Lecture 18 · room 202
HEMA · Università Cattolica del Sacro Cuore, Rome
Monday 30 November 2026
Lecture 17 · Random forests and boosting · Fri 27 Nov
mtry| Time | Min | |
|---|---|---|
| 17:00 | Last time · Set 15, the In class questions | 10’ |
| 17:10 | From linear to nonlinear · the perceptron, by hand | 40’ |
| 17:50 | Gradient descent | 10’ |
| 18:00 | Break | 10’ |
| 18:10 | One neuron is a GLM | 15’ |
| 18:25 | Hidden layers | 25’ |
| 18:50 | Set 16 · in Python | 10’ |
The most over-sold and under-understood model in business
A linear boundary — logistic regression — cannot follow a curve. Neural networks exist to draw the second kind. That is the whole motivation; everything else is machinery.
Note
They were inspired by the brain, in the 1940s. The inspiration is real and the analogy is thin: inputs arrive like signals at dendrites, the body sums them, the axon fires or does not. Keep the picture; do not lean on it.
From lecture 10 — now it matters:
| Statistics | Machine learning |
|---|---|
| Coefficients | Weights W |
| Intercept | Bias W_0 |
| Transformation, link | Activation function g |
| Fitting | Training |
| Observations | Training set |
The trade, stated once more
You are about to gain flexibility and lose interpretability. A neural network learns whatever relationship is there. It does not tell you what it learned, and it gives you no standard errors. Choose it when prediction is the whole job.
Inputs X_1, \dots, X_p, weights W_1, \dots, W_p, a bias W_0, and a step:
\hat y \;=\; \operatorname{sign}\!\Big(W_0 + \sum_{j=1}^{p} W_j X_j\Big), \qquad \operatorname{sign}(z) = \begin{cases} +1 & z \ge 0 \\ -1 & z < 0\end{cases}
In general: \hat y = g\big[\phi(X; W)\big] — a linear combination \phi, then an activation g.
Now make both linear: g(z) = z. You have \hat y = W_0 + \sum_j W_j X_j. That is linear regression. The simplest possible neural network is the model you started the course with.
Same equation, two philosophies:
Note
This is why a network can be more accurate and less useful at the same time. Both statements are about the same object.
Supervised: we have the true y. Start with random weights, then for each observation:
W(t+1) \;=\; W(t) + \eta\,(y - \hat y)\,X
One pass over all observations is an epoch. Repeat until nothing changes, or you run out of patience.
Two binary inputs, output 1 only when both are 1. W_1 = 0.3, W_2 = -0.1, no bias, \eta = 0.1, threshold \theta = 0.25.
| Obs | X_1 | X_2 | y | z | \hat y | error | W after |
|---|---|---|---|---|---|---|---|
| 1 | 0 | 0 | 0 | 0 | 0 | 0 | — |
| 2 | 0 | 1 | 0 | −0.1 | 0 | 0 | — |
| 3 | 1 | 0 | 0 | 0.3 | 1 | −1 | (0.2,\,-0.1) |
| 4 | 1 | 1 | 1 | 0.1 | 0 | +1 | (0.3,\,0.0) |
End of epoch 1: W = (0.3, 0.0), two mistakes. Epoch 2: two more. Epoch 3: one. Epoch 4: none, W = (0.2, 0.1), and it stops. Four rows, one rule, and the network has learned AND.
Every row, every number. W moves only on the rows marked wrong, and only along that row’s X.
Each mistake turns the boundary toward the point it got wrong. An epoch with no mistake means nothing will ever change again — that is convergence.
The perceptron rule is a special case of something universal. Given a loss L(W) — say mean squared error — move the weights downhill:
W_j(t+1) \;=\; W_j(t) - \eta\,\frac{\partial L(W)}{\partial W_j}
Attention
A convex loss is a bowl: you reach the minimum. A network’s loss is not. You find a valley, which depends on where you started — as with k-means, run it more than once.
Too small: it creeps — thousands of steps. Too large: it overshoots the valley and oscillates. Right: a smooth, decelerating descent. Practical variants: shrink \eta over time (annealing), or add inertia (momentum, \mu \approx 0.9) — each one more hyperparameter.
Apply the update to \hat y = XW with squared loss: W(t+1) = W(t) + \eta\, X^\top(y - XW).
It converges to \hat W = (X^\top X)^{-1} X^\top y. Exactly OLS. The network, trained by gradient descent, arrives where the formula was already standing.
| OLS | NN, identity activation | |
|---|---|---|
| Intercept | 1.01 | W_0 = 1.01 |
| X_1 | −0.02 | W_1 = −0.02 |
| X_2 | −0.35 | W_2 = −0.35 |
| X_3 | −0.52 | W_3 = −0.52 |
Identical coefficients, identical predictions. Nothing was gained — yet.
Train a network on the negative log-likelihood and gradient descent finds the MLE. More precisely: a single-neuron network with activation g is a GLM whose link is g^{-1}.
| Target | Regression | Link h(\mu) | Activation g(\lambda) |
|---|---|---|---|
| Continuous | Linear | identity | identity |
| Binary | Logistic | \log \frac{\mu}{1-\mu} | sigmoid \;\frac{1}{1+e^{-\lambda}} |
| Binary | Probit | \Phi^{-1}(\mu) | \Phi(\lambda) |
| Counts | Poisson | \log \mu | e^{\lambda} |
| Categorical | Multinomial | \log \frac{\mu_k}{1-\mu_k} | softmax |
Note
Take the link function, invert it, and you have the activation. Take the activation, invert it, and you have the link. You already know every model in this table from Module 1.
Soften the step into \sigma(\lambda) = 1/(1+e^{-\lambda}) and the output is a probability. One sigmoid neuron with cross-entropy loss is logistic regression, weights and all: cross-entropy is the Bernoulli negative log-likelihood under another name.
So far: input → output. Two layers, and nothing a GLM cannot do.
Now put L hidden neurons between them. Each one is a little GLM of the inputs, and the output is a GLM of them:
a_l = f\Big(W^{(1)}_{0l} + \sum_j W^{(1)}_{jl} X_j\Big), \qquad \hat y = g\Big(W^{(2)}_0 + \sum_l W^{(2)}_l\, a_l\Big)
One hidden neuron: still a line. Two: one bend. Five: the boundary follows the data. Each neuron is a line; the output layer blends them.
Set f and g both to the identity. Expand:
\hat y = W^{(2)}_0 + \sum_l W^{(2)}_l \Big(W^{(1)}_{0l} + \sum_j W^{(1)}_{jl} X_j\Big) \;=\; \text{a linear function of } X
Attention
A hidden layer with linear activations is over-parameterised linear regression. The bend comes from f — sigmoid, tanh, ReLU. Without it, there is no network, only a slower way to run lm().
Ten inputs, one hidden layer of five sigmoid neurons, 90/10 split
Training error keeps falling. Test error falls, reaches a minimum, then rises: past that point the network is learning the noise of the training set. Overfitting, drawn.
Note
Everything from lecture 7 — ridge, lasso, cross-validation for \lambda — applies unchanged. A network is a very flexible regression, and it overfits like one.
library(tidyverse); library(nnet); library(ISLR2)
d <- Default |> mutate(default = default == "Yes", balance = scale(balance), income = scale(income))
set.seed(1); tr <- sample(nrow(d), 7000)
fit <- nnet(default ~ balance + income + student, data = d[tr, ],
size = 5, decay = 0.01, maxit = 500, trace = FALSE) # 5 hidden, weight decay
p <- predict(fit, d[-tr, ]) # probabilities
mean((p > 0.5) == d$default[-tr]) # held-out accuracysize is L. decay is \lambda. Both are hyperparameters, and Friday’s lab is about choosing them without lying to yourself.
size to 1. Compare with glm(family = binomial) on the same rows: are the coefficients close? (They should be — why?)size = 20, decay = 0. Training accuracy? Test accuracy? Draw the curve you saw on slide 17 in your head.Neural networks, in Python · about 40 minutes · before Fri 4 Dec
Not graded. The test is a mirror, not an exam.
Statistics & Big Data · HEMA, Università Cattolica del Sacro Cuore