Machine Learning Basics · 15 min · 170 XP

Overfitting and how to spot it

Recognise a model that memorised its training data, and dial its flexibility back.

A model overfits when it learns the training data too well: noise, typos and one-off flukes included. The telltale sign is the gap between scores. Excellent on the training set, noticeably worse on validation. A model that's bad on both is underfitting: too simple, or missing the features that matter.

k-nearest neighbours makes this easy to see. To classify a new point, find the k closest training points and take a majority vote. With k = 1 every training point is its own nearest neighbour, so training accuracy is always 100%, and a single mislabelled point carves out a little region of wrong answers around itself. A larger k outvotes the noise, and past a point it starts to blur real boundaries.

Every model family has a dial like k: tree depth, the number of features, regularisation strength. Choose it on validation data, never training data. With little data, use cross-validation: split into, say, 5 folds, validate on each fold in turn, and average the scores, so one lucky split doesn't decide.

With scikit-learn (not run here)
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier

for k in (1, 3, 5, 9):
    scores = cross_val_score(KNeighborsClassifier(n_neighbors=k), X, y, cv=5)
    print(k, scores.mean())
Scenario
model            train accuracy   validation accuracy
deep tree        99%              71%
shallow tree     84%              82%

Loading your workspace…