Machine Learning in 5 Steps: How to Evaluate a Model

1 Introduction

As soon as model training finishes, a very practical question pops up: is this model actually any good? Answering that question is the job of the fourth step in the machine learning pipeline, Model Evaluation.

This is the sixth article in the machine learning fundamentals series. The previous article, Machine Learning Problem-Solving in 5 Steps: Model Training, covered how a model is trained and the common types of models. This one picks up from there: how do we quantify a model’s performance after training, what exactly is overfitting, and which evaluation metrics apply to classification versus regression tasks.

2 The Idea Behind Model Evaluation

In the earlier dataset preparation step, we split the data we had into a training set and a test set. Only the training set is fed to the model during training; when it comes time to evaluate, we deliberately switch to the test set — the portion the model has never seen from start to finish.

Why does it have to be data the model has never seen? An exam analogy makes this easiest to understand. Doing a pile of practice problems before the exam is like the model’s “training” phase; exam day itself is the “evaluation” phase, where a set of problems you’ve never seen before is used to check what you actually learned from practicing. The exam questions are usually similar in style to the practice problems, so as long as you genuinely understood the concepts while practicing, you should at least pass.

Conversely, if you only memorized the practice problems — question and answer together, as a set — you’ll be stumped the moment the exam phrases something differently, since the odds of the exact same question showing up again are low.

Models can fall into the same trap. A model that performs beautifully during training but falls apart during evaluation is said to be overfitting: it never actually learned how to solve the problem from the training data, it just memorized the answers. Evaluating with test data the model has never seen is exactly how we catch this.

3 Common Metrics for Classification Tasks

To call a model “good” or “bad,” we first need a metric that produces an actual score. Sticking with the exam analogy: imagine an exam made entirely of true/false questions, where a teacher grades each one as either fully correct or fully wrong — there’s no partial credit — so the score for the whole exam comes down to how many questions were answered correctly.

Classification models work the same way. Take image classification as an example: every image has exactly one correct category, and dividing the number of images the model classified correctly by the total number of images gives you Accuracy. If a model classifies 92 out of 100 images correctly, its Accuracy is 92%.

Besides Accuracy, F1 Score is another common metric for classification tasks, which a later article in this series will cover in more detail.

4 Common Metrics for Regression Tasks

As mentioned in Machine Learning in 5 Steps: How to Define the Problem, common machine learning tasks fall into two categories: classification and regression. Classification lets you count right and wrong answers, but a regression model outputs a continuous value with no clean notion of “correct” or “incorrect” — so how do you evaluate it?

The more intuitive approach is to look at how far off the prediction is from the true value. For example, if a regression model predicts a house price of 12,300 when the true price is 15,000, the error for that one sample is | 12300 – 15000 | = 2700.

Take the absolute value of the error for every sample, sum them up, and divide by the total number of samples, and you get an average error called Mean Absolute Error (MAE). A common variation is to square each sample’s error instead of taking its absolute value before averaging, which gives you Mean Square Error (MSE).

5 Evaluation Metrics in scikit-learn

In practice you don’t need to implement any of these metrics yourself — scikit-learn already ships ready-to-use implementations. The two figures below list the metrics scikit-learn commonly uses to evaluate regression and classification models, respectively:

For the full list and an explanation of each metric, see the Model evaluation section of the scikit-learn documentation.

6 Conclusion

This article introduced the idea of Model Evaluation: using test data the model has never seen to check what it actually learned during training, and choosing metrics appropriate to the task type — Accuracy for classification, and MAE or MSE for regression. We also covered overfitting, the case where a model memorizes the training data and, as a result, performs poorly during evaluation.

When evaluation results aren’t good, the problem isn’t necessarily in the evaluation step itself — more often something went wrong earlier: the problem definition may need revisiting, the dataset may need rebuilding, or the model may need retraining. Before moving on to the next step, “using the model,” these first four steps typically get run through several times, until the model reaches a good enough quality.