AI & Deep Learning

ACTL3143 & ACTL5111 Deep Learning for Actuaries

Author

Patrick Laub

Show the package imports

import random
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

Artificial Intelligence

Different goals of AI

Artificial intelligence describes an agent which is capable of:

Thinking humanly	Thinking rationally
Acting humanly	Acting rationally

AI eventually become dominated by one approach, called machine learning, which itself is now dominated by deep learning (neural networks).

There are AI algorithms for simple tasks that don’t use machine learning though.

You can study a 12 week course on AI and never touch on machine learning…

Shakey the Robot (~1966 – 1972)

Route-finding I

At its core, a pathfinding method searches a graph by starting at one vertex and exploring adjacent nodes until the destination node is reached, generally with the intent of finding the cheapest route. Although graph searching methods such as a breadth-first search would find a route if given enough time, other methods, which “explore” the graph, would tend to reach the destination sooner. An analogy would be a person walking across a room; rather than examining every possible route in advance, the person would generally walk in the direction of the destination and only deviate from the path to avoid an obstruction, and make deviations as minor as possible. (Source: Wikipedia)

Used in every GPS/Navigation app and…

The minimax algorithm

How many moves ahead do you want to think? In this example, the player is thinking two moves ahead.

Reminders about the minimax algorithm:

Assume all players are rational.
You want to maximise your score while your opponent wants to minimise it.
To solve the minimax problem, move backwards from the leaves of the minimax tree.

Chess

Deep Blue (1997)

In 1997, Gary Kasparov played chess against an AI called Deep Blue. Kasparov lost.

Machine Learning

Tried making a computer smart, too hard!

Make a computer that can learn to be smart.

The Venn diagram of Artificial Intelligence, Machine Learning, Neural Networks and Deep Learning. Adapted from (Shang et al., 2025).

Definition

“[Machine Learning is the] field of study that gives computers the ability to learn without being explicitly programmed” Arthur Samuel (1959)

Deep Learning Successes (Images)

Image Classification I

What is this?

Options:

punching bag
goblet
red wine
hourglass
balloon

Note

Hover over the options to see AI’s prediction (i.e. the probability of the photo being in that category).

Image Classification II

What is this?

Options:

sea urchin
porcupine
echidna
platypus
quill

Image Classification III

What is this?

Options:

dingo
malinois
German shepherd
muzzle
kelpie

ImageNet Challenge

ImageNet and the ImageNet Large Scale Visual Recognition Challenge (ILSVRC); originally 1,000 synsets.

AlexNet — a neural network developed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton — won the ILSVRC 2012 challenge convincingly.

Note:

‘Top-5’: The top 5 categories that the NN thought the image was
‘Top-5 error rate’: The proportion of times that the correct answer is not in the top 5 guesses

How were the images labelled?

“Two years later, the first version of ImageNet was released with 12 million images structured and labeled in line with the WordNet ontology. If one person had annotated one image/minute and did nothing else in those two years (including sleeping or eating), it would have taken 22 years and 10 months.

To do this in under two years, Li turned to Amazon Mechanical Turk, a crowdsourcing platform where anyone can hire people from around the globe to perform tasks cost-effectively.”

Needed a graphics card

A graphics processing unit (GPU)

A PC with the GPU and CPU marked in red and blue.

“4.2. Training on multiple GPUs A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it. It turns out that 1.2 million training examples are enough to train networks which are too big to fit on one GPU. Therefore we spread the net across two GPUs.”

Lee Sedol plays AlphaGo (2016)

Deep Blue was a win for AI, AlphaGo a win for ML.

I highly recommend this documentary about the event.

Generative Adversarial Networks (2014)

https://thispersondoesnotexist.com/

Diffusion models

Painting of avocado skating while wearing a hoodie

A surrealist painting of an alpaca studying for an exam

Deep Learning Successes (Text)

GPT

AI predictions in the classification demo were from GPT code.

Homework Get ChatGPT to:

generate images
translate code
explain code
run code
analyse a dataset
critique code
critique writing
voice chat with you

Compare to Copilot.

Code generation (GitHub Copilot)

Students get extra Copilot for free

Use a free trial then sign up for free education account

A student post from last year:

I strongly recommend taking a photo holding up your Academic Statement to your phone’s front facing camera when getting verified for the student account on GitHub. No other method of taking/uploading photo proofs worked for me. Furthermore, I had to make sure the name on the statement matched my profile exactly and also had to put in a bio.

Good luck with this potentially annoying process!

Homework It’s a slow process, so get this going early.

Classifying Machine Learning Tasks

A taxonomy of problems

Machine learning categories in ACTL3142.

New ones:

Reinforcement learning
Semi-supervised learning
Active learning

Supervised learning

The main focus of this course.

Regression

Given policy \hookrightarrow predict the rate of claims.
Given policy \hookrightarrow predict claim severity.
Given a reserving triangle \hookrightarrow predict future claims.

Classification

Given a claim \hookrightarrow classify as fraudulent or not.
Given a customer \hookrightarrow predict customer retention patterns.

Supervised learning: mathematically

Self-supervised learning

Self-supervised learning is a machine learning technique that trains models using unlabelled data by automatically creating supervisory signals from the data itself.

Data which ‘labels itself’. Example: language model.

‘Autoregressive’ (e.g. GPT) versus ‘masked’ model (e.g. BERT).

Example: image inpainting

Other examples: image super-resolution, denoising images.

Example: Deoldify images #1

A deoldified version of the famous “Migrant Mother” photograph.

Example: Deoldify images #2

A deoldified Golden Gate Bridge under construction.

Neural Networks

How do real neurons work?

A neuron ‘firing’

Similar to a biological neuron, an artificial neuron ‘fires’ when the combined input information exceeds a certain threshold. This activation can be seen as a step function. The difference is that the artificial neuron uses mathematical rules (e.g. weighted sum) to ‘fire’ whereas ‘firing’ in the biological neurons is far more complex and dynamic.

An artificial neuron

A neuron in a neural network with a ReLU activation.

The figure shows how we first compute the weighted sum of inputs, and then evaluate the summation using the step function. If the weighted sum is greater than the pre-set threshold, the neuron ‘fires’.

One neuron

\begin{aligned} z~=~&x_1 \times w_1 + \\ &x_2 \times w_2 + \\ &x_3 \times w_3 . \end{aligned}

a = \begin{cases} z & \text{if } z > 0 \\ 0 & \text{if } z \leq 0 \end{cases}

Here, x_1, x_2, x_3 are just some fixed data.

The weights w_1, w_2, w_3 should be ‘learned’.

One neuron with bias

The bias is a constant term added to the product of inputs and weights. It helps in shifting the entire activation function to either the negative or positive side. This shifting can either accelerate or delay the activation. For example, if the bias is negative, it will shift the entire curve to the right, making the activation harder. This is similar to delaying the activation.

\begin{aligned} z~=~&x_1 \times w_1 + \\ &x_2 \times w_2 + \\ &x_3 \times w_3 + b . \end{aligned}

a = \begin{cases} z & \text{if } z > 0 \\ 0 & \text{if } z \leq 0 \end{cases}

The weights w_1, w_2, w_3 and bias b should be ‘learned’.

A basic neural network

This neural network consists of an input layer with 6 neurons (x_1, x_2, x_3, x_4, x_5, x_6), an output layer with 3 neurons, and 2 hidden layers with 5 neurons in each layer. Since every neuron is linked to every other neuron, this is called a fully connected neural network.

Since we have 6 inputs and 1 bias in the input layer, each neuron in the first hidden layer has 6+1=7 parameters to learn.

Similarly, there are 5 neurons and 1 bias in each hidden layer. Each neurion in the second hidden layer has 5+1=6 parameters to learn.

Assuming the final hidden layer also has 5 neurons, each neuron in the output layer has 5+1=6 parameters to learn.

Step-function activation

Perceptrons

Brains and computers are binary, so make a perceptron with binary data. Seemed reasonable, impossible to train.

Modern neural network

Replace binary state with continuous state. Still rather slow to train.

Note

It’s a neural network made of neurons, not a “neuron network”.

Try different activation functions

Activation functions are essential for a neural network design. They provide the mathematical rule for ‘firing’ the neuron. There are many activation functions, and the choice of the activation function depends on the problem we are trying to solve. Note: If we use the ‘linear’ activation function at every neuron, then the regression learning problem becomes a simple linear regression. But if we use ‘ReLu’, ‘tanh’, or any other non-linear function, then, we can introduce non-linearity into the model so that the model can learn complex non-linear patterns in the data. There are activation functions in both the hidden layers and the output layer. The activation function in the hidden layer controls how the neural network learns complex non-linear patterns in the training data. The choice of activation function in the output layer determines the type of predictions we get.

Flexible

One can show that an MLP is a universal approximator, meaning it can model any suitably smooth function, given enough hidden units, to any desired level of accuracy (Hornik 1991). One can either make the model be “wide” or “deep”; the latter has some advantages…

Feature engineering

Doesn’t mean deep learning is always the best option!

A major part of traditional machine learning (TML) involves conducting feature engineering to extract relevant features manually. In contrast, representational learning does not involve heavy manual feature engineering, rather, it learns relevant features automatically from data during the task. Therefore, the effort spent on feature engineering in representational learning is minimal compared to TML.

Quiz

In this ANN, how many of the following are there:

features,
targets,
weights,
biases, and
parameters?

What is the depth?

There are 3 inputs, hence, 3 features.
There is 1 neuron in the output layer, hence, 1 target.
There are 3 \times 4 + 4 \times 4 + 4\times 1 = 32 arrows, hence, there are 32 weights in total.
Since there is 1 bias for each neuron, there are 9 biases in total.
The number of total parameters to learn equals to the sum of weights and biases, hence, there are 32+9=41 parameters in total.

California House Price Prediction

Imports needed for this demo

import random
from pathlib import Path

import numpy as np

from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.preprocessing import StandardScaler, MinMaxScaler

from keras.models import Sequential
from keras.layers import Dense, Input

from keras.callbacks import EarlyStopping
from keras.callbacks import ModelCheckpoint

Data science always starts with the data!

The target variable is the median house value for California districts, expressed in $100,000’s. This dataset was derived from the 1990 U.S. census, using one row per census block group. A block group is the smallest geographical unit for which the U.S. Census Bureau publishes sample data (a block group typically has a population of 600 to 3,000 people).

Columns

MedInc median income in block group
HouseAge median house age in block group
AveRooms average number of rooms per household
AveBedrms average # of bedrooms per household
Population block group population
AveOccup average number of household members
Latitude block group latitude
Longitude block group longitude
MedHouseVal median house value (target)

Import the data

features, target = fetch_california_housing(as_frame=True, return_X_y=True)
features

	MedInc	HouseAge	AveRooms	AveBedrms	Population	AveOccup	Latitude	Longitude
0	8.3252	41.0	6.984127	1.023810	322.0	2.555556	37.88	-122.23
1	8.3014	21.0	6.238137	0.971880	2401.0	2.109842	37.86	-122.22
2	7.2574	52.0	8.288136	1.073446	496.0	2.802260	37.85	-122.24
...	...	...	...	...	...	...	...	...
20637	1.7000	17.0	5.205543	1.120092	1007.0	2.325635	39.43	-121.22
20638	1.8672	18.0	5.329513	1.171920	741.0	2.123209	39.43	-121.32
20639	2.3886	16.0	5.254717	1.162264	1387.0	2.616981	39.37	-121.24

20640 rows × 8 columns

Train/validation/test split

X_main, X_test, y_main, y_test = train_test_split(
    features, target, test_size=0.2, random_state=1)
X_train, X_val, y_train, y_val = train_test_split(
    X_main, y_main, test_size=0.25, random_state=1)

num_features = features.shape[1]
print(X_train.shape, X_val.shape, X_test.shape)

(12384, 8) (4128, 8) (4128, 8)

Linear regression baseline

Refit the linear regression from earlier; we’ll compare neural networks against this baseline.

lr = LinearRegression()
lr.fit(X_train, y_train)

mse_lr_train = mean_squared_error(y_train, lr.predict(X_train))
mse_lr_val = mean_squared_error(y_val, lr.predict(X_val))

mse_train = {"Linear Regression": mse_lr_train}
mse_val = {"Linear Regression": mse_lr_val}

Our First Neural Network

What are Keras and PyTorch?

Keras is a common way of specifying, training, and using neural networks. It gives a simple interface to various backend libraries, including PyTorch.

The Keras application programming interface (API)

Create a Keras ANN model

Decide on the architecture: a simple fully-connected network with one hidden layer with 30 neurons.

Create the model:

1model = Sequential(
    [Input((num_features,)),
     Dense(30, activation="leaky_relu"), 
     Dense(1, activation="leaky_relu")]
)

1: Defines a feed-forward model architecture

This neural network architecture includes one hidden layer with 30 neurons and an output layer with 1 neuron. An activation function is specified (leaky_relu) for both the hidden layer and the output layer. In situations where no activation function is specified for the output layer, it assumes a linear activation.

What is meant by Sequential? The outputs in a given layer become inputs into the next layer, and so on.

Inspect the model

model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 30)             │           270 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 1)              │            31 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 301 (1.18 KB)

 Trainable params: 301 (1.18 KB)

 Non-trainable params: 0 (0.00 B)

Note: the output shapes have None for the number of rows because the predictions have not yet been made. The None is a placeholder for the potential predictions.

The model is initialised randomly

When fitting the ANN, we need to have some initial values for the weights and biases. These are chosen randomly.

model = Sequential([Dense(30, activation="leaky_relu"), Dense(1, activation="leaky_relu")])
model.predict(X_val.head(3), verbose=0)

array([[-139.05],
       [ -84.57],
       [  -5.82]], dtype=float32)

model = Sequential([Dense(30, activation="leaky_relu"), Dense(1, activation="leaky_relu")])
model.predict(X_val.head(3), verbose=0)

array([[-108.21],
       [ -64.74],
       [  -7.1 ]], dtype=float32)

We can see how rerunning the same code with the same input data results in significantly different predictions. This is due to the random initialization.

Controlling the randomness

random.seed(123)

model = Sequential([Dense(30, activation="leaky_relu"), Dense(1, activation="leaky_relu")])

display(model.predict(X_val.head(3), verbose=0))

random.seed(123)
model = Sequential([Dense(30, activation="leaky_relu"), Dense(1, activation="leaky_relu")])

display(model.predict(X_val.head(3), verbose=0))

array([[-81.4 ],
       [-48.06],
       [ -2.77]], dtype=float32)

array([[-81.4 ],
       [-48.06],
       [ -2.77]], dtype=float32)

By setting the seed, we can control for the randomness.

Fit the model

random.seed(123)

model = Sequential([
    Dense(30, activation="leaky_relu"),
    Dense(1, activation="leaky_relu")
])

model.compile("adam", "mse")
%time hist = model.fit(X_train, y_train, epochs=5, verbose=False)
hist.history["loss"]

CPU times: user 3.77 s, sys: 133 ms, total: 3.91 s
Wall time: 3.84 s

[63.02413558959961,
 3.4991047382354736,
 3.3900036811828613,
 2.1773459911346436,
 2.335667848587036]

The above code explains how we would fit a basic neural network.

Define the seed for reproducibility.
Define the architecture of the model.
Compile the model.
Fit the model.
Return the calculate loss function (mse) at the end of each epoch.

Compiling the model (step 3) involves giving instructions on how we want the model to be trained. At the least, we must define the optimizer and loss function. The optimizer explains how the model should learn (how the model should update the weights), and the loss function states the objective that the model needs to optimize. In the above code, we use adam as the optimizer and mse (mean squared error) as the loss function.

The fit() function (step 4) takes in the training data, and runs the entire dataset through 5 epochs before training completes. What this means is that the model is run through the entire dataset 5 times. Suppose we start the training process with the random initialization, run the model through the entire data, calculate the mse (after 1 epoch), and update the weights using the adam optimizer. Then we run the model through the entire dataset once again with the updated weights, to calculate the mse at the end of the second epoch. Likewise, we would run the model 5 times before the training completes.

Epoch: Each step looks at a batch of data (say, the first 10 observations), fits the model, compares with the actual values, and updates the weights and biases to improve the loss function (this is 1 update). Then it moves on to the next batch of data, and updates the weights and biases another time… Once it reaches the end of the dataset, it loops back to the beginning of the dataset. This is defined as one single epoch.

%time command computes and prints the amount of time spend on training. By setting verbose=False we can avoid printing of intermediate results during training. Setting verbose=True is useful when we want to observe how the neural network is training.

Make predictions

y_pred = model.predict(X_train[:3], verbose=0)
y_pred

array([[ 0.29],
       [-0.91],
       [-0.17]], dtype=float32)

Note

The .predict gives us a ‘matrix’ not a ‘vector’. Calling .flatten() will convert it to a ‘vector’.

print(f"Original shape: {y_pred.shape}")
y_pred = y_pred.flatten()
print(f"Flattened shape: {y_pred.shape}")
y_pred

Original shape: (3, 1)
Flattened shape: (3,)

array([ 0.29, -0.91, -0.17], dtype=float32)

Plot the predictions

One problem with the predictions is that lots of predictions include negative values, which is unrealistic for house prices. We might have to rethink the activation function in the output layer.

Assess the model

y_pred = model.predict(X_val, verbose=0)
mean_squared_error(y_val, y_pred)

2.845539574957286

mse_train["Basic ANN"] = mean_squared_error(
    y_train, model.predict(X_train, verbose=0)
)
mse_val["Basic ANN"] = mean_squared_error(y_val, model.predict(X_val, verbose=0))

Some predictions are negative:

y_pred = model.predict(X_val, verbose=0)
y_pred.min(), y_pred.max()

(np.float32(-4.7767005), np.float32(7.326745))

y_val.min(), y_val.max()

(np.float64(0.225), np.float64(5.00001))

Force positive predictions

We noted that a lot of the predictions include negative values which is unrealistic for house prices. How can we force the predictions to be positive?

Try running for longer

random.seed(123)

model = Sequential([
    Dense(30, activation="leaky_relu"),
    Dense(1, activation="leaky_relu")
])

model.compile("adam", "mse")

%time hist = model.fit(X_train, y_train, epochs=50, verbose=False)

CPU times: user 35.2 s, sys: 1.03 s, total: 36.3 s
Wall time: 35.5 s

We will train the same neural network architecture with more epochs (epochs=50) to see if the results improve.

Loss curve

plt.plot(range(1, 51), hist.history["loss"])
plt.xlabel("Epoch")
plt.ylabel("MSE");

The loss curve experiences a sudden drop even before finishing 5 epochs and remains consistently low. This indicates that increasing the number of epochs from 5 to 50 does not significantly increase the accuracy.

Loss curve

plt.plot(range(2, 51), hist.history["loss"][1:])
plt.xlabel("Epoch")
plt.ylabel("MSE");

The above code filters out the MSE value from the first epoch. It plots the vector of MSE values starting from the 2nd epoch. By doing so, we can observe the fluctuations in the MSE values across different epochs more clearly. Results show that the model does not benefit from increasing the epochs.

Predictions

y_pred = model.predict(X_val, verbose=0)
print(f"Min prediction: {y_pred.min():.2f}")
print(f"Max prediction: {y_pred.max():.2f}")

Min prediction: -2.35
Max prediction: 8.03

plt.scatter(y_pred, y_val)
plt.xlabel("Predictions")
plt.ylabel("True values")
add_diagonal_line()

mse_train["Long run ANN"] = mean_squared_error(
    y_train, model.predict(X_train, verbose=0)
)
mse_val["Long run ANN"] = mean_squared_error(y_val, model.predict(X_val, verbose=0))

While there is an improvement, we are still seeing some negative predictions. What else can we try, other than training the NN for longer?

Try different activation functions

We can choose a different activation function for the final layer, namely, an activation function that guarantees a positive output.

We should be mindful when selecting the activation function. Both tanh and sigmoid functions restrict the output values to the range of [0,1]. This is not sensible for house price modelling. softplus does not have that problem. Also, softplus ensures the output is positive which is realistic for house prices.

Enforce positive outputs (softplus)

random.seed(123)

model = Sequential([
    Dense(30, activation="leaky_relu"),
    Dense(1, activation="softplus")
])

model.compile("adam", "mse")

%time hist = model.fit(X_train, y_train, epochs=50, \
    verbose=False)

losses = np.round(hist.history["loss"], 2)
print(losses[:5], "...", losses[-5:])

CPU times: user 34.1 s, sys: 892 ms, total: 35 s
Wall time: 34.3 s
[5.65 5.64 5.64 5.64 5.64] ... [5.64 5.64 5.64 5.64 5.64]

Plot the predictions

Plots illustrate how all the outputs were stuck at zero. Irrespective of how many epochs we run, the output would always be zero.

Enforce positive outputs (\mathrm{e}^{\,x})

random.seed(123)

model = Sequential([
    Dense(30, activation="leaky_relu"),
    Dense(1, activation="exponential")
])

model.compile("adam", "mse")

%time hist = model.fit(X_train, y_train, epochs=5, verbose=False)

losses = hist.history["loss"]
print(losses)

CPU times: user 3.44 s, sys: 88.1 ms, total: 3.53 s
Wall time: 3.46 s
[50286.49609375, 6.613603591918945, 6.612048149108887, 6.609488010406494, 6.605171203613281]

Training the model again with an exponential activation function will give nan values. This is because the outputs can explode easily.

Same as transforming the target

A benefit of NNs is that you don’t have to manually transform the features like you do when fitting a GLM.

(Kelley Pace & Barry, 1997) studied the California house price dataset. This was one of the models they fit:

$$\begin{align} \ln(\text{MedHouseVal}) = & \beta_0 + \beta_1\text{MedInc} + \beta_2\text{MedInc}^2 + \beta_3\text{MedInc}^3 \\ & + \beta_4\ln(\text{HouseAge})+ \beta_5\ln(\text{AveRooms}/\text{Population}) \\ & + \beta_6\ln(\text{AveBedrms}/\text{Population}) + \beta_7\ln(\text{Population}/\text{AveOccup}) \\ & + \beta_8\ln(\text{AveOccup}) \end{align}$$

Note

Fitting \ln(\text{Median Value}) is mathematically identical to the exponential activation function in the final layer (but metrics are in different units).

Good to know others results

If you find that someone else has fit a model to the same dataset, you can compare your metrics to theirs in your report. For example, (Kelley Pace & Barry, 1997) studied this dataset. You can compare the R^2 of their models to the ones we fit here.

GPT can double-check these results

You can ask GPT to fit the linear model shown above to the data using Python and calculate the R^2.

I’d previously given it the CSV of the data.

The resulting R^2 is equal to the one documented in the article.

Preprocessing

Re-scaling the inputs

Neural networks prefer if the inputs range between -1 and 1.

scaler = StandardScaler()
scaler.fit(X_train)

X_train_sc = scaler.transform(X_train)
X_val_sc = scaler.transform(X_val)
X_test_sc = scaler.transform(X_test)

Note: We apply both the fit and transform operations on the train data. However, we only apply transform on the validation and test data.

plt.hist(X_train.iloc[:, 0])
plt.hist(X_train_sc[:, 0])
plt.legend(["Original", "Scaled"]);

We can see how the original values for the input varied between 0 and 10, and how the scaled input values are now between -2 and 2.5.

Same model with scaled inputs

random.seed(123)

model = Sequential([
    Dense(30, activation="leaky_relu"),
    Dense(1, activation="exponential")
])

model.compile("adam", "mse")

%time hist = model.fit( \
    X_train_sc, \
    y_train, \
    epochs=50, \
    verbose=False)

CPU times: user 33.2 s, sys: 807 ms, total: 34 s
Wall time: 33.4 s

Loss curve

plt.plot(range(1, 51), hist.history["loss"])
plt.xlabel("Epoch")
plt.ylabel("MSE");

Loss curve

plt.plot(range(2, 51), hist.history["loss"][1:])
plt.xlabel("Epoch")
plt.ylabel("MSE");

Predictions

y_pred = model.predict(X_val_sc, verbose=0)
print(f"Min prediction: {y_pred.min():.2f}")
print(f"Max prediction: {y_pred.max():.2f}")

Min prediction: 0.00
Max prediction: 8.11

plt.scatter(y_pred, y_val)
plt.xlabel("Predictions")
plt.ylabel("True values")
add_diagonal_line()

mse_train["Exp ANN"] = mean_squared_error(
    y_train, model.predict(X_train_sc, verbose=0)
)
mse_val["Exp ANN"] = mean_squared_error(y_val, model.predict(X_val_sc, verbose=0))

Now the predictions are always non-negative.

Comparing MSE (smaller is better)

On training data:

mse_train

{'Linear Regression': 0.5291948207479792,
 'Basic ANN': 2.8278137790106026,
 'Long run ANN': 0.5999134938371403,
 'Exp ANN': 0.35574261079097375}

On validation data (expect worse, i.e. bigger):

mse_val

{'Linear Regression': 0.5059420205381369,
 'Basic ANN': 2.845539574957286,
 'Long run ANN': 0.6106608478598903,
 'Exp ANN': 0.347549802095409}

Note: The error on the validation set is usually higher than the training set.

Comparing models (train)

train_results = pd.DataFrame(
    {"Model": mse_train.keys(), "MSE": mse_train.values()}
)
train_results.sort_values("MSE", ascending=False)

	Model	MSE
1	Basic ANN	2.827814
2	Long run ANN	0.599913
0	Linear Regression	0.529195
3	Exp ANN	0.355743

Comparing models (validation)

val_results = pd.DataFrame(
    {"Model": mse_val.keys(), "MSE": mse_val.values()}
)
val_results.sort_values("MSE", ascending=False)

	Model	MSE
1	Basic ANN	2.845540
2	Long run ANN	0.610661
0	Linear Regression	0.505942
3	Exp ANN	0.347550

The neural network with exponential activation function, scaled inputs and 50 epochs performed the best on the training and validation sets.

Early Stopping

Choosing when to stop training

Early stopping can be seen as a regularization technique to avoid overfitting. The plot shows that both training error and validation error decrease at the beginning of training process. However, after a while, validation error starts to increase while training error keeps on decreasing. This is an indication of overfitting. Overfitting leads to poor performance on the unseen data, which is seen here through the gradual increase of validation error. Early stopping can track the model’s performance through the training process and stop the training at the right time.

Try early stopping

Hinton calls it a “beautiful free lunch”

2random.seed(123)
3model = Sequential([
    Dense(30, activation="leaky_relu"),
    Dense(1, activation="exponential")
])
4model.compile("adam", "mse")

5es = EarlyStopping(restore_best_weights=True, patience=15)

%time hist = model.fit(X_train_sc, y_train, epochs=1_000, \
6    callbacks=[es], validation_data=(X_val_sc, y_val), verbose=False)
7print(f"Keeping model at epoch #{len(hist.history['loss'])-15}.")

2: Sets the random seed
3: Constructs the sequential model
4: Configures the training process with optimiser and loss function
5: Defines the early stopping object. Here, the patience parameter tells how many epochs the neural network has to wait with no improvement before the process stops. patience=15 indicates that the neural network will wait for 15 epochs without any improvement before it stops training. restore_best_weights=True ensures that model’s weights will be restored to the best model, i.e., the model we saw before 15 epochs earlier
6: Fits the model with early stopping object passed in
7: Prints the outs

CPU times: user 22.6 s, sys: 570 ms, total: 23.1 s
Wall time: 22.7 s
Keeping model at epoch #12.

Loss curve

We can look at the training and validation loss function ("mse") across epochs. The validation error is lowest at the end of epoch 8 (starting from 0), so the weights are restored to that model.

plt.plot(hist.history["loss"])
plt.plot(hist.history["val_loss"])
plt.legend(["Training", "Validation"]);

Loss curve II

plt.plot(hist.history["loss"])
plt.plot(hist.history["val_loss"])
plt.ylim([0, 8])
plt.legend(["Training", "Validation"]);

Predictions

Comparing models (validation)

	Model	MSE
1	Basic ANN	2.845540
2	Long run ANN	0.610661
0	Linear Regression	0.505942
4	Early stop ANN	0.359737
3	Exp ANN	0.347550

In this case, early stopping did not improve the ANN (0.387 vs. 0.369). Ultimately, we hope that these methods improve the model but that’s not always the case. This shows the importance of validating your models.

The test set

Evaluate only the final/selected model on the test set.

mean_squared_error(y_test, model.predict(X_test_sc, verbose=0))

0.36174060505869826

model.evaluate(X_test_sc, y_test, verbose=False)

0.3617406189441681

Evaluating the model on the unseen test set provides an unbiased view on how the model will perform. Since we configured the model to track ‘mse’ as the loss function, we can simply use model.evaluate() function on the test set and get the same answer.

Another useful callback

random.seed(123)
model = Sequential(
    [Dense(30, activation="leaky_relu"), Dense(1, activation="exponential")]
)
model.compile("adam", "mse")
mc = ModelCheckpoint(
    "best-model.keras", monitor="val_loss", save_best_only=True
)
es = EarlyStopping(restore_best_weights=True, patience=5)
hist = model.fit(
    X_train_sc,
    y_train,
    epochs=100,
    validation_split=0.1,
    callbacks=[mc, es],
    verbose=False,
)
Path("best-model.keras").stat().st_size

ModelCheckpoint is also another useful callback function that can be used to save the model at some intervals during training. This is useful when training large datasets. If the training process gets interrupted at some point, last saved set of weights from model checkpoints can be used to resume the training process instead of starting from the beginning.

References

Keras model methods

compile: specify the loss function and optimiser
fit: learn the parameters of the model
predict: apply the model
evaluate: apply the model and calculate a metric

random.seed(12)
model = Sequential()
model.add(Dense(1, activation="relu"))
model.compile("adam", "poisson")
model.fit(X_train, y_train, verbose=0)
y_pred = model.predict(X_val, verbose=0)
print(model.evaluate(X_val, y_val, verbose=0))

4.442193984985352

Package Versions

from watermark import watermark
print(watermark(python=True, packages="keras,matplotlib,numpy,pandas,seaborn,scipy,torch"))

Python implementation: CPython
Python version       : 3.14.3
IPython version      : 9.13.0

keras     : 3.14.1
matplotlib: 3.10.9
numpy     : 2.4.4
pandas    : 3.0.2
seaborn   : 0.13.2
scipy     : 1.17.1
torch     : 2.11.0

Glossary

activations, activation function
artificial neural network
biases (in neurons)
callbacks
classification problem
cost/loss function
deep network, network depth
dense or fully-connected layer
early stopping
epoch
feed-forward neural network

Keras, TensorFlow, PyTorch
labelled/unlabelled data
machine learning
matplotlib
minimax algorithm
neural network architecture
perceptron
ReLU
representation learning
sigmoid activation function
targets
training/validation/test split
weights (in a neuron)

References

Kelley Pace, R., & Barry, R. (1997). Sparse spatial autoregressions. Statistics & Probability Letters, 33(3), 291–297.

Shang, Z., Chauhan, V., Devi, K., & Patil, S. (2025). Artificial intelligence, the digital surgeon: Unravelling its emerging footprint in healthcare – the narrative review. Journal of Multidisciplinary Healthcare, 17, 4011–4022.