Entity Embedding

ACTL3143 & ACTL5111 Deep Learning for Actuaries

Author

Patrick Laub

Show the package imports

import random
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

from keras.models import Sequential
from keras.layers import Dense
from keras.callbacks import EarlyStopping

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder, StandardScaler, OrdinalEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression
from sklearn import set_config

set_config(transform_output="pandas")

Word Embeddings

Overview

In order for deep learning models to process language, we need to supply that language to the model in a way it can digest, i.e. a quantitative representation such as a 2-D matrix of numerical values.

Popular methods for converting text into numbers include:

One-hot encoding
Bag of words
TF-IDF
Word vectors (transfer learning)

Word Vectors

One-hot representations capture word ‘existence’ only, whereas word vectors capture information about word meaning as well as location.
This enables deep learning NLP models to automatically learn linguistic features.
Word2Vec & GloVe are popular algorithms for generating word embeddings (i.e. word vectors).

Word Vectors II

Word vectors are a type of word embedding which can return numerical representations of words in a continuous vector space. Their representations capture semantic knowledge of the words. For example, we can see how words of the same gender are positioned closer to each other in a n-dimensional space. We also see that capital cities are near their countries, and countries that are geographically close are grouped together.

Overarching concept is to assign each word within a corpus to a particular, meaningful location within a multidimensional space called the vector space.
Initially each word is assigned to a random location.
BUT by considering the words that tend to be used around a given word within the corpus, the locations of the words shift.

Remember this diagram?

Embeddings will gradually improve during training.

Embeddings are numerical representations of categorical data that were learned during the supervised learning process. However, numerical representations like Word2Vec & GloVe are popular algorithms for generating word embeddings that were trained by others, i.e. they are pre-trained.

Word2Vec

Key idea: You’re known by the company you keep.

Two algorithms are used to calculate embeddings:

Continuous bag of words: uses the context words to predict the target word
Skip-gram: uses the target word to predict the context words

Predictions are made using a neural network with one hidden layer. Through backpropagation, we update a set of “weights” which become the word vectors.

Word2Vec training methods

Continuous bag of words is a *centre word prediction* task

Skip-gram is a *neighbour word prediction* task

Suggested viewing

Computerphile (2019), Vectoring Words (Word Embeddings), YouTube (16 mins).

The skip-gram network

The skip-gram model. Both the input vector \boldsymbol{x} and the output \boldsymbol{y} are one-hot encoded word representations. The hidden layer is the word embedding of size N.

Word Vector Arithmetic

Relationships between words becomes vector math.

E.g., if we calculate the direction and distance between the coordinates of the words Paris and France, and trace this direction and distance from London, we should be close to the word England.

Word Embeddings II

Pretrained word embeddings

GloVe (Global Vectors) are pre-trained word embeddings from Stanford, trained on Wikipedia + Gigaword (6 billion tokens, ~400,000 word vocabulary).

GloVe loading code

from pathlib import Path
from urllib.request import urlretrieve
import zipfile, numpy as np

_zip = Path("../data/glove.6B.zip")
if not _zip.exists():
    urlretrieve("https://downloads.cs.stanford.edu/nlp/data/glove.6B.zip", _zip)

words, _rows = [], []
with zipfile.ZipFile(_zip) as _zf:
    with _zf.open("glove.6B.300d.txt") as _f:
        for _line in _f:
            _parts = _line.split()
            words.append(_parts[0].decode("utf-8"))
            _rows.append(_parts[1:])

vectors = np.array(_rows, dtype=np.float32)
vectors /= np.linalg.norm(vectors, axis=1, keepdims=True)
word_to_index = {w: i for i, w in enumerate(words)}

def vec(word):
    return vectors[word_to_index[word]]

def similarity(a, b):
    return float(vec(a) @ vec(b))

def nearest(vector, n=10, exclude=()):
    v = vector / np.linalg.norm(vector)
    sims = vectors @ v
    results = []
    for i in np.argsort(-sims):
        w = words[i]
        if w not in exclude:
            results.append((w, float(sims[i])))
            if len(results) == n:
                break
    return results

def analogy(a, b, c, n=10):
    """a is to b as c is to ?   e.g. analogy("man", "king", "woman") → queen"""
    query = vec(b) - vec(a) + vec(c)
    return nearest(query, n=n, exclude={a, b, c})

f"The size of the vocabulary is {len(words)}"

'The size of the vocabulary is 400000'

Look up a word vector

vec("pizza")

array([ 0.04,  0.07,  0.06, -0.  , -0.03,  0.03, -0.01, -0.04, -0.07,
        0.01, -0.04, -0.1 , -0.03,  0.09, -0.05,  0.  , -0.02, -0.02,
       -0.05,  0.06,  0.05,  0.11, -0.01, -0.02,  0.06, -0.01,  0.04,
        0.  ,  0.06, -0.18, -0.08,  0.07,  0.05, -0.04, -0.08,  0.05,
       -0.06, -0.05, -0.07,  0.04,  0.02,  0.01,  0.  ,  0.08, -0.02,
        0.05,  0.15,  0.02,  0.01, -0.03, -0.01, -0.1 ,  0.05,  0.08,
       -0.03, -0.04, -0.01,  0.05,  0.02, -0.04,  0.12, -0.04, -0.  ,
       -0.07, -0.02, -0.05, -0.03,  0.07, -0.04,  0.04,  0.08,  0.02,
       -0.01, -0.06,  0.02, -0.03, -0.02, -0.01, -0.01, -0.05, -0.02,
        0.06,  0.01, -0.06, -0.02, -0.07,  0.04,  0.04, -0.05,  0.01,
        0.04,  0.02, -0.02, -0.02, -0.  ,  0.01, -0.04,  0.03, -0.06,
        0.01,  0.05, -0.01,  0.  , -0.07, -0.01, -0.06,  0.02,  0.11,
       -0.03,  0.02,  0.05,  0.  ,  0.04, -0.09,  0.06, -0.09, -0.09,
        0.04, -0.05, -0.01, -0.03,  0.02,  0.12, -0.02, -0.04,  0.03,
        0.01,  0.02, -0.05, -0.02,  0.01,  0.06, -0.03, -0.  ,  0.03,
       -0.03,  0.  ,  0.03, -0.02,  0.06,  0.02,  0.01, -0.04, -0.06,
       -0.04,  0.02, -0.01,  0.06, -0.01, -0.07, -0.07,  0.15,  0.05,
       -0.01, -0.07, -0.04, -0.02,  0.04, -0.05, -0.07,  0.07,  0.06,
       -0.06,  0.05, -0.  ,  0.03,  0.01, -0.01,  0.07, -0.03,  0.01,
        0.03, -0.07,  0.05, -0.08,  0.09,  0.01,  0.03,  0.08, -0.11,
       -0.  ,  0.03,  0.05, -0.01,  0.02, -0.02,  0.04,  0.06,  0.05,
       -0.04,  0.09,  0.11, -0.03, -0.02, -0.03,  0.02, -0.12, -0.03,
       -0.06,  0.03,  0.06, -0.1 ,  0.17,  0.04, -0.12, -0.03,  0.11,
       -0.  ,  0.03, -0.  , -0.06,  0.03,  0.06, -0.08, -0.1 ,  0.02,
        0.03, -0.03, -0.03,  0.11,  0.08,  0.06,  0.01, -0.  , -0.11,
       -0.09, -0.01, -0.09, -0.04, -0.04,  0.04,  0.07, -0.02, -0.01,
        0.05, -0.01,  0.17,  0.01, -0.13,  0.03, -0.05, -0.02, -0.01,
       -0.12, -0.07, -0.03, -0.01,  0.06, -0.03, -0.06,  0.17,  0.08,
       -0.01,  0.1 ,  0.02,  0.05, -0.02,  0.04, -0.04,  0.03,  0.05,
       -0.06, -0.02,  0.02,  0.01, -0.06,  0.04,  0.06,  0.07, -0.02,
       -0.09, -0.07,  0.01,  0.06,  0.13, -0.01, -0.17,  0.01, -0.18,
       -0.02, -0.02,  0.05, -0.04,  0.03, -0.01,  0.02,  0.08, -0.04,
        0.05,  0.01,  0.03, -0.  , -0.03,  0.04, -0.07, -0.05,  0.05,
        0.  , -0.06,  0.01], dtype=float32)

len(vec("pizza"))

Each word is characterised by 300 numbers; a 300-dimensional vector.

Find nearby word vectors

We can find words similar to a given word, or compute the similarity between two words.

nearest(vec("python"))

[('python', 1.0),
 ('monty', 0.6837381720542908),
 ('perl', 0.519283652305603),
 ('cleese', 0.5092198848724365),
 ('pythons', 0.5007115006446838),
 ('php', 0.49423137307167053),
 ('grail', 0.4683017432689667),
 ('scripting', 0.467612624168396),
 ('skit', 0.4474538564682007),
 ('javascript', 0.4312553107738495)]

similarity("python", "java")

0.35804617404937744

similarity("python", "sport")

0.005185101181268692

similarity("python", "r")

0.06078970432281494

What does ‘similarity’ mean?

The ‘similarity’ scores

similarity("sydney", "melbourne")

0.7951619625091553

are normally based on cosine distance.

x = vec("sydney")
y = vec("melbourne")
x.dot(y) / (np.linalg.norm(x) * np.linalg.norm(y))

np.float32(0.795162)

similarity("sydney", "aarhus")

0.14399726688861847

Weng’s GoT Word2Vec

In the Game of Thrones (GoT) word embedding space, the top similar words to “king” and “queen” are:

model.most_similar("king")

('kings', 0.897245) 
('baratheon', 0.809675) 
('son', 0.763614)
('robert', 0.708522)
('lords', 0.698684)
('joffrey', 0.696455)
('prince', 0.695699)
('brother', 0.685239)
('aerys', 0.684527)
('stannis', 0.682932)

model.most_similar("queen")

('cersei', 0.942618)
('joffrey', 0.933756)
('margaery', 0.931099)
('sister', 0.928902)
('prince', 0.927364)
('uncle', 0.922507)
('varys', 0.918421)
('ned', 0.917492)
('melisandre', 0.915403)
('robb', 0.915272)

Combining word vectors

You can summarise a sentence by averaging the individual word vectors.

sv = (vec("melbourne") + vec("has") + vec("better") + vec("coffee")) / 4
len(sv), sv[:5]

(300, array([-0.03,  0.03,  0.01,  0.01, -0.01], dtype=float32))

As it turns out, averaging word embeddings is a surprisingly effective way to create word embeddings. It’s not perfect (as you’ll see), but it does a strong job of capturing what you might perceive to be complex relationships between words.

Recipe recommender

Recipes are the average of the word vectors of the ingredients.

Nearest neighbours used to classify new recipes as potentially delicious.

Analogies with word vectors

Obama is to America as ___ is to Australia.

\text{Obama} - \text{America} + \text{Australia} = ?

analogy("america", "obama", "australia")

[('barack', 0.5462056994438171),
 ('canberra', 0.48004958033561707),
 ('australian', 0.4731597900390625),
 ('rudd', 0.46835529804229736),
 ('bush', 0.431667685508728),
 ('mccain', 0.42993825674057007),
 ('gillard', 0.42073196172714233),
 ('australians', 0.4089258909225464),
 ('sydney', 0.40701723098754883),
 ('zealand', 0.3963095545768738)]

Testing more associations

analogy("paris", "france", "london")

[('britain', 0.7673852443695068),
 ('england', 0.62794429063797),
 ('uk', 0.6197735071182251),
 ('british', 0.6013830900192261),
 ('ireland', 0.5365166664123535),
 ('u.k.', 0.5294837355613708),
 ('scotland', 0.5195812582969666),
 ('australia', 0.5150107145309448),
 ('wales', 0.5144591927528381),
 ('europe', 0.5047693848609924)]

What country is to London like France is to Paris?

\text{France} + \text{London} - \text{Paris} = ?

France is a country whose capital city is Paris. If we remove Paris and add London (a city), we are looking for the country for which London is the capital city.

Quickly get to bad associations

analogy("man", "king", "woman")

[('queen', 0.6713277101516724),
 ('princess', 0.5432624816894531),
 ('throne', 0.538610577583313),
 ('monarch', 0.5347574949264526),
 ('daughter', 0.49802514910697937),
 ('mother', 0.49564433097839355),
 ('elizabeth', 0.4832652807235718),
 ('kingdom', 0.47747093439102173),
 ('prince', 0.4668240547180176),
 ('wife', 0.4647327661514282)]

analogy("man", "programmer", "woman")

[('programmers', 0.4976009428501129),
 ('freelance', 0.41725867986679077),
 ('educator', 0.40316858887672424),
 ('businesswoman', 0.3929097056388855),
 ('designer', 0.39289426803588867),
 ('translator', 0.38584306836128235),
 ('technician', 0.3751075267791748),
 ('computer', 0.3749142289161682),
 ('animator', 0.3677002191543579),
 ('homemaker', 0.3675471544265747)]

What is the ‘woman’ version of a computer programmer? Seems a strange question to ask, and we get some even stranger responses…

Bias in NLP models

The Verge (2016), Twitter taught Microsoft’s AI chatbot to be a racist a****** in less than a day.

… there are serious questions to answer, like how are we going to teach AI using public data without incorporating the worst traits of humanity? If we create bots that mirror their users, do we care if their users are human trash? There are plenty of examples of technology embodying — either accidentally or on purpose — the prejudices of society, and Tay’s adventures on Twitter show that even big corporations like Microsoft forget to take any preventative measures against these problems.

The `analogy` function cheats a little bit

nearest(vec("programmer") - vec("man") + vec("woman"))

[('programmer', 0.7800661325454712),
 ('programmers', 0.4976009428501129),
 ('freelance', 0.41725867986679077),
 ('educator', 0.40316858887672424),
 ('businesswoman', 0.3929097056388855),
 ('designer', 0.39289426803588867),
 ('translator', 0.38584306836128235),
 ('technician', 0.3751075267791748),
 ('computer', 0.3749142289161682),
 ('animator', 0.3677002191543579)]

To get the ‘nice’ analogies, analogy ignores the input words as possible answers.

# ignore (don't return) keys from the input
if w not in exclude:
    results.append((w, float(sims[i])))

Car Crash NLP Part II

Predict injury severity

Now we are note predicting the number of cars involved in a crash, but the injury severity.

features = df["SUMMARY_EN"]
target = LabelEncoder().fit_transform(df["INJSEVB"])

X_main, X_test, y_main, y_test = \
    train_test_split(features, target, test_size=0.2, random_state=1)
X_train, X_val, y_train, y_val = \
    train_test_split(X_main, y_main, test_size=0.25, random_state=1)
X_train.shape, X_val.shape, X_test.shape

((4169,), (1390,), (1390,))

Using TF-IDF Vectorization

Rather than BoW, we use TF-IDF (not removing stop-words). Now, words that appear frequently across the police reports are given less importance.

from sklearn.feature_extraction.text import TfidfVectorizer

max_tokens = 1_000
vect = TfidfVectorizer(
    max_features=max_tokens,
    lowercase=True,
    token_pattern=r"(?u)\b\w+\b",  # Similar to "lower_and_strip_punctuation"
)

X_train_txt = vect.fit_transform(X_train).toarray()
X_val_txt = vect.transform(X_val).toarray()
X_test_txt = vect.transform(X_test).toarray()

vocab = vect.get_feature_names_out()
print(list(vocab[:50]))

['0', '1', '10', '105', '12', '15', '150', '16', '17', '18', '19', '1990', '1991', '1992', '1993', '1994', '1995', '1996', '1997', '1998', '1999', '2', '20', '2000', '2001', '2002', '2003', '2004', '2005', '2006', '2007', '21', '22', '23', '24', '25', '26', '27', '28', '29', '3', '30', '30mph', '31', '32', '33', '34', '35', '35mph', '36']

The TF-IDF vectors

pd.DataFrame(X_train_txt, columns=vocab, index=X_train.index)

	0	1	10	105	12	15	150	16	17	18	...	worked	working	works	would	yaw	year	years	yellow	yield	zone
2532	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.027580	0.0	0.019694	0.0	0.000000	0.0	0.0
6209	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.045898	0.0	0.032775	0.0	0.000000	0.0	0.0
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
206	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.000000	0.0	0.000000	0.0	0.051053	0.0	0.0
6356	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.000000	0.0	0.035219	0.0	0.000000	0.0	0.0

4169 rows × 1000 columns

Feed TF-IDF into an ANN

random.seed(42)
tfidf_model = keras.models.Sequential([
    layers.Input((X_train_txt.shape[1],)),
    layers.Dense(250, "relu"),
    layers.Dense(1, "sigmoid")
])

tfidf_model.compile("adam", "binary_crossentropy", metrics=["accuracy"])
tfidf_model.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 250)            │       250,250 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 1)              │           251 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 250,501 (978.52 KB)

 Trainable params: 250,501 (978.52 KB)

 Non-trainable params: 0 (0.00 B)

Fit & evaluate

es = keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True,
    monitor="val_accuracy", verbose=2)

if not Path("models/tfidf-model.keras").exists():
    tfidf_model.fit(X_train_txt, y_train, epochs=1_000, callbacks=[es],
        validation_data=(X_val_txt, y_val), verbose=0)
    tfidf_model.save("models/tfidf-model.keras")
else:
    tfidf_model = keras.models.load_model("models/tfidf-model.keras")

Epoch 17: early stopping
Restoring model weights from the end of the best epoch: 7.

tfidf_model.evaluate(X_train_txt, y_train, verbose=0, batch_size=1_000)

[0.131618469953537, 0.9589829444885254]

tfidf_model.evaluate(X_val_txt, y_val, verbose=0, batch_size=1_000)

[0.26651373505592346, 0.8949640393257141]

Keep text as sequence of tokens

from sklearn.feature_extraction.text import CountVectorizer
import re

max_length = 500
max_tokens = 1_000

# Build vocabulary using CountVectorizer
count_vect = CountVectorizer(max_features=max_tokens-1, lowercase=True,
    token_pattern=r"(?u)\b\w+\b")
count_vect.fit(X_train)
vocab = [""] + list(count_vect.get_feature_names_out())  # Index 0 reserved for padding
word_to_idx = {word: idx for idx, word in enumerate(vocab)}

def texts_to_sequences(texts, word_to_idx, max_length):
    sequences = []
    for text in texts:
        tokens = re.findall(r"(?u)\b\w+\b", text.lower())
        seq = [word_to_idx.get(t, 0) for t in tokens][:max_length]
        seq = seq + [0] * (max_length - len(seq))  # Pad with zeros
        sequences.append(seq)
    return np.array(sequences)

X_train_txt = texts_to_sequences(X_train, word_to_idx, max_length)
X_val_txt = texts_to_sequences(X_val, word_to_idx, max_length)
X_test_txt = texts_to_sequences(X_test, word_to_idx, max_length)

print(vocab[:50])

['', '0', '1', '10', '105', '12', '15', '150', '16', '17', '18', '19', '1990', '1991', '1992', '1993', '1994', '1995', '1996', '1997', '1998', '1999', '2', '20', '2000', '2001', '2002', '2003', '2004', '2005', '2006', '2007', '21', '22', '23', '24', '25', '26', '27', '28', '29', '3', '30', '30mph', '31', '32', '33', '34', '35', '35mph']

A sequence of integers

X_train_txt[0]

array([880, 249, 619, 469, 870, 319,  96, 620,  77, 963, 469, 870, 568,
       620,  77, 392, 958, 849, 493, 870, 493, 224, 620,  77, 605, 818,
        61, 513, 924, 514, 317, 284, 191, 919, 513, 741, 491, 178, 108,
       320, 969,  61, 513, 924, 514, 317, 284, 191, 919, 513, 741, 235,
       178,  77, 691,   0, 900, 788, 870, 161, 605, 818, 741, 957, 313,
       109, 843, 984,  77, 807, 672, 809, 870, 675, 823, 957,  64, 507,
        48, 587, 870, 900, 382, 957, 604, 109, 874, 968, 601,  93, 134,
       219, 133, 870, 885, 620, 870, 249, 943,  77,  17, 179,   0,   0,
       957, 840, 133, 870, 493, 355, 818, 469, 513, 883, 954, 387, 870,
       788, 889, 193, 815, 501, 244, 919, 521, 944,  77,  24, 297,   0,
       957, 606, 469, 513, 924, 118, 870, 759, 493, 976, 501, 486, 889,
       411, 843, 975, 870, 529, 920, 420, 943, 154, 502, 521, 125, 943,
       231, 870, 919, 502, 306, 761,  77, 667, 949, 984,   0, 531,   0,
       118, 870, 493, 397, 870, 320, 943, 840, 469, 870, 568, 620, 870,
       493, 469,   0, 889, 870, 118, 667, 949, 944, 334, 870, 493, 109,
       848, 870, 398, 620, 943, 984, 502, 398, 521, 239, 166, 950, 180,
       889, 377, 728, 469, 870, 493, 109, 968, 166, 896, 314, 889, 262,
       870, 306, 620, 943,  77,   0, 995, 624, 554, 205, 889, 150, 469,
       413, 435,   0, 863, 678, 563, 387, 281, 441, 165, 682, 109,   0,
       431, 957, 626, 446, 958, 889, 661, 935,   0, 133, 870, 885, 620,
       870, 249, 431, 830, 869, 975, 870, 529, 194, 431, 422, 332,   0,
       889, 216, 446, 919, 177, 840, 609,   0, 968, 889, 411, 975, 431,
       761, 870, 667, 182, 116, 431, 868, 761, 944, 333, 870, 493, 109,
       431,   0, 387, 465, 431, 957, 609, 481, 469, 870, 249, 870, 250,
       677, 342, 387, 943, 957, 210, 880, 949, 907, 639, 870, 513, 533,
       521, 781, 620, 905, 513, 870, 250, 712, 957, 210, 125,  77, 306,
       718, 337,   0, 633,   0, 870, 129, 358, 210, 889, 943, 473, 108,
         0, 339,  87, 431, 778, 429,   0, 870, 493, 109, 609, 840, 125,
       966, 125, 870, 118, 667, 949,   0, 870, 900,   0, 870, 306, 620,
       944,  77,  51, 995, 624, 554, 469, 413, 435, 957, 960, 446, 243,
       525, 387,  77, 588, 218, 133, 870, 885, 620, 870, 249, 431, 957,
       626, 446, 958, 449, 397, 989, 975, 870, 249, 619, 431, 830, 431,
       957, 606, 469, 513,  22, 125, 870, 900, 529, 920, 420, 431, 610,
       870, 949, 469, 870, 521, 355, 818, 422,   0, 889, 919, 521, 109,
       206, 870, 493, 431,   0, 125, 870, 949, 828, 502, 919, 109, 839,
       469, 870, 568, 620, 870, 493, 431, 169, 177, 244, 609, 142, 943,
       431, 630, 761, 870, 667, 949, 984, 502, 531, 109,   0, 626,  95,
       431, 334, 870, 493, 431, 856, 571, 482, 109, 957, 412, 889, 150,
       195, 638, 133,  77, 518,   0])

Feed LSTM a sequence of one-hots

from keras.layers import CategoryEncoding, Bidirectional, LSTM
random.seed(42)
one_hot_model = Sequential([Input(shape=(max_length,), dtype="int64"),
    CategoryEncoding(num_tokens=max_tokens, output_mode="one_hot"),
    Bidirectional(LSTM(24)),
    Dense(1, activation="sigmoid")])
one_hot_model.compile(optimizer="adam",
    loss="binary_crossentropy", metrics=["accuracy"])
one_hot_model.summary()

Model: "sequential_1"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ category_encoding               │ (None, 500, 1000)      │             0 │
│ (CategoryEncoding)              │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ bidirectional (Bidirectional)   │ (None, 48)             │       196,800 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 1)              │            49 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 196,849 (768.94 KB)

 Trainable params: 196,849 (768.94 KB)

 Non-trainable params: 0 (0.00 B)

Fit & evaluate

es = keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True,
    monitor="val_accuracy", verbose=2)

if not Path("models/one-hot-model.keras").exists():
    one_hot_model.fit(X_train_txt, y_train, epochs=1_000, callbacks=[es],
        validation_data=(X_val_txt, y_val), verbose=0);
    one_hot_model.save("models/one-hot-model.keras")
else:
    one_hot_model = keras.models.load_model("models/one-hot-model.keras")

one_hot_model.evaluate(X_train_txt, y_train, verbose=0, batch_size=1_000)

[0.6978467702865601, 0.6217318177223206]

one_hot_model.evaluate(X_val_txt, y_val, verbose=0, batch_size=1_000)

[0.6978550553321838, 0.6158273220062256]

Custom embeddings

from keras.layers import Embedding
embed_lstm = Sequential([Input(shape=(max_length,), dtype="int64"),
    Embedding(input_dim=max_tokens, output_dim=32, mask_zero=True),
    Bidirectional(LSTM(24)),
    Dense(1, activation="sigmoid")])
embed_lstm.compile("adam", "binary_crossentropy", metrics=["accuracy"])
embed_lstm.summary()

Model: "sequential_2"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ embedding (Embedding)           │ (None, 500, 32)        │        32,000 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ bidirectional_1 (Bidirectional) │ (None, 48)             │        10,944 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 1)              │            49 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 42,993 (167.94 KB)

 Trainable params: 42,993 (167.94 KB)

 Non-trainable params: 0 (0.00 B)

Fit & evaluate

es = keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True,
    monitor="val_accuracy", verbose=2)

if not Path("models/embed-lstm.keras").exists():
    embed_lstm.fit(X_train_txt, y_train, epochs=1_000, callbacks=[es],
        validation_data=(X_val_txt, y_val), verbose=0);
    embed_lstm.save("models/embed-lstm.keras")
else:
    embed_lstm = keras.models.load_model("models/embed-lstm.keras")

embed_lstm.evaluate(X_train_txt, y_train, verbose=0, batch_size=1_000)

[0.7265174984931946, 0.5706404447555542]

embed_lstm.evaluate(X_val_txt, y_val, verbose=0, batch_size=1_000)

[0.7588797807693481, 0.5446043014526367]

embed_lstm.evaluate(X_test_txt, y_test, verbose=0, batch_size=1_000)

[0.7490332126617432, 0.5482014417648315]

Entity Embedding

Entity embedding is a new way of handling categorical variables.

Revisit the French motor dataset

Code

from pathlib import Path
from sklearn.datasets import fetch_openml

if not Path("../data/freq_data.csv").exists():
    freq = fetch_openml(data_id=41214, as_frame=True).frame
    freq.to_csv("../data/freq_data.csv", index=False)
else:
    freq = pd.read_csv("../data/freq_data.csv")

freq

	IDpol	ClaimNb	Exposure	Area	VehPower	VehAge	DrivAge	BonusMalus	VehBrand	VehGas	Density	Region
0	1.0	1.0	0.10000	D	5.0	0.0	55.0	50.0	B12	Regular	1217.0	R82
1	3.0	1.0	0.77000	D	5.0	0.0	55.0	50.0	B12	Regular	1217.0	R82
...	...	...	...	...	...	...	...	...	...	...	...	...
678011	6114329.0	0.0	0.00274	B	4.0	0.0	60.0	50.0	B12	Regular	95.0	R26
678012	6114330.0	0.0	0.00274	B	7.0	6.0	29.0	54.0	B12	Diesel	65.0	R72

678013 rows × 12 columns

Data dictionary

Variable	Description	Preprocessing
`IDpol`	Policy number (unique identifier)	Dropped
`ClaimNb`	Number of claims on the given policy	Target
`Exposure`*	Total exposure in yearly units	Normalised
`Area`	Area code (ordinal)	Ordinal Encode
`VehPower`	Power of the car (ordinal encoded)	Normalised
`VehAge`	Age of the car in years	Normalised
`DrivAge`	Age of the (most common) driver in years	Normalised
`BonusMalus`	Bonus–malus level between 50 and 230 (with reference level 100)	Normalised
`VehBrand`*	Car brand (nominal)	One-hot
`VehGas`	Diesel or regular fuel car (binary)	One-hot
`Density`	Density of inhabitants per km² in the city of the living place of the driver	Normalised
`Region`*	Regions in France (prior to 2016)	One-hot

The model

Have \{ (\mathbf{x}_i, y_i) \}_{i=1, \dots, n} for \mathbf{x}_i \in \mathbb{R}^{47} and y_i \in \mathbb{N}_0.

Assume the distribution Y_i \sim \mathsf{Poisson}(\lambda(\mathbf{x}_i))

We have \mathbb{E} Y_i = \lambda(\mathbf{x}_i). The NN takes \mathbf{x}_i & predicts \mathbb{E} Y_i.

Note

For insurance, this is a bit weird. The exposures are different for each policy.

\lambda(\mathbf{x}_i) is the expected number of claims for the duration of policy i’s contract.

Normally, \text{Exposure}_i \not\in \mathbf{x}_i, and \lambda(\mathbf{x}_i) is the expected rate per year, then Y_i \sim \mathsf{Poisson}(\text{Exposure}_i \times \lambda(\mathbf{x}_i)).

What values do we see in the data?

Code

freq = freq.drop("IDpol", axis=1).head(25_000)

X_train, X_test, y_train, y_test = train_test_split(
  freq.drop("ClaimNb", axis=1), freq["ClaimNb"], random_state=36861)

# Reset each index to start at 0 again.
X_train_raw = X_train.reset_index(drop=True)
X_test_raw = X_test.reset_index(drop=True)

X_train_raw["Area"].value_counts()
X_train_raw["VehBrand"].value_counts()
X_train_raw["VehGas"].value_counts()
X_train_raw["Region"].value_counts()

Area
C    5514
D    4116
     ... 
B    2387
F     444
Name: count, Length: 6, dtype: int64

VehBrand
B1     4998
B2     4906
       ... 
B11     283
B14     140
Name: count, Length: 11, dtype: int64

VehGas
Regular    10658
Diesel      8092
Name: count, dtype: int64

Region
R24    6493
R82    2112
       ... 
R42      48
R43      26
Name: count, Length: 22, dtype: int64

How we preprocessed last time

As a reminder, last time we preprocessed the categorical data through one-hot encoding/dummy encoding.

from sklearn.compose import make_column_transformer

ct = make_column_transformer(
  (OneHotEncoder(sparse_output=False, drop="first"), ["VehGas", "VehBrand", "Region"]),
  (OrdinalEncoder(), ["Area"]),
  remainder=StandardScaler(),
  verbose_feature_names_out=False
)
X_train = ct.fit_transform(X_train_raw)

X_train_raw.head(3)

	Exposure	Area	VehPower	VehAge	DrivAge	BonusMalus	VehBrand	VehGas	Density	Region
0	1.00	A	7.0	8.0	50.0	52.0	B2	Diesel	13.0	R24
1	0.79	B	7.0	7.0	28.0	80.0	B12	Diesel	65.0	R21
2	1.00	C	6.0	13.0	30.0	50.0	B1	Regular	133.0	R53

X_train.head(3)

	VehGas_Regular	VehBrand_B12	VehBrand_B2	...	Area	Exposure	VehPower	VehAge	DrivAge	BonusMalus	Density
0	0.0	0.0	1.0	...	0.0	1.129272	0.366510	0.223226	0.374405	-0.524020	-0.394690
1	0.0	1.0	0.0	...	1.0	0.566087	0.366510	0.046100	-1.131699	1.122382	-0.381092
2	1.0	0.0	0.0	...	2.0	1.129272	-0.167408	1.108854	-0.994781	-0.641620	-0.363309

3 rows × 39 columns

Categorical Variables & Entity Embeddings

Region column

One-hot encoding

One hot encoding is a way to assign numerical values to nominal variables. One hot encoding is different from ordinal encoding in the way in which it transforms the data. Ordinal encoding assigns a numerical integer to each unique category of the data column and returns one integer column. In contrast, one hot encoding returns a binary vector for each unique category. As a result, what we get from one hot encoding is not a single column vector, but a matrix with number of columns equal to the number of unique categories in that nominal data column.

oh = OneHotEncoder(sparse_output=False)
X_train_oh = oh.fit_transform(X_train_raw[["Region"]])
X_test_oh = oh.transform(X_test_raw[["Region"]])
print(list(X_train_raw["Region"][:5]))
X_train_oh.head()

['R24', 'R21', 'R53', 'R24', 'R82']

	Region_R11	Region_R21	Region_R22	Region_R23	Region_R24	Region_R25	Region_R26	Region_R31	Region_R41	Region_R42	...	Region_R53	Region_R54	Region_R72	Region_R73	Region_R74	Region_R82	Region_R83	Region_R91	Region_R93	Region_R94
0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
1	0.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
3	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
4	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0

5 rows × 22 columns

Train on one-hot inputs

For the sake of explaining entity embeddings, we will train a neural network on just one categorical variable which is one-hot encoded.

1num_regions = len(oh.categories_[0])

random.seed(12)
2model = Sequential([
  Dense(2, input_dim=num_regions),
  Dense(1, activation="exponential")
])

3model.compile(optimizer="adam", loss="poisson")

es = EarlyStopping(verbose=True)
hist = model.fit(X_train_oh, y_train, epochs=100, verbose=0,
    validation_split=0.2, callbacks=[es])                       
hist.history["val_loss"][-1]

1: Computes the number of unique categories in the encoded column and store it in num_regions
2: Constructs the neural network. This time, it is a neural network with 1 hidden layer and 1 output layer. Dense(2, input_dim=num_regions) takes in an input matrix with columns = num_regions and transforms it down to an output with 2 neurons
3: Steps 3-6 are similar to what we saw during training with ordinal encoded variables

/Users/z3535837/Library/CloudStorage/Dropbox/Lecturing/ACTL3143/DeepLearningForActuaries/.venv/lib/python3.14/site-packages/keras/src/layers/core/dense.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
  super().__init__(activity_regularizer=activity_regularizer, **kwargs)

Epoch 10: early stopping

0.7677876949310303

Make a fake batch of data

Make a fake batch of data where one observation is from each region (essentially an identity matrix). We use this fake batch of data to see what the hidden layer’s activation looks like under each of the 22 categories.

X = np.eye(num_regions)
pd.DataFrame(X, columns=oh.categories_[0])

	R11	R21	R22	R23	R24	R25	R26	R31	R41	R42	...	R53	R54	R72	R73	R74	R82	R83	R91	R93	R94
0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
1	0.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
20	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0	0.0
21	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0

22 rows × 22 columns

model.layers[0](X)

tensor([[-0.0616, -0.2373],
        [ 0.0328,  0.0805],
        [-0.3284, -0.2670],
        [ 0.0154, -0.3086],
        [ 0.3865, -0.2786],
        [ 0.1066, -0.4211],
        [ 0.2431, -0.2671],
        [ 0.1179,  0.3248],
        [ 0.6360, -0.4440],
        [ 0.1381,  0.0310],
        [-0.2182, -0.3625],
        [ 0.7618, -0.2255],
        [ 0.5025, -0.0393],
        [ 0.7906, -0.4361],
        [-0.1988, -0.1829],
        [-0.2341, -0.1018],
        [ 0.7947,  0.1001],
        [ 0.5226, -0.1785],
        [-0.0475, -0.6383],
        [ 0.1016,  0.2053],
        [ 0.0257,  0.1546],
        [ 0.3212,  0.5033]], grad_fn=<AddBackward0>)

We see above the values of the 2 neurons in the hidden layer for each region as input.

The first layer

We can also extract the layer, get its wieghts and compute manually.

1layer = model.layers[0]
2W, b = layer.get_weights()
3X.shape, W.shape, b.shape

1: Extracts the layer
2: Gets the weights and biases and stores the weights in W and biases in b
3: Returns the shapes of the matrices

((22, 22), (22, 2), (2,))

X @ W + b

array([[-0.06, -0.24],
       [ 0.03,  0.08],
       [-0.33, -0.27],
       [ 0.02, -0.31],
       [ 0.39, -0.28],
       [ 0.11, -0.42],
       [ 0.24, -0.27],
       [ 0.12,  0.32],
       [ 0.64, -0.44],
       [ 0.14,  0.03],
       [-0.22, -0.36],
       [ 0.76, -0.23],
       [ 0.5 , -0.04],
       [ 0.79, -0.44],
       [-0.2 , -0.18],
       [-0.23, -0.1 ],
       [ 0.79,  0.1 ],
       [ 0.52, -0.18],
       [-0.05, -0.64],
       [ 0.1 ,  0.21],
       [ 0.03,  0.15],
       [ 0.32,  0.5 ]])

W + b

array([[-0.06, -0.24],
       [ 0.03,  0.08],
       [-0.33, -0.27],
       [ 0.02, -0.31],
       [ 0.39, -0.28],
       [ 0.11, -0.42],
       [ 0.24, -0.27],
       [ 0.12,  0.32],
       [ 0.64, -0.44],
       [ 0.14,  0.03],
       [-0.22, -0.36],
       [ 0.76, -0.23],
       [ 0.5 , -0.04],
       [ 0.79, -0.44],
       [-0.2 , -0.18],
       [-0.23, -0.1 ],
       [ 0.79,  0.1 ],
       [ 0.52, -0.18],
       [-0.05, -0.64],
       [ 0.1 ,  0.21],
       [ 0.03,  0.15],
       [ 0.32,  0.5 ]], dtype=float32)

The above codes manually compute and return the same answers as before. Remember that our \mathbf X is the identity matrix, so any matrix multiplication with \mathbf X becomes redundant (X @ W + b == W + b). This is extremely valuable as, if your categorical variable had, say, 10,000 different category possibilities, the matrix operation becomes intensive.

Just a look-up operation

We can consider this as just a look-up operation, where each row of the matrix W+b is some vector representation of each category. For example, if we want to know how region 11 is represented, we just look at the first row of the matrix W+b.

display(list(oh.categories_[0]))

['R11',
 'R21',
 'R22',
 'R23',
 'R24',
 'R25',
 'R26',
 'R31',
 'R41',
 'R42',
 'R43',
 'R52',
 'R53',
 'R54',
 'R72',
 'R73',
 'R74',
 'R82',
 'R83',
 'R91',
 'R93',
 'R94']

W + b

array([[-0.06, -0.24],
       [ 0.03,  0.08],
       [-0.33, -0.27],
       [ 0.02, -0.31],
       [ 0.39, -0.28],
       [ 0.11, -0.42],
       [ 0.24, -0.27],
       [ 0.12,  0.32],
       [ 0.64, -0.44],
       [ 0.14,  0.03],
       [-0.22, -0.36],
       [ 0.76, -0.23],
       [ 0.5 , -0.04],
       [ 0.79, -0.44],
       [-0.2 , -0.18],
       [-0.23, -0.1 ],
       [ 0.79,  0.1 ],
       [ 0.52, -0.18],
       [-0.05, -0.64],
       [ 0.1 ,  0.21],
       [ 0.03,  0.15],
       [ 0.32,  0.5 ]], dtype=float32)

This is what entity embedding does.

Turn the region into an index

To use the entity embedding functionality, we need to turn the categories into indices. We can use the oridinal encoder to do so.

oe = OrdinalEncoder()
X_train_reg = oe.fit_transform(X_train_raw[["Region"]])
X_test_reg = oe.transform(X_test_raw[["Region"]])

for i, reg in enumerate(oe.categories_[0][:3]):
  print(f"The Region value {reg} gets turned into {i}.")

The Region value R11 gets turned into 0.
The Region value R21 gets turned into 1.
The Region value R22 gets turned into 2.

Use an Embedding layer

Feed this new version of the data into the NN, using an embedding layer.

from keras.layers import Embedding
num_regions = X_train_raw["Region"].nunique()

random.seed(12)
model = Sequential([
  Embedding(input_dim=num_regions, output_dim=2),
  Dense(1, activation="exponential")
])

model.compile(optimizer="adam", loss="poisson")

es = EarlyStopping(verbose=True)
hist = model.fit(X_train_reg, y_train, epochs=100, verbose=0,
    validation_split=0.2, callbacks=[es])
hist.history["val_loss"][-1]

Epoch 4: early stopping

0.7686290144920349

model.layers

[<Embedding name=embedding_1, built=True>, <Dense name=dense_6, built=True>]

Embedding layer can learn the optimal representation for a category of a categorical variable, during training. In the above example, encoding the variable Region using ordinal encoding and passing it through an embedding layer learns the optimal representation for the region during training. Ordinal encoding followed with an embedding layer is a better alternative to one-hot encoding. It is computationally less expensive (compared to generating large matrices in one-hot encoding) particularly when the number of categories is high.

Keras’ Embedding Layer

model.layers[0].get_weights()[0]

array([[ 0.06, -0.08],
       [-0.01,  0.03],
       [-0.05, -0.01],
       [ 0.09, -0.12],
       [ 0.24, -0.22],
       [ 0.16, -0.19],
       [ 0.18, -0.18],
       [-0.05,  0.1 ],
       [ 0.38, -0.35],
       [ 0.03, -0.02],
       [ 0.02, -0.07],
       [ 0.36, -0.3 ],
       [ 0.2 , -0.15],
       [ 0.43, -0.38],
       [-0.03, -0.01],
       [-0.07,  0.03],
       [ 0.26, -0.16],
       [ 0.25, -0.21],
       [ 0.14, -0.21],
       [-0.02,  0.05],
       [-0.02,  0.04],
       [-0.03,  0.12]], dtype=float32)

X_train_raw["Region"].head(4)

0    R24
1    R21
2    R53
3    R24
Name: Region, dtype: str

X_sample = X_train_reg[:4].to_numpy()
X_sample

array([[ 4.],
       [ 1.],
       [12.],
       [ 4.]])

enc_tensor = model.layers[0](X_sample)
keras.ops.convert_to_numpy(enc_tensor).squeeze()

array([[ 0.24, -0.22],
       [-0.01,  0.03],
       [ 0.2 , -0.15],
       [ 0.24, -0.22]], dtype=float32)

The regions in get converted from text to index to vector representation in the embedding layer for each observation.

Entity embedding is especially useful when there is a very large number of categories (such as 1,000 or 10,000) and you want to reduce the number of columns. If your categorical variable has n categories, a simple rule of thumb is to use n^{1/4} for the dimension of the entity embedding.

In this case, the dimension chosen was 2. The intuition behind this is that regions can be represented easily over a 2-dimensional plane (by their geographical coordinates).

The learned embeddings

If we only have two-dimensional embeddings we can plot them.

points = model.layers[0].get_weights()[0]
plt.scatter(points[:,0], points[:,1])
for i in range(num_regions):
  plt.text(points[i,0]+0.01, points[i,1] , s=oe.categories_[0][i])

While it not always the case, entity embeddings can at times be meaningful instead of just being useful representations. The above figure shows how plotting the learned embeddings help reveal regions which might be similar (e.g. coastal areas, hilly areas etc.).

Entity embeddings

Each category is initially assigned a random vector, but they move during training. These movements follow some sort of logic, for example in this graph countries in Europe cluster together and countries in Asia cluster together.

Embeddings & other inputs

Often times, we deal with both categorical and numerical variables together. The following diagram shows a recommended way of inputting numerical and categorical data in to the neural network. Numerical variables are inherently numeric and do not require entity embedding. On the other hand, categorical variables must undergo entity embedding to convert to number format.

Illustration of a neural network with both continuous and categorical inputs.

We can’t do this with Sequential models…

Given we want numerical variables to do one thing and categorical variables to do another, they need to be preprocessed separately and independently. The outputs of these independent (non-sequential) preprocesses become inputs to the next hidden layer (now back to sequential).

Keras’ Functional API

Sequential models are easy to use and do not require many specifications, however, they cannot model complex neural network architectures. Keras Functional API approach on the other hand allows the users to build complex architectures.

Converting Sequential models

from keras.models import Model
from keras.layers import Input

random.seed(12)

model = Sequential([
  Dense(30, "leaky_relu"),
  Dense(1, "exponential")
])

model.compile(
  optimizer="adam",
  loss="poisson")

hist = model.fit(
  X_train_oh, y_train,
  epochs=1, verbose=0,
  validation_split=0.2)
hist.history["val_loss"][-1]

0.7696098685264587

random.seed(12)

inputs = Input(shape=(X_train_oh.shape[1],))
x = Dense(30, "leaky_relu")(inputs)
out = Dense(1, "exponential")(x)
model = Model(inputs, out)

model.compile(
  optimizer="adam",
  loss="poisson")

hist = model.fit(
  X_train_oh, y_train,
  epochs=1, verbose=0,
  validation_split=0.2)
hist.history["val_loss"][-1]

0.7716028690338135

See one-length tuples.

The above code shows how to construct the same neural network using sequential models and Keras functional API. Every sequential model can be converted into the functional style, but not every functional model can be converted to the sequential style.

In the functional API approach, we must specify the shape of the input layer, and explicitly define the inputs and outputs of a layer before specifying the model. model = Model(inputs, out) specifies the inputs and outputs of the model. This manner of specifying the inputs and outputs of the model allows the user to combine several inputs (inputs which are preprocessed in different ways) to finally build the model. One example would be combining entity embedded categorical variables, and scaled numerical variables.

Wide & Deep network

Add a skip connection from input to output layers.

from keras.layers \
    import Concatenate

inp = Input(shape=X_train.shape[1:])
hidden1 = Dense(30, "leaky_relu")(inp)
hidden2 = Dense(30, "leaky_relu")(hidden1)
concat = Concatenate()(
  [inp, hidden2])
output = Dense(1)(concat)
model = Model(
    inputs=[inp],
    outputs=[output])

The functional API method can unlock some new non-sequential NN architectures, such as the Wide & Deep network. One version of the inputs is processed through dense hidden layers, and the other version skips the processing. The two versions are then concatenated into a new layer which is used to create the output layer.

Naming the layers

For complex networks, it is often useful to give meaningful names to the layers.

input_ = Input(shape=X_train.shape[1:], name="input")
hidden1 = Dense(30, activation="leaky_relu", name="hidden1")(input_)
hidden2 = Dense(30, activation="leaky_relu", name="hidden2")(hidden1)
concat = Concatenate(name="combined")([input_, hidden2])
output = Dense(1, name="output")(concat)
model = Model(inputs=[input_], outputs=[output])

Inspecting a complex model

from keras.utils import plot_model

plot_model(model, show_layer_names=True)

model.summary(line_length=75)

Model: "functional_16"

┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
┃ Layer (type)        ┃ Output Shape      ┃   Param # ┃ Connected to      ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│ input (InputLayer)  │ (None, 39)        │         0 │ -                 │
├─────────────────────┼───────────────────┼───────────┼───────────────────┤
│ hidden1 (Dense)     │ (None, 30)        │     1,200 │ input[0][0]       │
├─────────────────────┼───────────────────┼───────────┼───────────────────┤
│ hidden2 (Dense)     │ (None, 30)        │       930 │ hidden1[0][0]     │
├─────────────────────┼───────────────────┼───────────┼───────────────────┤
│ combined            │ (None, 69)        │         0 │ input[0][0],      │
│ (Concatenate)       │                   │           │ hidden2[0][0]     │
├─────────────────────┼───────────────────┼───────────┼───────────────────┤
│ output (Dense)      │ (None, 1)         │        70 │ combined[0][0]    │
└─────────────────────┴───────────────────┴───────────┴───────────────────┘

 Total params: 2,200 (8.59 KB)

 Trainable params: 2,200 (8.59 KB)

 Non-trainable params: 0 (0.00 B)

The plot of the model becomes much easier to understand and interpret.

French Motor Dataset with Embeddings

The desired architecture

Preprocess all French motor inputs

Transform the categorical variables to integers:

1num_brands, num_regions = X_train_raw[["VehBrand", "Region"]].nunique()

ct = make_column_transformer(
2  (OrdinalEncoder(), ["VehBrand", "Region", "Area", "VehGas"]),
3  remainder=StandardScaler(),
4  verbose_feature_names_out=False
)
5X_train = ct.fit_transform(X_train_raw)
6X_test = ct.transform(X_test_raw)

1: Stores separately the number of unique categorical in the nominal variables, as would require these values later for entity embedding
2: Contructs columns transformer by first ordinally encoding all categorical variables (ordinal and nominal). Nominal variables are ordinal encoded here just as an intermediate step before this is the required input format for entity embedding layers
3: Applies standard scaling to all other numerical variables
4: Choose the simpler style of column names for the transformed dataframes
5: Fits the column transformer to the train set and transforms it
6: Transforms the test set using the column transformer fitted using the train set

Split the brand and region data apart from the rest:

X_train_brand = X_train["VehBrand"]
X_train_region = X_train["Region"]
X_train_rest = X_train.drop(["VehBrand", "Region"], axis=1)

X_test_brand = X_test["VehBrand"]
X_test_region = X_test["Region"]
X_test_rest = X_test.drop(["VehBrand", "Region"], axis=1)

Organise the inputs

Make a Keras Input for: vehicle brand, region, & others.

veh_brand = Input(shape=(1,), name="veh_brand")
region = Input(shape=(1,), name="region")
other_inputs = Input(shape=X_train_rest.shape[1:], name="other_inputs")

Create embeddings and join them with the other inputs.

1from keras.layers import Reshape

random.seed(1337)
2veh_brand_ee = Embedding(input_dim=num_brands, output_dim=2,
    name="veh_brand_ee")(veh_brand)                                
3veh_brand_ee = Reshape(target_shape=(2,))(veh_brand_ee)

4region_ee = Embedding(input_dim=num_regions, output_dim=2,
    name="region_ee")(region)
5region_ee = Reshape(target_shape=(2,))(region_ee)

6x = Concatenate(name="combined")([veh_brand_ee, region_ee, other_inputs])

1: Imports Reshape class from keras.layers library
2: Constructs the embedding layer by specifying the input dimension (the number of unique categories) and output dimension (the number of dimensions we want the input to be summarised in to)
3: Reshapes the output to match the format required at the model building step
4: Constructs the embedding layer by specifying the input dimension (the number of unique categories) and output dimension
5: Reshapes the output to match the format required at the model building step
6: Combines the entity embedded matrices and other inputs together

Complete the model and fit it

Feed the combined embeddings & continuous inputs to some normal dense layers.

x = Dense(30, "relu", name="hidden")(x)
out = Dense(1, "exponential", name="out")(x)

1model = Model([veh_brand, region, other_inputs], out)
model.compile(optimizer="adam", loss="poisson")

2hist = model.fit((X_train_brand, X_train_region, X_train_rest),
    y_train, epochs=100, verbose=0,
    callbacks=[EarlyStopping(patience=5)], validation_split=0.2)
np.min(hist.history["val_loss"])

1: Model building stage requires all inputs to be passed in together
2: Passes in the three sets of data, since the format defined at the model building stage requires 3 data sets

np.float64(0.6853498220443726)

Plotting this model

plot_model(model, show_layer_names=True)

Why we need to reshape

plot_model(model, show_layer_names=True, show_shapes=True)

The plotted model shows how, for example, region starts off as a matrix with (None,1) shape. This indicates that region was a column matrix with some number of rows. Entity embedding the region variable resulted in a 3D array of shape ((None,1,2)) which is not the required format for concatenating. Therefore, we reshape it using the Reshape function. This results in column array of shape, (None,2) which is what we need for concatenating.

Scale By Exposure

Two different models

Have \{ (\mathbf{x}_i, y_i) \}_{i=1, \dots, n} for \mathbf{x}_i \in \mathbb{R}^{47} and y_i \in \mathbb{N}_0.

Model 1: Say Y_i \sim \mathsf{Poisson}(\lambda(\mathbf{x}_i)).

But, the exposures are different for each policy. \lambda(\mathbf{x}_i) is the expected number of claims for the duration of policy i’s contract.

Model 2: Say Y_i \sim \mathsf{Poisson}(\text{Exposure}_i \times \lambda(\mathbf{x}_i)).

Now, \text{Exposure}_i \not\in \mathbf{x}_i, and \lambda(\mathbf{x}_i) is the rate per year.

Just take continuous variables

For convenience, the following code only considers the numerical variables during this implementation.

1ct = make_column_transformer(
2  ("passthrough", ["Exposure"]),
3  ("drop", ["VehBrand", "Region", "Area", "VehGas"]),
4  remainder=StandardScaler(),
5  verbose_feature_names_out=False
)
6X_train = ct.fit_transform(X_train_raw)
7X_test = ct.transform(X_test_raw)

1: Starts defining the column transformer
2: Lets Exposure pass through the neural network as it is without peprocessing
3: Drops the categorical variables (for the ease of implementation)
4: Scales the remaining variables
5: Choose the simpler style of column names for the transformed dataframes
6: Fits and transforms the train set
7: Only transforms the test set

Split exposure apart from the rest:

X_train_exp = X_train["Exposure"]
X_test_exp = X_test["Exposure"]
X_train_rest = X_train.drop("Exposure", axis=1)
X_test_rest = X_test.drop("Exposure", axis=1)

Organise the inputs:

exposure = Input(shape=(1,), name="exposure")
other_inputs = Input(shape=X_train_rest.shape[1:], name="other_inputs")

Make & fit the model

Feed the continuous inputs to some normal dense layers.

random.seed(1337)
x = Dense(30, "relu", name="hidden1")(other_inputs)
x = Dense(30, "relu", name="hidden2")(x)
lambda_ = Dense(1, "exponential", name="lambda")(x)

out = lambda_ * exposure # In past, need keras.layers.Multiply()[lambda_, exposure]
model = Model([exposure, other_inputs], out)
model.compile(optimizer="adam", loss="poisson")

es = EarlyStopping(patience=10, restore_best_weights=True, verbose=1)
hist = model.fit((X_train_exp, X_train_rest),
    y_train, epochs=100, verbose=0,
    callbacks=[es], validation_split=0.2)
np.min(hist.history["val_loss"])

Epoch 27: early stopping
Restoring model weights from the end of the best epoch: 17.

np.float64(0.9118543863296509)

Plot the model

plot_model(model, show_layer_names=True)

Package Versions

from watermark import watermark
print(watermark(python=True, packages="keras,matplotlib,numpy,pandas,seaborn,scipy,torch"))

Python implementation: CPython
Python version       : 3.14.3
IPython version      : 9.13.0

keras     : 3.14.1
matplotlib: 3.10.9
numpy     : 2.4.4
pandas    : 3.0.2
seaborn   : 0.13.2
scipy     : 1.17.1
torch     : 2.11.0

Glossary

entity embeddings
Input layer
Keras functional API

Reshape layer
skip connection
wide & deep network

Word Embeddings

Overview

Word Vectors

Word Vectors II

Remember this diagram?

Word2Vec

Word2Vec training methods

The skip-gram network

Word Vector Arithmetic

Word Embeddings II

Pretrained word embeddings

Look up a word vector

Find nearby word vectors

What does ‘similarity’ mean?

Weng’s GoT Word2Vec

Combining word vectors

Recipe recommender

Analogies with word vectors

Testing more associations

Quickly get to bad associations

Bias in NLP models

The analogy function cheats a little bit

Car Crash NLP Part II

Predict injury severity

Using TF-IDF Vectorization

The TF-IDF vectors

Feed TF-IDF into an ANN

Fit & evaluate

Keep text as sequence of tokens

A sequence of integers

Feed LSTM a sequence of one-hots

Fit & evaluate

Custom embeddings

Fit & evaluate

Entity Embedding

Revisit the French motor dataset

Data dictionary

The model

What values do we see in the data?

How we preprocessed last time

Categorical Variables & Entity Embeddings

Region column

One-hot encoding

Train on one-hot inputs

Make a fake batch of data

The first layer

Just a look-up operation

Turn the region into an index

Use an Embedding layer

Keras’ Embedding Layer

The learned embeddings

Entity embeddings

Embeddings & other inputs

Keras’ Functional API

Converting Sequential models

Wide & Deep network

Naming the layers

Inspecting a complex model

French Motor Dataset with Embeddings

The desired architecture

Preprocess all French motor inputs

Organise the inputs

Complete the model and fit it

Plotting this model

Why we need to reshape

Scale By Exposure

Two different models

Just take continuous variables

Make & fit the model

Plot the model

Package Versions

Glossary

The `analogy` function cheats a little bit