Tokenization & Embeddings

The playlist opens video 72 with a question worth sitting with: what is the most important requirement for building any NLP application?

The answer is not the model. It is how do you turn words into numbers. Every NLP input — a review, a sentence, a document — is a set of words, and computers process numbers. This step is called vectorization, and the history of NLP is largely the history of getting better at it.

Take two toy sentences: "mat cat mat" and "cat rat rat".

One-hot encoding

Build a vocabulary of unique words — {mat, cat, rat} — and give each word a vector that is 1 in its own slot and 0 everywhere else.

mat -> [1, 0, 0]
cat -> [0, 1, 0]
rat -> [0, 0, 1]

Two fatal problems: the vector length equals vocabulary size (so ~50,000 dimensions for real text, almost all zeros), and every pair of words is equidistant. cat is exactly as similar to dog as it is to bulldozer. The representation encodes identity and nothing else.

Bag of words

Count occurrences instead of just presence. "mat cat mat" → [2, 1, 0]. Better — it carries frequency — but word order is gone entirely, and the similarity problem is untouched.

TF-IDF

Weight counts by how informative a word is across the corpus. Rare words score higher, common words lower. Still no notion of meaning.

Word embeddings — the real jump

Train a neural network on a large corpus (say, all of Wikipedia). The network learns the contexts each word appears in, and represents each word as a dense n-dimensional vector — 64, 256, 512, whatever you pick.

The payoff is semantic meaning is captured. Similar words get similar vectors. king and queen land near each other; cricketer lands far away.

The playlist's intuition for what the dimensions mean is worth keeping, with a caveat attached:

The caveat matters: the network never tells you what a dimension means. These labels are a teaching device. Real embedding dimensions are entangled and individually uninterpretable. What is true is the relational structure — similar words end up close in the space.

Before embedding anything you must decide what a "token" is. Three options:

Word-level tokenization — what the playlist uses throughout for clarity — has a hard failure: any word not in your vocabulary becomes <UNK> and its meaning is destroyed. Every typo, every proper noun, every new word.

Subword tokenization solves this. Frequent words stay whole; rare words split into known pieces.

Byte Pair Encoding (BPE)

BPE repeatedly merges the most frequent adjacent pair of initial symbols. Those symbols may be characters or bytes. Byte-level BPE starts with a complete byte alphabet; character BPE without fallback can still encounter OOV.

Training:

1. Start: vocabulary = all 256 individual bytes.
2. Count every adjacent pair in the corpus.
3. Merge the most frequent pair into one new token; add it to the vocabulary.
4. Repeat until you reach the target vocabulary size.

Illustrative merge vocabulary on low low low lower lowest (not a complete frequency-ordered training run: after low, low _ would beat e r):

start        l o w _ l o w _ l o w _ l o w e r _ l o w e s t
merge "l o"  -> "lo"       (most frequent pair)
             lo w _ lo w _ lo w _ lo w e r _ lo w e s t
merge "lo w" -> "low"
             low _ low _ low _ low e r _ low e s t
merge "e r"  -> "er"
             low _ low _ low _ low er _ low e s t

Final vocabulary contains low, er, e, s, t. Now lower tokenizes as low + er — two known pieces, never <UNK>. And slower, never seen in training, becomes s + low + er. Graceful.

With a complete byte alphabet or byte fallback, ordinary input text can fall back to bytes instead of <UNK>. This is not guaranteed by BPE alone; special- token handling and normalization are separate parts of the tokenizer contract.

Variants you will meet

Vocabulary size is an architectural decision

Raschka repeatedly flags this. Gemma's distinguishing feature is "the rather large vocabulary size (to support multiple languages better)." Mistral Small 3.1 gets lower inference latency partly from "their custom tokenizer." Bigger vocabulary means shorter sequences (cheaper attention, which is quadratic) but a larger embedding table and a larger output projection.

There is a reporting wrinkle worth knowing about: Raschka notes that Google "often exclude[s] embedding parameters to make the model appear smaller, except in cases... where it is convenient to include them to make the model appear larger." When you compare parameter counts across labs, check what is being counted.

An embedding layer is a lookup table, nothing more:

import torch
import torch.nn as nn

V       = 50_000    # vocabulary size
d_model = 512       # embedding dimension

embed = nn.Embedding(V, d_model)   # a (V, d_model) learned matrix

token_ids = torch.tensor([[10001, 1634, 92]])   # (B=1, T=3)
x = embed(token_ids)                            # (1, 3, 512)

nn.Embedding is equivalent to one-hot-encoding the IDs and multiplying by the table — but implemented as an index, which is why it is fast. The table is learned: those weights are trained with everything else.

Two details that matter in practice:

• Weight tying. Many models share the embedding matrix with the final output projection ((d_model, V)). Saves V × d_model parameters — ~26M at V=50k, d=512 — and often helps quality.
• Scaling. The original paper multiplies embeddings by sqrt(d_model) before adding positional encodings, so the two have comparable magnitude.

This is the hinge of the whole course.

At a fixed checkpoint, the input embedding for a token ID is context-independent. Its parameters can still change every training step. bank has one lookup vector at that checkpoint, and that same vector is used in:

• "I deposited money at the bank"
• "We sat on the river bank"

Same numbers. Different meanings. The playlist's phrasing is precise and worth adopting: an embedding does not capture the meaning of a word, it captures the average meaning of that word across the training corpus.

The Apple demonstration

Suppose we train two-dimensional embeddings with axes taste and technology, on a corpus containing the word apple:

If 9,000 of 10,000 sentences use apple as fruit and 1,000 as company, the learned vector sits at high-taste, low-technology. Flip the corpus ratio and the vector flips — the representation is a property of your data distribution, not of the word.

Now translate: "I bought an Apple phone while I was eating an orange."

The static embedding for apple says "fruit" — it was trained on a fruit-dominant corpus. It is wrong here. What we need is an embedding that reads the sentence, notices phone, raises the technology component, lowers the fruit component — and is not fooled by orange sitting nearby.

Static vs contextual, stated precisely

Self-attention is a mechanism that takes static embeddings as input and produces contextual embeddings as output. That sentence is the entire content of module 03, and everything after it is a variation on how to compute it efficiently.

Same count in, same count out, same dimension — but each output has absorbed information from the whole sequence.

A tokenizer is a pipeline: normalization, pre-tokenization, vocabulary-model segmentation, special-token post-processing, and decoding. SentencePiece supports both BPE and Unigram; it is not another name for Unigram. Normalization may lose case or Unicode distinctions, so a round-trip test must respect its documented normalization contract. Byte coverage does not make normalization reversible.

Checkpoint compatibility includes exact ID ordering, merge/model file, special-token IDs and chat template. Equal vocabulary size is insufficient: row 42 can represent a different token. Test leading spaces, repeated whitespace, combining characters, UTF-8 bytes, empty input, BOS/EOS/PAD insertion and literal special-token-looking text. Untrusted text must not accidentally inject privileged control tokens through a tokenizer's special-token option.

For token IDs i_t, lookup is X_t=E[i_t,:]. If upstream row gradient is g_t, then dL/dE[j,:]=sum_(t:i_t=j) g_t. Repeated IDs accumulate. A tied output head also uses E in logits=H E.T; its generally dense output gradient adds to this lookup gradient. Tying saves a table, not all gradient computation.

import torch
torch.manual_seed(2)
embedding = torch.nn.Embedding(5, 3)
ids = torch.tensor([1, 1, 4])
embedding(ids).sum().backward()
torch.testing.assert_close(embedding.weight.grad[1], torch.full((3,), 2.0))
torch.testing.assert_close(embedding.weight.grad[4], torch.ones(3))
assert embedding.weight.grad[0].count_nonzero() == 0
text = " leading space; café; e\u0301; \U0001f642"
assert bytes(text.encode("utf-8")).decode("utf-8") == text
print("Repeated token gradients accumulate; UTF-8 bytes round-trip.")

The byte check does not prove every tokenizer is lossless. Exercise solution: for vocabulary 50,000 and width 1024, tying saves 51,200,000 parameters, or 102,400,000 raw bytes at two bytes per weight. Optimizer-state savings depend on the training recipe. Primary details: SentencePiece and the tokenizer pipeline.

The earlier low illustration explains a merge vocabulary, not the complete optimization history. Now run a complete training process whose decisions can be reconstructed from its output. Download train_bpe.py. Its inspectable trainer needs only Python's standard library; the optional library counterpart uses Hugging Face Tokenizers, tested with tokenizers==0.22.1.

python content/courses/transformers/code/train_bpe.py --output-dir /tmp/course-bpe
python content/courses/transformers/code/train_bpe.py --library --output-dir /tmp/course-bpe

The first command writes tokenizer.json and merge-trace.json. The second also writes tokenizer-hf.json, a different format loadable by the Tokenizers library. The educational JSON is intentionally not presented as a compatible Hugging Face artifact. Neither tokenizer is compatible with an existing model's embedding rows without retraining or an explicit vocabulary migration.

Specify the experiment before counting

The corpus is seven independent strings: ["low", "low", "low", "lower", "lowest", "newer", "newest"]. There is no lowercasing, Unicode normalization, whitespace stripping, appended end-of-word marker, or cross-example concatenation. All 256 byte values exist before training, including bytes absent from the corpus. Initial token ID b represents exactly byte value b; merge rank r creates ID 256 + r.

This definition matters. Counting low low as one document permits pairs involving its space. Counting two separate low examples does not invent a boundary pair. Appending an end-of-word marker changes both frequencies and the meaning of the learned tokens. Tokenizer variants are not interchangeable just because all of them say "BPE."

At each round, count adjacent token-ID pairs across all examples. Repeated examples contribute repeatedly. Choose the largest count, break ties by lexicographically smallest (left_id, right_id), replace nonoverlapping matches left to right, and recount the changed corpus. Stop after the requested number of merges, when no pair remains, or when the winning count falls below two. The explicit tie rule makes this run independent of input-example order.

The complete run is:

The initial corpus has 31 bytes. Although twelve merges were allowed, only seven pass the frequency threshold. The JSON trace includes every candidate pair and its count, not just each winner. That lets a test independently recompute the argmax and the tie rule. Counts belong to the current segmented corpus, not the original characters: after creating lo, the old l,o windows are gone.

Overlap, rank order and unseen text

Consider aaaa. The adjacent pair (a,a) has frequency three because windows at offsets 0, 1 and 2 overlap. A left-to-right replacement produces two tokens, aa,aa, reducing token count by two, not three. Therefore "winning frequency" and "number of replacements" are different quantities. The lab explicitly tests this edge case rather than assuming token reduction equals pair count.

Encoding new text does not refit frequencies. Start from its bytes and apply the stored merges in training-rank order. For this corpus:

lower   -> [257, 258]       -> [b'low', b'er']
slower  -> [115, 257, 258]  -> [b's', b'low', b'er']

Greedy longest-string matching is not the definition of BPE encoding. Retaining the merge ranks is part of the model. Saving only the final token strings loses that information. The loader also rejects operands that refer to token IDs not yet defined by earlier merges.

UTF-8 encodes some characters with multiple bytes. One individual token can therefore contain only part of a character. Decode by concatenating all token bytes first, then decoding UTF-8. Independently decoding each token can fail even when the whole sequence is valid. Arbitrary generated byte-token sequences can themselves be invalid UTF-8; this educational decoder raises instead of silently replacing invalid sequences. Round-trip guarantees apply to text encoded by this tokenizer, not to every possible list of token IDs.

The tests cover empty input, tabs, line breaks, leading spaces, accented and combining characters, multibyte symbols, and deterministic randomized strings. <BOS> is ordinary literal text in this lab, not an executable special token. A real chat pipeline needs a separate, explicit special-token policy and must not let user text inject trusted role delimiters.

Transfer the contract to a production library

train_library_tokenizer() uses the library's BPE model, BpeTrainer, a full byte alphabet, and matched byte-level pretokenizer/decoder. There is no custom normalizer and add_prefix_space=False; round-trip tests protect whitespace. The library pretokenizer introduces boundaries that the raw-byte experiment does not, so the two trainers are not expected to produce identical IDs or merge traces. Compare documented contracts, not accidental token numbers.

The library pipeline separates normalization, pretokenization, model encoding, and postprocessing. Use its native JSON to retain those settings together; a vocabulary file alone is insufficient. See the Tokenizers pipeline documentation and BPE trainer API.

The inspectable implementation rescans the corpus every round, roughly O(number_of_merges * corpus_tokens) time, and stores full frequency snapshots. This is deliberate for a seven-string audit, not a recommendation for training a production vocabulary. Use the optimized library for large corpora; keep small fixtures to verify boundaries, normalization and artifact reloads.

Experiments and interpretation

1. Duplicate newest repeatedly and predict which early winner changes. Retrain and locate the first differing trace row. A vocabulary is a learned property of the sampling distribution, not just its distinct words.
2. Train with a one-merge budget on aaaa. Explain the three counted windows versus two replacements before reading the assertion.
3. Normalize composed and decomposed accented text before training. Which original strings can still be reconstructed exactly? Normalization can make semantic matching easier while intentionally abandoning byte-level identity.
4. Compare corpus token totals at several vocabulary sizes on held-out text. Include embedding/output parameter cost and softmax work, not only sequence compression. Do not choose a vocabulary solely on its training compression.
5. Save, reload, and compare IDs, not only decoded text. Two encodings can reconstruct the same text while selecting different embedding rows.

Continue to NLP text preprocessing for the broader pipeline and the translation lab for the separate BOS, EOS and PAD contracts used during model training.