Text Representation: From Counts to Embeddings

A model needs numbers. The history of NLP is largely the history of better answers to "what numbers?", and each step solved a specific limitation of the one before it.

Sparse representations

One-hot and bag of words

One-hot gives every word an orthogonal basis vector: 50,000 dimensions, all zeros but one. Every pair of distinct words has cosine similarity exactly zero, so cat and kitten are as unrelated as cat and bureaucracy.

Bag of words sums one-hot vectors over a document, producing counts. It discards word order entirely — "dog bites man" and "man bites dog" are identical.

TF-IDF

Weight each term by how often it appears in this document and how rare it is across the corpus:

The IDF term is the interesting half. A word appearing in every document has \(\mathrm{df}=N\), so the displayed scikit-learn smoothed IDF is one, not zero. TF-IDF downweights common terms relative to rare ones; explicit max_df or a stopword policy removes them. A rare term gets larger IDF but is not necessarily predictive of the task. The TfidfTransformer API specifies this convention and optional row normalization.

Practical settings that matter:

TfidfVectorizer(
    ngram_range=(1, 2),      # unigrams + bigrams captures "not good", "new york"
    min_df=5,                # drop terms in fewer than 5 documents — mostly typos
    max_df=0.7,              # drop terms in >70% of documents — corpus-specific stopwords
    sublinear_tf=True,       # 1 + log(tf): 10 occurrences is not 10x as relevant
    strip_accents="unicode",
)

sublinear_tf is the setting people miss. Raw term frequency assumes relevance grows linearly with count, which is wrong — a document mentioning "python" 50 times is not 50× more about Python than one mentioning it once. The log damping consistently helps.

TF-IDF plus a linear model remains a strong baseline for topical text classification: it trains in seconds, needs no GPU, is fully interpretable (inspect the coefficients), and on many datasets lands within a few points of a fine-tuned transformer. Always run it first.

BM25 is TF-IDF's better-engineered relative and the standard for retrieval. It adds term-frequency saturation (a parameter \(k_1\) bounds the contribution of repeated terms) and document-length normalisation (\(b\)), both of which TF-IDF handles differently. BM25 is widespread, not universal across production search.

Word embeddings

The distributional hypothesis

Words appearing in similar contexts have similar meanings. Every word embedding method is an operationalisation of this claim.

word2vec

Two architectures, both shallow:

• Skip-gram: given a centre word, predict its context words. Better for rare words and small corpora.
• CBOW: given context words, predict the centre word. Faster, better for frequent words.

The skip-gram objective:

That denominator sums over the entire vocabulary at every step — computationally impossible for \(V = 10^6\). The fix that made word2vec practical:

Negative sampling replaces the full softmax with a binary classification: is this (word, context) pair real, or drawn from noise?

This noise-discrimination objective is different from normalized full-softmax likelihood, not merely a faster evaluation of its denominator. A common setup uses \(k=5\)–20 negatives drawn from the unigram distribution raised to the 3/4 power — an empirical choice that samples rare words more often than their raw frequency would.

Subsampling frequent words discards a token with probability \(\max(0,1-\sqrt{t/f(w)})\) in this simplified subsampling rule, on the reasoning that the millionth occurrence of "the" carries almost no information.

The famous analogies

Genuinely remarkable, and genuinely over-sold. The standard evaluation excludes the three input words from the answer candidates — without that exclusion, the nearest vector to the result is frequently king itself. The analogy structure is real but weaker and more dataset-dependent than the headline suggests.

GloVe

Where word2vec is predictive, GloVe is explicitly a matrix factorisation of global co-occurrence statistics:

The sum is evaluated over nonzero co-occurrences, avoiding \(\log0\); \(X_{ij}\) is the co-occurrence count and \(f\) caps the influence of very frequent pairs. The motivating insight is that ratios of co-occurrence probabilities encode meaning: \(P(\text{solid}\mid\text{ice}) / P(\text{solid}\mid\text{steam})\) is large, while the same ratio for "water" is near 1.

In practice word2vec and GloVe perform comparably; the choice is not important.

fastText

Represent each word as a bag of character n-grams plus the word itself: where → <wh, whe, her, ere, re>, <where>.

Two consequences that matter:

1. Out-of-vocabulary words get vectors. An unseen word is the sum of its n-gram vectors, so misspellings, new words, and rare technical terms all work.
2. Morphology is captured for free. run, running, runner share n-grams and therefore share representation, which is a large advantage for morphologically rich languages (Finnish, Turkish, Arabic).

The shared, fatal limitation

One vector per word type, regardless of context. bank gets a single vector that must serve "river bank" and "investment bank". Polysemy is unrepresentable, and the vector ends up as a blend of all senses weighted by corpus frequency.

That limitation is what contextual embeddings exist to remove, and it is worth seeing as the direct motivation for everything that followed.

Contextual embeddings

Produce a different vector for each occurrence of a word, computed from its sentence.

Which layer? Different layers encode different things — lower layers carry surface and morphological information, middle layers carry syntax, upper layers carry semantics and task-specific structure. For feature extraction, a concatenation or average of the last four layers usually beats the final layer alone, because the final layer is specialised toward the pretraining objective.

The [CLS] trap

Do not use raw BERT [CLS] embeddings for semantic similarity. Out of the box they perform worse than averaged GloVe vectors on sentence-similarity benchmarks. The reason is that BERT's pretraining objective never asked for sentence vectors to be comparable by cosine distance, and the resulting embedding space is highly anisotropic — all vectors occupy a narrow cone, so cosine similarities are compressed into a small range near 1.

Sentence embeddings

Trained explicitly so that cosine similarity means semantic similarity.

SimCSE's unsupervised version is elegantly simple: encode the same sentence twice with dropout active, treat the two encodings as a positive pair, and use other sentences in the batch as negatives. Dropout is the entire augmentation, and it works remarkably well.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("BAAI/bge-base-en-v1.5")
emb = model.encode(sentences, normalize_embeddings=True, batch_size=64)
sim = emb @ emb.T                              # cosine, since vectors are normalised

Normalise before comparing. With L2-normalised vectors, the dot product is cosine similarity, which lets you use fast inner-product search.

Follow the exact checkpoint's input protocol. E5 commonly uses query: and passage: prefixes. BGE English v1.5 instead specifies a retrieval query instruction and does not apply that instruction to documents. These are not interchangeable family-wide templates. See the BGE v1.5 model card.

Choosing an embedding model

Evaluate on your own data. MTEB rankings are strongly influenced by the benchmark composition, and a model two places lower on the leaderboard may be substantially better on your domain. A few hundred labelled query–document pairs is enough to tell.

Vector search

Exact nearest-neighbour search is \(O(Nd)\) per query — fine for \(10^5\) vectors, too slow for \(10^8\).

Metadata filtering is the practical differentiator. "Find similar documents from this tenant, in this date range, with this permission level" is the real requirement, and pre-filtering versus post-filtering has very different performance characteristics. Systems that handle filtered vector search well are worth the dependency.

Other representations

Hybrid retrieval is the practical default. Dense embeddings capture semantic similarity and miss exact matches (product codes, rare names, specific numbers); BM25 does the reverse. Combining the two — usually with reciprocal rank fusion — can improve recall, but can also introduce irrelevant candidates or poor fusion weights. Measure the chosen relevance unit and filtered corpus.

Bias in embeddings

Embeddings absorb the statistical regularities of their training corpus, including the prejudicial ones. The word-embedding association test found the same associations as the human implicit association test: European-American names with pleasant words, male terms with career terms, female terms with family terms.

Debiasing by projecting out a "gender direction" was shown to be largely cosmetic — the association is recoverable from the remaining geometry, because it is distributed rather than localised. The honest position: measure the bias in your specific application, evaluate downstream disparity rather than the embedding geometry, and treat the embedding as a component whose failures must be handled at the system level.

Self-check

Runnable count weighting and exact retrieval oracle

With three documents and document frequency two, smoothed IDF is \(1+\log(4/3)\). A ubiquitous term has IDF one. The normalized dot-product oracle below gives the exact candidate ordering against which an approximate index should be evaluated; approximate-search recall concerns agreement with this neighbor set, not necessarily human relevance.

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.preprocessing import normalize
docs = ["common apple apple", "common apple berry", "common carrot"]
vectorizer = TfidfVectorizer(norm=None)
matrix = vectorizer.fit_transform(docs)
vocab = vectorizer.vocabulary_
assert np.isclose(vectorizer.idf_[vocab["common"]], 1)
assert np.isclose(vectorizer.idf_[vocab["apple"]], 1+np.log(4/3))
normalized = normalize(matrix)
query = normalize(vectorizer.transform(["apple"]))
scores = (query @ normalized.T).toarray()[0]
assert scores[0] > scores[1] > scores[2]
assert np.isclose(np.linalg.norm(normalized.toarray(), axis=1), 1).all()
# BM25 term contribution with a declared positive-IDF convention.
frequency = np.array([2., 1., 0.])
length = np.array([3., 3., 2.])
k1, b = 1.2, .75
idf = np.log(1+(3-2+.5)/(2+.5))
bm25 = idf*frequency*(k1+1)/(frequency+k1*(1-b+b*length/length.mean()))
assert bm25[0] > bm25[1] > bm25[2]
print("TF-IDF", scores, "BM25 apple contribution", bm25)

What is an SGNS gradient? For a positive dot product \(s=v^Tu\), negative-log sigmoid loss has \(\partial L/\partial v=(\sigma(s)-1)u\); a negative pair gives \(\sigma(s)u\). Input and context embeddings are separate parameter tables. For sentence pooling, divide the sum of valid token states by their attention-mask count, not padded length. Contrastive InfoNCE uses a positive similarity relative to competing negatives with temperature; false negatives can push semantically equivalent sentences apart. Index/model revision, normalization and distance metric must match at ingestion and querying. Report bytes, latency and recall together when comparing exact and approximate search.

1. Why does scikit-learn's default IDF not remove ubiquitous terms, and which settings do?
2. What problem does negative sampling solve in word2vec, and what does it replace?
3. Give two things fastText can do that word2vec cannot, and say why.
4. Why do raw BERT [CLS] embeddings underperform averaged GloVe on sentence similarity?
5. What is SimCSE's unsupervised augmentation?
6. When would you choose BM25 over a dense embedding model, and what beats both?
7. Why must query and document prefixes be used with E5-style models?

Where to go next

• Language Models — the models that produce contextual embeddings.
• RAG & Retrieval — putting embeddings to work.
• Text Classification — the classic consumer of these representations.