Pretrained Model Families

There are three architectural families, a handful of pretraining objectives, and a long list of checkpoints. Knowing which family fits which task — and why the field converged on decoder-only for generation — saves more time than any amount of prompt engineering.

The three families

Encoder-only

BERT

The original BERT recipe combined two objectives:

• Masked language modelling — mask 15% of tokens and predict them from bidirectional context. The 80/10/10 split (replace with [MASK] / a random token / unchanged) exists because [MASK] never appears at fine-tuning time, so training on it exclusively creates a train–inference mismatch.
• Next sentence prediction — do these two segments follow each other? RoBERTa found NSP unnecessary in its controlled training comparisons. This does not prove every sentence-level objective hurts: ALBERT uses sentence-order prediction. See the RoBERTa study.

Bidirectional context is BERT's defining property and the reason it dominated classification and tagging: predicting a masked word uses both sides, which is strictly more information than a causal model has.

The successors

ELECTRA's insight is worth internalising: MLM computes a loss on only 15% of positions, wasting 85% of each forward pass. Replaced-token detection gives every position a binary training signal, which is why ELECTRA-style pretraining reaches BERT quality with a fraction of the compute.

Why encoders still matter

With representative labels, a small fine-tuned encoder can outperform a prompted large decoder on a narrow task with lower serving cost. Neither the quality ranking nor a fixed cost ratio is guaranteed. Bidirectional encoders are strong retrieval and reranking candidates; decoder-derived models can serve these tasks too. Compare matched evaluation data, hardware, and latency targets.

Use an encoder for: classification, NER and token tagging, extractive QA, sentence embeddings, and cross-encoder reranking.

Decoder-only

Trained on next-token prediction with causal attention. The family that won.

The modern decoder recipe

Common choices in dense decoders, not a specification shared by every model:

Mixture of experts is the other major structural choice: replace the FFN with \(N\) experts and route each token to \(k\) of them (typically \(k = 1\) or 2). Total parameters grow without evaluating every expert per token. Mixtral 8x7B is a specific roughly 47B-total/13B-active example, not eight independent 7B models. Shared layers, routing, and communication still cost compute. Experts need storage somewhere; distributed placement or offloading reduces per-device residency at additional communication or transfer cost.

Why decoder-only won

1. Universal objective. Next-token prediction applies to any text with no annotation.
2. Full training signal. Every position contributes a loss in one forward pass; MLM uses 15%.
3. Simplicity. No cross-attention, one attention pattern, one stack.
4. Scaling behaviour. Clean, predictable power laws.
5. In-context learning. Emerges from the objective, and turns one model into many task-specific ones without training.

Encoder–decoder

T5

Cast every task as text-to-text. Translation, classification, regression, summarisation — all become "input text → output text", with a task prefix. Pretrained with span corruption: mask contiguous spans and generate them, which is closer to the generation task than BERT's single-token masking.

The unified framing was influential well beyond T5, and the "instruction as prefix" idea is a direct ancestor of instruction tuning.

The family

Encoder–decoders remain the right choice where input and output are genuinely different objects and the input must be encoded once and attended to many times — translation, speech recognition, and summarisation of a fixed source.

Multilingual models

The curse of multilinguality: at fixed capacity, adding languages helps low-resource ones and hurts high-resource ones. Mitigations are more parameters, language-specific adapters, MoE routing, and temperature-sampling the language mix so low-resource languages are over-sampled relative to their corpus share.

Cross-lingual zero-shot transfer is the practical payoff: fine-tune XLM-R on English NER and it works reasonably on 50 other languages without any labelled data in them. This is the single most useful property of multilingual encoders.

Domain-specific models

Domain pretraining can help when vocabulary and distribution differ. PubMedBERT reported benefits on biomedical benchmarks; vocabulary, corpus, and training recipe all contribute. A fragmented medical term is not inherently meaningless to a subword model, and domain pretraining is not universally superior.

Check the tokenizer's fertility (tokens per word) on your domain text. If your key terms cost 6 tokens each, measure truncation, latency, and downstream quality before paying for vocabulary changes and retraining embeddings.

Choosing a checkpoint

Practical selection criteria

Base and instruct models behave completely differently and confusing them is a common early mistake. A base model given "What is the capital of France?" may continue with more questions, because that is what its training distribution contains.

Benchmark contamination is pervasive. Assume public test sets are in the training data of any model trained on the web. A private evaluation set drawn from your own distribution is worth more than any leaderboard position.

Sizes and what they can do

Task-specific fine-tuning of a small model frequently beats a large model prompted zero-shot, at a fraction of the cost. The large model's advantage is generality, not per-task quality — which is why the LLM-labels-then-distil pipeline is so effective.

Self-check

1. Why did RoBERTa drop next sentence prediction?
2. What does ELECTRA's objective change, and why is it more sample-efficient?
3. Give five common decoder design choices, their tradeoffs, and a counterexample to treating a family name as a fixed architecture.
4. Why did decoder-only win over encoder–decoder for general use?
5. When is a 100M encoder the right answer over a 70B LLM?
6. What is the curse of multilinguality and what mitigates it?
7. How would you decide whether a domain-specific model is worth using?

Worked checkpoint and objective checks

A 7B-parameter model has a weight-only lower bound of 14 GB at two bytes per parameter or 3.5 GB at four bits. Quantisation scales, unquantised layers, KV cache, activations, and runtime workspaces are additional. A model fitting RAM does not imply meeting a latency SLO. Count bytes before assigning hardware.

This offline experiment constructs random tiny models, not useful pretrained checkpoints. It checks the actual objective contracts: BERT predicts masked positions without shifting labels; a causal LM shifts labels internally.

import os
os.environ["USE_TF"] = "0"  # select the PyTorch backend before importing Transformers
import torch
from transformers import BertConfig, BertForMaskedLM, GPT2Config, GPT2LMHeadModel

torch.set_num_threads(1)
torch.manual_seed(7)
ids = torch.tensor([[1, 5, 9, 2]])
bert = BertForMaskedLM(BertConfig(vocab_size=16, hidden_size=8,
    num_hidden_layers=1, num_attention_heads=2, intermediate_size=16,
    hidden_dropout_prob=0, attention_probs_dropout_prob=0))
masked = ids.clone()
masked[0, 2] = 3
labels = torch.full_like(ids, -100)
labels[0, 2] = ids[0, 2]
out = bert(masked, labels=labels)
manual = torch.nn.functional.cross_entropy(out.logits[:, 2], ids[:, 2])
torch.testing.assert_close(out.loss, manual)
gpt = GPT2LMHeadModel(GPT2Config(vocab_size=16, n_embd=8, n_layer=1,
    n_head=2, n_positions=8, resid_pdrop=0, embd_pdrop=0, attn_pdrop=0))
causal = gpt(ids, labels=ids)
manual = torch.nn.functional.cross_entropy(
    causal.logits[:, :-1].reshape(-1, 16), ids[:, 1:].reshape(-1))
torch.testing.assert_close(causal.loss, manual)
assert out.logits.shape == causal.logits.shape == (1, 4, 16)
print("MLM position selection and causal label shift verified")

Answers. NSP removal is evidence about a training setup, not all auxiliary losses. ELECTRA supplies replaced/original supervision at more positions, but MLM's unlabelled positions still provide context and gradients. T5 instead encodes corrupted text and generates removed spans with sentinel boundaries. A small encoder is preferable only after its quality and serving budget pass your test. Evaluate domain adaptation against the original checkpoint using the same split and seeds; compare token fertility as a diagnostic, not the objective.

Where to go next

• Language Models — objectives, perplexity, scaling.
• LLM Prompting & Alignment — turning a base model into an assistant.
• Transformers Deep Dive — the architecture in detail.