NLP Fundamentals: Tokenization, Embeddings, and Transformers
How text becomes vectors, why subword units won, and what the transformer changed. Read the mechanism behind each layer of the stack.
Most introductions to NLP present the stack as a menu: tokenisation, embeddings, transformers, and a large language model at the end. Each layer is usually described by what it is. The more useful description is what each layer fixes about the layer below it, because the history of this field is a series of problems being solved in order.
Direct answer: Text has to become numbers, and each step solves a specific failure of the previous one. Subword tokenisation solves the open-vocabulary problem — a fixed word vocabulary cannot represent every word, and splitting rare words into smaller units can. Embeddings solve the discreteness problem — vectors can be compared by arithmetic rather than equality. The transformer solves the sequencing problem — earlier architectures processed tokens in order, so each one waited for the previous one. And the large-scale pretraining-plus-fine-tuning pipeline solves the labelling problem — it starts from raw text and needs labelled examples only at the end.

Layer one: tokenisation, and the open-vocabulary problem
Neural models operate on a fixed vocabulary of discrete tokens. Translation, however, is "an open-vocabulary problem": any input may contain a word the model has never seen.
Sennrich et al. describe the approach that became standard: making the model "capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units". The intuition behind it is worth stating because it generalises beyond translation: "various word classes are translatable via smaller units than words" — names by character copying or transliteration, compounds by compositional translation, and cognates and loanwords by phonological and morphological transformations. In other words, a rare word is not a mystery to be looked up but a structure that can be assembled from parts that are not rare.
Their method was byte pair encoding, and they discuss "the suitability of different word segmentation techniques, including simple character n-gram models and a segmentation based on the byte pair encoding compression algorithm". The compression framing is the memorable part: the vocabulary is built by repeatedly merging the most frequent adjacent pair, so the units are whatever the corpus treats as atomic.
Kudo and Richardson built the tooling that made this practical and, importantly, language-independent. Their SentencePiece is "a language-independent subword tokenizer and detokenizer designed for Neural-based text processing". The limitation it removes is stated precisely: "existing subword segmentation tools assume that the input is pre-tokenized into word sequences", whereas SentencePiece "can train subword models directly from raw sentences, which allows us to make a purely end-to-end and language independent system". They validated it on English-Japanese machine translation and "find that it is possible to achieve comparable accuracy to direct subword training from raw sentences".
That last point is the one to keep.
It is a modelling decision, and a language without whitespace or with rich morphology changes the result.
Layer two: embeddings, and the discreteness problem
A token id is a label, not a meaning. Two tokens never compare, and the distance between them carries no information. Embeddings replace the label with a vector, so that similarity is arithmetic.
The first generation of embeddings were static: one vector per word type, learned from co-occurrence. Their weakness was polysemy — the same vector for "bank" in "river bank" and "bank account", because the model had no way to know which sense was meant.
Contextual embeddings fix this by making the vector depend on the sentence. BERT, the model that made the approach standard, was designed to "pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers". The word "bidirectional" is the mechanism and the source of the gain: an earlier left-to-right model could only use context before the token, so the representation of a token was necessarily incomplete.
The other thing BERT changed was the shape of the downstream work. The pre-trained BERT model "can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications". The reported results were "new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7%".
A useful caution comes from the replication study. RoBERTa found that "BERT was significantly undertrained, and can match or exceed the performance of every model published after it", and that "hyperparameter choices have significant impact on the final results". The authors go on to "raise questions about the source of recently reported improvements". That is worth remembering whenever a leaderboard entry appears: part of what looks like an architectural advance may be a better-tuned earlier one.
Layer three: the transformer, and the sequencing problem
Before 2017, the dominant sequence models were recurrent or convolutional, "based on complex recurrent or convolutional neural networks in an encoder-decoder configuration", and — critically — they processed tokens sequentially. Every token waited for its predecessor. That is why they were hard to parallelise and why long contexts were expensive.
Vaswani et al. removed the recurrence: a network "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely". The benefit they measured was both quality and cost: 28.4 BLEU on WMT 2014 English-to-German, "improving over the existing best results, including ensembles by over 2 BLEU", and a single-model state-of-the-art 41.8 BLEU on English-to-French "after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature".
The important engineering consequence is not the benchmark. It is that removing the sequential dependency made training parallelisable, and parallelisability is what made scale affordable — and scale is what made the rest of this stack possible.
Layer four: making embeddings usable at scale
One more piece of engineering deserves attention because it connects this article to retrieval, RAG, and vector databases.
BERT-style encoders are not usable for semantic search directly. As Reimers and Gurevych explain, sentence-pair models "require that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT". The construction of BERT "makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering".
Their fix is architectural. Sentence-BERT uses "siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity", which "reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT".
That is a 47,000-fold reduction in compute for the same answer, and it is what makes the modern retrieval stack possible at all. Note what it costs: each sentence must be embedded independently, so the embedding can no longer depend on the query it will be compared with. The architecture trades expressiveness for throughput, and that trade is the right one when you are searching ten million candidates.
The layer you did not expect: the label
The final step is what made the whole thing usable. GPT-3 is described as "an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model", and its central finding is about labels: scaling "greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches". The contrast drawn in the abstract is that fine-tuning "still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples", whereas "humans can generally perform a new language task from only a few examples or from simple instructions".
That is the transition from a research model to a tool: a task becomes a prompt rather than a labelled dataset.
How to check a stack you did not build
- What is the vocabulary, and how were out-of-vocabulary words handled? If the answer is "there are none", the system will fail on names, identifiers, and morphology it was not shown.
- Are the embeddings contextual or static? If static, the system cannot distinguish senses, and no amount of prompt engineering will fix it.
- Is the encoder bi-encoder or cross-encoder? A cross-encoder gives better ranking and cannot serve retrieval at scale; a bi-encoder gives worse ranking and can. This is the single most consequential architectural choice in a retrieval system.
- How many labels does a new task need? If the answer is thousands, the pipeline is not leveraging pretraining the way it could.
- Was the baseline tuned as carefully as the new model? The RoBERTa replication result makes this a mandatory question, not a courtesy.
For what this stack can and cannot do, see The Limits of AI; for the retrieval layer it enables, see Vector Databases: Embeddings, ANN Search, and RAG Applications.
A worked example
The following is hypothetical and illustrative. A team is building semantic search over a year of support tickets. Before choosing a model, it runs three checks drawn from the stack's own history. First, it inspects tokenisation: tickets contain product codes and customer names, so it confirms the tokenizer splits unseen words into subword units, the open-vocabulary fix Sennrich et al. described, rather than mapping them to a single unknown token. Second, it checks whether embeddings are contextual. Because the word 'bank' appears in both a payment context and a river-name context, a static vector would collapse the senses; a bidirectional encoder keeps them apart, the point Devlin et al. made. Third, it separates retrieval from ranking: a bi-encoder embeds tickets independently for fast cosine search, while a cross-encoder reranks the shortlist against the full query, the trade Reimers and Gurevych identified. The bounded outcome is a search that finds the right ticket for a phrase it never saw verbatim, with the reranker lifting the correct match into the top three. The team accepts slower ranking on a shortlist in exchange for recall at scale.
A counterargument
The 'each layer fixes the one below' story is a clean retrospective that the sources themselves complicate. The article cites RoBERTa's finding that BERT was significantly undertrained and that hyperparameter choices have a large impact, which raises the question of how much of the BERT and transformer gain was architecture rather than tuning. It also concedes that SentencePiece is a language-dependent modelling decision and that SBERT trades expressiveness for throughput, both of which break the neat chain.
A critic would say the progression is a pedagogical arrangement of independent results, not a dependency the papers assert; tokenisation, embeddings, and attention were developed in parallel, and calling them a single fix-by-fix ladder overstates the coherence of the field.
References & Further Reading
- Ashish Vaswani et al., 2017. Attention Is All You Need — The transformer architecture that the NLP and LLM sections build on.
- Jacob Devlin et al., 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — Bidirectional encoder pretraining and the masked-language-model objective.
- Rico Sennrich, Barry Haddow, and Alexandra Birch, 2015. Neural Machine Translation of Rare Words with Subword Units — The subword vocabulary approach that fixed out-of-vocabulary problems.
- Taku Kudo and John Richardson, 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing — The tokenizer family used by T5, Llama, and most modern open models.
- Nils Reimers and Iryna Gurevych, 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — The bi-encoder recipe behind modern semantic search and RAG retrieval.