How transformer models work.

The problem a language model solves

A language model does one thing. Given a sequence of text, it produces a probability distribution over what comes next. Everything else, including answering a question, summarizing a document, or returning structured data, is that same operation applied repeatedly.

The difficulty is that the meaning of a word depends on the words around it, sometimes on words a long way away. In the sentence “The certificate the supplier submitted last March expired, so it must be renewed before the audit”, the word “it” refers to the certificate, eleven words earlier. Any model that cannot connect those two positions will answer questions about that sentence badly. Most of the history below is about how models learned to make that connection.

Token

A piece of text the model works with. Usually a word fragment rather than a whole word. Context limits and processing cost are counted in tokens.

Embedding

A list of numbers representing a piece of text. Texts with similar meaning end up close together, which is what makes search by meaning possible.

Parameter

One of the numbers learned during training. A model described as 70B has seventy billion of them.

Context window

How much text the model can consider in a single call. Everything the model uses for one answer has to fit inside it.

Before transformers

For decades the standard approach was the n-gram model. It predicted the next word from the handful of words directly before it, using counts collected from a corpus. It worked well enough for spelling correction and early speech recognition, and it failed at anything longer, because it had no way to know that “supplier” and “vendor” are related. Each word was a separate symbol with its own counts.

Word embeddings, popularised by word2vec in 2013, fixed that. Every word was given a vector learned from the company it keeps, so similar words landed near each other and a model could generalise across them. The limitation was that each word had exactly one vector. “Bank” had a single representation whether the sentence was about a river or a payment.

Recurrent neural networks read a sentence one token at a time and carried a hidden state forward, which in principle let information travel any distance. The LSTM, introduced in 1997, added gates that decided what to keep and what to discard, and it held on to information much longer than a plain recurrent network. Two problems remained. Information still had to pass through every intermediate step, so long-range links degraded. And because each step depended on the previous one, training could not be spread across the sequence, which put a hard ceiling on the amount of text a model could learn from.

In 2014 the sequence-to-sequence design paired two of these networks. One read the input and compressed it into a single fixed-length vector, and one generated the output from that vector. It made machine translation work far better than before, and it had an obvious weak point. Everything the model knew about a long input had to survive in one vector of a few hundred numbers.

Attention arrives

Later in 2014, Bahdanau and colleagues removed that bottleneck. Instead of compressing the input into one vector, they let the decoder look back at every position of the input and take a weighted average, with the weights computed fresh for each output word. Translation quality on long sentences improved immediately, and the weights turned out to be readable. When the model produced a French word, the weights concentrated on the English word it corresponded to.

For three years attention was an addition to recurrent models. In 2017 a group at Google published a paper whose title stated the finding plainly. Attention was all you needed. They removed the recurrence entirely and kept only attention and ordinary feed-forward layers.

Two things followed. Any position could now reach any other position in a single step rather than by passing information along a chain, so long-range dependencies stopped degrading with distance. And because no step waited for the previous one, the whole sequence could be processed at once during training. That second property is what made it possible to train on the volume of text that current models are trained on. The architecture is called the transformer, and its outline has not fundamentally changed since.

Tokens and embeddings

Before a model sees text, the text is cut into tokens by a fixed vocabulary of perhaps fifty to two hundred thousand entries. Common words are single tokens. Rarer and more technical material is split into fragments, so a standard reference such as “ISO 27001” may become several tokens, and a long compound word in Dutch or German may become five or six.

This is worth knowing for practical reasons. Context limits and processing costs are counted in tokens rather than words, and the same document costs more tokens in some languages than in others. It also explains a class of odd model behaviour, such as difficulty counting letters in a word the model never sees as letters.

Each token is then looked up in a table and replaced by a vector. At that point the vector says nothing about context. The layers that follow are what turn a context-free vector for “bank” into a representation that reflects whether this particular sentence is about a river or a payment.

Self-attention, step by step

Self-attention is the operation that lets each position gather information from the others. For every token, the model computes three vectors from its current representation using three sets of learned weights.

• A query, which represents what this token is looking for.
• A key, which represents what this token offers to others.
• A value, which is the information this token passes on when selected.

The model takes one token's query and compares it against the key of every token in the sequence, using a dot product. A large result means those two positions are relevant to each other. The scores are divided by a constant to keep them in a workable range, then passed through a softmax, which turns them into weights that add up to one. The output for that token is the weighted sum of every token's value vector.

Applied to the earlier sentence, when the model works on the token “it”, the query for “it” matches the key for “certificate” far more strongly than the keys for the other words, so the output for “it” is dominated by the value vector of “certificate”. The pronoun has been resolved. Nobody wrote a rule for that. The weights that produce the query and key vectors were learned from text, and this behaviour is what they settled into.

Multiple heads

One set of query, key, and value weights can only express one notion of relevance at a time. Real text needs several at once. Grammatical agreement, the subject of a clause, and a reference to something in an earlier paragraph are all different kinds of relationship.

A transformer therefore runs the attention operation many times in parallel, each with its own smaller set of weights. These are called heads. Their outputs are joined together and passed through one more learned projection. Researchers examining trained models have found heads that behave in distinguishable ways, some tracking the token directly before, some following syntactic links, and some carrying references over long distances. This division of labour is not designed, it emerges during training, and it is not always tidy.

Telling the model about order

Attention as described so far has a curious property. It has no idea what order the tokens are in. It computes a weighted average over a set, and a set has no sequence. “The supplier audits the customer” and “The customer audits the supplier” would look identical.

The original transformer solved this by adding a fixed pattern of sine and cosine values to each token's vector, different for every position. Later models learned a table of position vectors instead. Both work, and both share a weakness. A model trained with positions up to some limit has nothing sensible to say about position beyond it.

Rotary position embedding, from 2021, takes a different approach. It rotates the query and key vectors by an angle proportional to the position, so that when two of them are compared the result depends on the distance between the tokens rather than on where each one sits in absolute terms. This turned out to extend to longer inputs much more gracefully, and it is now used in most current open models.

The rest of the block

Attention moves information between positions. It does not do much processing of the information itself. That is the job of the second half of a transformer block, an ordinary feed-forward network applied to each position separately. In a conventional model most of the parameters live in these layers rather than in attention.

Two further pieces make deep stacks trainable. A residual connection adds each sublayer's input to its output, so information and gradients have a direct path through the network. Layer normalization keeps the scale of the numbers stable. Current models normalize before each sublayer rather than after, which trains more reliably at depth.

One block is attention followed by a feed-forward network. A model is that block repeated, from a dozen times in a small model to more than a hundred in a large one. Depth is what lets later layers build on relationships that earlier layers established.

Three families of model

The same block can be arranged in three ways, and the arrangement decides what the model is good for.

Encoder-only

Every token attends in both directions, so each position is informed by the whole input. The model produces a representation of text that already exists rather than new text. BERT, from 2018, is the well-known example. This is the shape used by embedding models and by the rerankers described below.

Decoder-only

A mask prevents each token from attending to anything later than itself, and the model is trained to predict the next token. Because it cannot see ahead, it can generate. Almost every model marketed as a large language model has this shape, and it is what Docusense uses for extraction, drafting, summarizing, and policy comparison.

Encoder-decoder

An encoder reads the input and a decoder writes the output, with an extra attention step that lets the decoder look at the encoder's representation. This was the original 2017 design and it remains a good fit for translation and similar transformation tasks.

How a model is trained

Training happens in stages, and knowing which stage produced which behaviour explains a lot about how these systems act.

1. Pretraining

   The model predicts the next token across a very large body of text. No labelling is needed, because the text is its own answer key. This stage is where language ability and general knowledge come from, and it accounts for nearly all of the computation.

2. Instruction tuning

   The model is then trained on examples of instructions paired with good responses. A pretrained model continues text. An instruction-tuned model answers the request.

3. Preference tuning

   Finally the model is adjusted toward the responses people preferred when shown pairs to compare. Reinforcement learning from human feedback did this first, and simpler methods such as direct preference optimization now achieve similar results with less machinery.

One consequence matters for any serious application. Instruction tuning makes a model cooperative, and a cooperative model will answer a question it has no support for just as readily as one it does. Getting a reliable result depends far more on what you place in front of the model and what you allow it to return than on which model you pick.

What changed since 2017

The architecture from the 2017 paper is still recognisable in a model released this year. What changed is scale and a long list of engineering refinements, several of which decide what is practical to deploy.

Scale came first. Work published in 2020 established that performance improves predictably as parameters, data, and computation grow together, which turned model development into an engineering exercise with a known return. A 2022 correction showed that the models of the period were substantially undertrained for their size, and the field moved toward training smaller models on much more data. That change is why a model you can run yourself today outperforms a far larger model from a few years ago.

Longer context

The 2017 model handled a few hundred tokens. Current models handle hundreds of thousands. The cost of attention grows with the square of the sequence length, so this needed new work rather than bigger machines. FlashAttention reorganized the computation so the full attention matrix never has to be written to memory. Sliding-window and other sparse patterns limit which positions each token looks at. Grouped-query attention shrinks the cache that dominates memory while text is being generated.

Better handling of position

Rotary position embedding, published in 2021, rotates the query and key vectors by an angle that depends on where the token sits. The attention score then depends on the distance between two tokens rather than their absolute positions, which extends to longer inputs more gracefully than a fixed table of learned positions. Most current open models use it.

Mixture of experts

The feed-forward layer is replaced by many smaller ones, and a router sends each token to a couple of them. A model can hold far more parameters without spending proportionally more computation on each token, which is how several recent models offer large capacity at a workable cost.

Output you can parse

Constrained decoding blocks the tokens that would break a required JSON schema while the model generates. The result is valid structured output rather than prose that has to be salvaged with a regular expression. Docusense relies on this for every extraction and drafting step.

Models that work before answering

Some models are trained to produce a long run of intermediate reasoning tokens before the final answer. This costs latency and tokens, and it helps most on problems with several dependent steps.

Smaller models and quantization

Storing weights at 8 or 4 bits instead of 16 cuts memory use substantially with a modest quality cost, and small models have improved considerably. Together these make it realistic to run capable models on hardware inside a customer network, which is what an air-gapped deployment requires.

What transformer models do not do

A transformer produces a probability distribution over the next token. It has no representation of truth, and no internal signal that separates a well-supported statement from a plausible invention. Text that is fluent and correct and text that is fluent and wrong are produced by the same process and look the same on the way out.

What the model absorbed during pretraining is a statistical average of an enormous amount of text, not a set of records it can look up. It cannot tell you where something it says came from, because there is no stored source to point at. It also knows nothing about your organization. Your policies, your certificates, your product specifications, and what you told a particular customer two years ago were not in its training data and cannot be recalled from it.

Between calls it remembers nothing. Every fact a model uses for a given answer has to be inside the prompt for that call.

These limits set the design of any system built on this technology and are the reason Docusense works the way it does. Find the actual documents first. Put them in front of the model. Require the output to cite what it used. Show a person the sources next to the suggestion, and let that person decide.

Where Docusense uses them

An instance runs four kinds of model, each configured separately so an administrator can choose the right one for each job and change any of them without touching the others.

• A document processing model reads uploaded files where the source needs interpretation rather than plain text extraction, for example scanned pages and tables.
• A completion model extracts questions and requirements, writes summaries, drafts suggested answers from retrieved sources, and compares items against policy text.
• An embedding model turns questions, answers, and document sections into vectors so that related material can be found by meaning.
• A reranking model reads a question together with each shortlisted candidate and re-scores them before anything is used.

In a self-hosted deployment all four run inside your environment, and the core deployment can operate without an internet connection. Theretrieval page explains how these models are combined with search so that an answer stays tied to a source a reviewer can open.