Timeline of transformers

This is a timeline of transformers, documenting the history of the transformer architecture in machine learning, from its predecessors in the early 1990s to the present day. It covers the development of the architecture itself, the models built on it, and research that substantially extended its capabilities or applications.

Sample questions

The following are some interesting questions that can be answered by reading this timeline:

• Which foundational architectures and ideas preceded the transformer, and how did they set the stage for it?
  • Sort the full timeline by "Event type" and look for the group of rows with value "Prelude".
  • You will see the progression from the Elman network (1990) and LSTM (1997) through seq2seq (2014), Bahdanau attention (2014), Luong attention (2015), Google Neural Machine Translation (2016), and the decomposable attention paper (2016) — each removing a limitation that the previous architecture could not overcome, until Uszkoreit's hypothesis that recurrence was unnecessary entirely motivated the transformer.

• Which transformer models have been commercially deployed at scale, and what were the key architectural or training innovations that distinguished each?
  • Sort the full timeline by "Event type" and look for the group of rows with value "Model launch".
  • You will see the progression from GPT-1 and BERT (2018) through GPT-2, RoBERTa, T5, and Megatron-LM (2019), GPT-3 and the Vision Transformer (2020), DALL-E, RoPE, the Decision Transformer, and AlphaFold 2 (2021), DALL-E 2, the Switch Transformer, FlashAttention, ChatGPT, and Whisper (2022), LLaMA, GPT-4, LLaMA 2, Mistral 7B, GPT-4V, and Gemini (2023), and Claude 3, Llama 3, GPT-4o, o1, and DeepSeek (2024–2025).

• How has transformer research expanded into domains beyond natural language processing?
  • Sort the full timeline by "Event type" and look for the group of rows with value "Research" and "Research (application)".
  • You will see efficient attention research (Longformer, Reformer, Sparse Transformers — 2020), RoPE positional encoding (2021), and DeepMind's AlphaFold 2 application to protein structure prediction (2021) — illustrating how transformer research has simultaneously deepened the architecture's theoretical foundations and extended its reach into computer vision, reinforcement learning, speech recognition, and structural biology.

• Other events are described under the following types: "Prelude", "Model launch", and "Research".

Big picture

Time period (approximately)	Development summary	More details
1990–2016	The RNN era and the road to attention	Sequence modelling is dominated by recurrent neural networks (RNNs), with LSTM (1997) becoming the standard architecture for language tasks. RNNs cannot be parallelized and struggle to propagate information across long sequences, placing a practical ceiling on model scale and quality. The seq2seq encoder–decoder paradigm emerges in 2014, and the attention mechanism is introduced the same year by Bahdanau, Cho, and Bengio to solve the fixed-size vector bottleneck that limits seq2seq quality on long inputs. Luong et al. (2015) refine attention further, introducing the dot-product formulation that would directly influence the 2017 transformer. By 2016, Google Translate has replaced a decade of statistical machine translation work with a neural seq2seq model developed in nine months, demonstrating the pace of progress enabled by neural sequence modelling. The decomposable attention paper (2016) by Uszkoreit and colleagues then shows that attention alone — without recurrence — can match LSTMs on NLP tasks, directly motivating the hypothesis that recurrence is unnecessary.
2017–2019	The transformer is introduced and the pretraining paradigm is established	The transformer architecture is introduced in June 2017 by researchers at Google, replacing recurrence with multi-head self-attention and enabling full parallelization over token sequences for the first time. Within a year, OpenAI introduces GPT-1 (2018) and Google introduces BERT (2018), establishing the pretraining-then-finetuning paradigm: a single large transformer pretrained on unlabeled text can be fine-tuned to achieve state-of-the-art results across diverse NLP tasks. GPT-2 (2019) demonstrates strong zero-shot generalization, RoBERTa (2019) shows that training procedure matters as much as architecture, and T5 (2019) unifies all NLP tasks into a single text-to-text framework. By the end of 2019, the transformer has displaced all prior architectures as the dominant approach to NLP, and Megatron-LM (2019) demonstrates that multi-GPU model parallelism can scale transformers to billions of parameters.
2020–2021	Scaling, generalization, and the expansion beyond language	GPT-3 (2020) demonstrates that scaling alone — without architectural changes — produces striking few-shot and zero-shot capabilities, establishing the scaling hypothesis as the dominant research direction. The transformer expands decisively beyond NLP: the Vision Transformer (2020) applies transformers to image recognition, DALL-E (2021) extends the architecture to image generation, AlphaFold 2 (2021) applies a transformer variant to protein structure prediction with Nobel Prize-winning results, and the Decision Transformer (2021) extends transformers to reinforcement learning. RoPE (2021) introduces a positional encoding that would later become the field's standard. Efficient attention research — Longformer, Reformer, and Sparse Transformers — addresses the quadratic scaling bottleneck. The transformer is no longer an NLP architecture but a general-purpose sequence modelling framework applicable across domains.
2022–2023	The public breakthrough and the open-weight ecosystem	ChatGPT (November 2022) brings transformer-based AI to global public attention for the first time, reaching 100 million users in two months and triggering a wave of investment, regulatory attention, and competitive development. FlashAttention (2022) and Whisper (2022) become standard infrastructure components. GPT-4 (March 2023) achieves human-level performance on professional benchmarks. Simultaneously, an open-weight ecosystem emerges as a counterweight to closed commercial models: LLaMA (February 2023) and LLaMA 2 (July 2023) from Meta AI, and Mistral 7B (September 2023) from Mistral AI, demonstrate that carefully trained smaller models can match or approach much larger proprietary ones. Multimodal transformers become standard, with GPT-4V (2023) and Gemini (2023) integrating vision, audio, and language in widely deployed systems. The EU AI Act and US executive orders on AI are accelerated by the public impact of ChatGPT and GPT-4.
2024–present	Reasoning models, multimodality, and geopolitical disruption	Inference-time scaling emerges as a new axis of capability improvement: OpenAI's o1 (September 2024) demonstrates that allocating more compute at inference time for chain-of-thought reasoning substantially improves performance on complex tasks, challenging the assumption that capability gains require ever-larger training runs. Natively multimodal transformers become standard with GPT-4o (May 2024), processing text, audio, and images in a single unified architecture. Anthropic's Claude 3 (March 2024) establishes that frontier capability is no longer exclusive to OpenAI. DeepSeek-V3 and DeepSeek-R1 (late 2024/early 2025) demonstrate that frontier-class transformer training is achievable at a fraction of previously assumed costs — reportedly around $5.6 million — establishing China as a peer competitor in frontier transformer research and disrupting assumptions about the economics and geopolitics of large language model development. The open-weight ecosystem matures with Llama 3 (2024), making frontier-class open models freely available for the first time.

Full timeline

Inclusion criteria

Entries are included if they meet one or more of the following criteria:

• Architectural predecessors: work that directly motivated or enabled the transformer, including sequence modelling architectures (RNNs, LSTMs), the seq2seq paradigm, and attention mechanisms developed in the 2010s.
• Transformer architecture research: papers introducing significant modifications to the transformer architecture itself, including new attention mechanisms, positional encodings, normalization schemes, or efficiency improvements (e.g. FlashAttention, RoPE, grouped-query attention).
• Landmark model releases: transformer-based models whose release represents a qualitative step change in capability, scale, accessibility, or public impact — not incremental version updates to existing models.
• Expansion beyond NLP: the first or most significant applications of the transformer architecture to a new domain (computer vision, speech recognition, protein structure prediction, reinforcement learning, image generation), where the work demonstrates the architecture's generality rather than merely applying it to a new dataset.
• Infrastructure and training methods: techniques that substantially changed how transformers are trained or deployed at scale, including model parallelism, efficient attention, and inference-time compute scaling.
• Geopolitical and ecosystem milestones: events that substantially changed the landscape of who develops transformer models and under what terms, including the emergence of open-weight models and major competitive developments across different national contexts.

Entries are generally excluded if they involve:

• Incremental fine-tuned variants of existing models without substantial architectural or capability novelty (e.g. individual LoRA fine-tunes, domain-specific BERT variants).
• Narrow application papers that apply a transformer to a specific task without contributing to the architecture's development or demonstrating a new domain of applicability.
• Benchmark or evaluation papers that assess existing transformer models without introducing new models or architectural contributions.
• Papers selected from arXiv search results without independent evidence of significance (e.g. citation count, coverage in multiple secondary sources, subsequent adoption).
• Individual employee joins or departures at AI organizations, funding rounds below landmark size, or other organizational events not directly connected to transformer development.

Year (month and date)	Event type	Details
1990	Prelude	The Elman network, a type of recurrent neural network (RNN), becomes a well-cited early example of sequence modelling. In theory, information from one token can propagate arbitrarily far down the sequence, but in practice the vanishing-gradient problem leaves the model's state at the end of a long sentence without precise, extractable information about preceding tokens. Addressing this limitation would motivate subsequent work on gating mechanisms and, eventually, the transformer architecture.
1997 (November)	Prelude	Schmidhuber and Hochreiter publish long short-term memory (LSTM), a type of recurrent neural network (RNN) that introduces gating mechanisms to mitigate the vanishing-gradient problem, allowing efficient learning of long-sequence modelling. A key architectural element is the use of multiplicative gating units, in which the outputs of some neurons modulate the outputs of others. LSTM becomes the standard architecture for long-sequence modelling and remains so until the publication of the transformer in 2017. Like all RNNs, however, LSTM still processes tokens sequentially — one at a time from first to last — and cannot operate in parallel over all tokens in a sequence, a limitation that the transformer would later overcome.
2014 (September–October)	Prelude	Two concurrently published papers introduce the sequence-to-sequence (seq2seq) encoder–decoder architecture for machine translation, commonly cited as the originators of the paradigm. The first, by Sutskever, Vinyals, and Le at Google Brain, uses two long short-term memories (LSTMs) with 380 million parameters: an encoder LSTM that converts an input sequence of tokens into a fixed-size vector, and a decoder LSTM that converts that vector into an output sequence. The second, by Cho, van Merriënboer, Bahdanau, and Bengio at Université de Montréal, uses gated recurrent units (GRUs) instead of LSTMs, with 130 million parameters. Both models suffer from a fundamental bottleneck: without an attention mechanism, the entire input sequence must be compressed into a single fixed-size vector before decoding begins, degrading output quality for long inputs. Addressing this bottleneck would motivate the attention mechanism introduced the following year.
2014 (September)	Prelude	Bahdanau, Cho, and Bengio introduce an attention mechanism for seq2seq models, solving the fixed-size vector bottleneck that degraded translation quality for long inputs. Dubbed the "RNN search" model — because it emulates searching through a source sentence during decoding — the mechanism allows the decoder to dynamically attend to different parts of the input sequence at each decoding step, rather than relying on a single compressed vector. This is the first additive attention mechanism, conceptually distinct from the multiplicative gating units used in LSTM. The paper is later recognized as one of the foundational precursors to the transformer: one of the co-authors of the 2017 "Attention Is All You Need" paper, Jakob Uszkoreit, would cite attention-based seq2seq work as the direct motivation for asking whether recurrence could be removed entirely.
2015 (August)	Prelude	Luong, Pham, and Manning at Stanford compare global and local attention architectures for machine translation, building on the additive attention mechanism introduced by Bahdanau et al. the previous year. Their "global" attention model attends to all source tokens at each decoding step, while their "local" attention model uses a sliding window over the source, reducing computation. They find that local attention reduces translation time while mixed attention yields higher quality than global attention alone. The paper establishes attention as a modular, composable component of seq2seq architectures, and the dot-product formulation it introduces would directly influence the scaled dot-product attention used in the 2017 transformer.
2016 (September)	Prelude	Google revamps Google Translate to use Google Neural Machine Translation, replacing the previous model based on statistical machine translation. The new model is a seq2seq model where both the encoder and decoder are 8-layer bidirectional LSTMs. It takes nine months to develop, yet outperforms the statistical approach that took ten years to build, demonstrating the rapid progress enabled by neural sequence modelling. The system would itself be superseded in 2020 when Google Translate upgrades to a transformer-based encoder.
2016 (September)	Prelude	Parikh, Täckström, Das, and Uszkoreit publish "A Decomposable Attention Model for Natural Language Inference", applying a self-attention mechanism to feedforward networks — which are easy to parallelize — and achieving state-of-the-art results on natural language inference with an order of magnitude fewer parameters than LSTMs. Uszkoreit suspects from this result that attention without recurrence would be sufficient for language translation — the hypothesis that would become the title "Attention Is All You Need" the following year. His father, Hans Uszkoreit, a well-known computational linguist, is skeptical of the idea at the time, illustrating how the transformer's core premise ran against conventional wisdom in the field.
2017 (June 12)	Model launch	Researchers at Google publish "Attention Is All You Need", introducing the original transformer architecture. The model is an encoder–decoder with 100 million parameters, motivated by improving seq2seq for machine translation by removing recurrence entirely and processing all tokens in parallel via a multi-head attention mechanism. Unlike prior RNN-based models, the transformer's lack of recurrence makes it straightforwardly parallelizable on GPUs, enabling training at previously impractical scales — a property that would prove critical to the scaling of large language models in subsequent years. As early as spring 2017, even before the preprint is published, one of the co-authors applies the decoder-only variant of the architecture to generate fictitious Wikipedia articles, foreshadowing the decoder-only designs that would later dominate the field. Four days after publication, most of the same authors publish MultiModel, a multimodal transformer architecture, demonstrating the generality of the design beyond text.
2018 (June 11)	Model launch	OpenAI releases "Improving Language Understanding by Generative Pre-Training", introducing the Generative Pre-trained Transformer (GPT-1), a decoder-only transformer with 117 million parameters trained by unsupervised language modelling on the BooksCorpus dataset, then fine-tuned on specific downstream tasks. The paper is authored by Radford, Narasimhan, Salimans, and Sutskever. GPT-1 establishes the pretraining-then-finetuning paradigm for NLP that BERT would extend the same year and that would become the dominant framework for large language model development. Starting with GPT-1, the OpenAI GPT series of decoder-only transformers becomes state of the art in natural language generation, a line of work that would culminate in GPT-3 (2020) and eventually ChatGPT (2022).
2018 (October)	Model launch	Google researchers Devlin, Chang, Lee, and Toutanova publish "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", introducing BERT (Bidirectional Encoder Representations from Transformers), an encoder-only transformer pretrained on English Wikipedia (12 GB) and BookCorpus (4 GB) using two self-supervised tasks: masked token prediction, in which random tokens are hidden and the model must recover them from context, and next-sentence prediction. Unlike the decoder-only GPT-1, BERT processes context from both left and right simultaneously, making it better suited for representation learning and classification tasks than for text generation. In its large configuration BERT has 345 million parameters. BERT achieves state-of-the-art results across a wide range of NLP benchmarks upon release and is open-sourced by Google. In October 2019, Google begins using BERT to process almost every English search query, marking one of the first large-scale deployments of a transformer model in a consumer product used by billions of people.
2019 (February 14)	Model launch	OpenAI releases Generative Pre-trained Transformer 2 (GPT-2), a decoder-only transformer with 1.5 billion parameters in its largest configuration — roughly 10 times larger than GPT-1. GPT-2 is trained on WebText, a dataset of 40 GB of text scraped from outbound links on Reddit with at least three upvotes, comprising approximately 8 million documents. The model demonstrates strong zero-shot performance across a range of language tasks without task-specific fine-tuning, suggesting that sufficiently large language models begin to acquire general capabilities from pretraining alone. OpenAI initially releases only a smaller version of the model, citing concerns about potential misuse for generating disinformation, making it one of the first high-profile cases in which an AI lab delays a model release on safety grounds — a decision that attracts both praise and criticism from the research community.
2019 (July)	Model launch	Facebook AI Research (FAIR) and the University of Washington introduce RoBERTa (Robustly Optimized BERT Approach), a retraining of BERT with improved training methodology rather than a new architecture. The key findings are that BERT was significantly undertrained: training for longer, on more data, with larger batches, removing the next-sentence prediction task, and training on longer sequences all yield consistent improvements. RoBERTa has 125 million parameters in its base configuration and is trained on 161 GB of text — ten times more than BERT — comprising English Wikipedia, BookCorpus, CC-News, OpenWebText, and Stories. RoBERTa achieves state-of-the-art results on several NLP benchmarks, demonstrating that careful attention to training procedure can be as important as architectural innovation.
2019 (August)	Model launch	NVIDIA introduces Megatron-LM, a transformer language model with 8.3 billion parameters — at the time the largest transformer ever trained, 24 times the size of BERT and 5.6 times the size of GPT-2. Megatron-LM is trained on 512 GPUs using 8-way model parallelism and 64-way data parallelism on a 37 GB WebText dataset. The key technical contribution is a model parallelism approach that splits parameters across multiple GPUs without requiring a new compiler or code re-wiring, solving a fundamental obstacle to training transformer models beyond the memory capacity of a single GPU. Megatron-LM surpasses state-of-the-art results on wikitext perplexity and Lambada accuracy. The work establishes model parallelism as a viable path to scaling transformers to hundreds of billions of parameters, a technique that NVIDIA and Microsoft would extend two years later with the 530-billion-parameter Megatron-Turing NLG.
2019 (October)	Model launch	Raffel, Shazeer, Roberts, Lee, Narang, Matena, Zhou, Li, and Liu at Google publish "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer", introducing T5 (Text-to-Text Transfer Transformer), an encoder–decoder transformer that reframes every NLP task — translation, summarization, question answering, classification — as a text-to-text problem, with both inputs and outputs always being strings of text. This unification allows a single model architecture and training objective to be applied across all tasks without task-specific output heads or modifications. T5 is pretrained on the Colossal Clean Crawled Corpus (C4), a 750 GB filtered subset of Common Crawl, and released in multiple sizes ranging from 60 million to 11 billion parameters. The paper conducts one of the most systematic studies of transfer learning design choices published to date, evaluating the effect of pretraining objectives, model architectures, dataset sizes, and fine-tuning procedures across more than 50 NLP benchmarks. T5 achieves state-of-the-art results on many of these benchmarks and the text-to-text framing it establishes becomes widely adopted, influencing the design of subsequent instruction-tuned models including ChatGPT.
2020 (June)	Model launch	Google Translate upgrades from an RNN-encoder–RNN-decoder model to a transformer-encoder–RNN-decoder model. A purely transformer-based decoder does not appear to significantly increase translation quality over the RNN decoder, while the RNN decoder is substantially faster, motivating the hybrid approach. The upgrade demonstrates that the transformer's gains are most pronounced on the encoding side. The system had previously been upgraded in 2016 from statistical machine translation to a purely RNN-based seq2seq model; this second upgrade completes the transition of one of the world's most widely used translation systems to transformer-based encoding.
2020 (June 11)	Model launch	OpenAI releases Generative Pre-trained Transformer 3 (GPT-3) in beta, a decoder-only transformer with 175 billion parameters — more than 100 times larger than GPT-2 — trained on a 570 GB filtered subset of Common Crawl, WebText2, Books1, Books2, and English Wikipedia. GPT-3 demonstrates striking few-shot and zero-shot performance across a wide range of tasks including translation, question answering, and arithmetic, without any gradient updates or fine-tuning at inference time. The results suggest that scaling model size and training data alone, without architectural changes, produces qualitative leaps in capability — a finding that would strongly influence the subsequent direction of large language model research. GPT-3 is accessed via API rather than released as open weights, establishing a pattern of controlled commercial access that several other labs would follow. The model becomes the foundation for ChatGPT two years later.
2020 (January–April)	Research	A wave of research addresses the quadratic scaling of transformer attention with sequence length — a fundamental bottleneck preventing transformers from processing long documents efficiently. Beltagy, Peters, and Cohan at the Allen Institute for AI introduce the Longformer, which replaces full all-to-all attention with a combination of local sliding-window attention and global attention on selected tokens, reducing complexity from O(n²) to O(n) and enabling processing of documents up to 4,096 tokens — eight times longer than BERT. Concurrently, Kitaev, Kaiser, and Levskaya at Google introduce the Reformer, which reduces attention complexity to O(n log n) using locality-sensitive hashing to identify which tokens are most likely to attend to each other, and reversible layers to reduce memory consumption. Child, Gray, Radford, and Sutskever at OpenAI introduce Sparse Transformers, which use fixed sparse attention patterns to generate long coherent text and images. Together these papers establish efficient attention as a major research direction, motivating subsequent work including FlashAttention (2022) and the long-context models that would become standard by 2024.
2020 (May)	Research	Gulati, Qin, Chiu, Parmar, Zhang, Yu, Han, Wang, Zhang, Wu, and Pang at Google introduce the Conformer, a transformer architecture for speech recognition that combines multi-head self-attention with convolutional layers in each block. Standard transformers model global interactions between tokens effectively but struggle to capture local acoustic patterns; convolutional networks capture local features effectively but lack the ability to model long-range dependencies. The Conformer addresses both limitations simultaneously by sandwiching a multi-head self-attention module between two feed-forward modules and a convolution module within each encoder block, achieving state-of-the-art results on the LibriSpeech benchmark. The Conformer architecture becomes the dominant approach to end-to-end speech recognition and is subsequently adopted in large-scale speech systems including Google's production speech recognition infrastructure. It establishes the pattern — later followed by Whisper (2022) — of adapting the transformer to audio by treating the speech signal as a sequence of local features rather than discrete tokens.
2020 (June)	Research	Xiong, Yang, He, Zheng, Zheng, Xing, Zhang, Lan, Wang, and Liu publish "On Layer Normalization in the Transformer Architecture", formally establishing the superiority of the pre-LN convention over the post-LN convention used in the original 2017 transformer. In the original post-LN transformer, layer normalization is applied after each sublayer's residual connection, which produces unstable gradients at the start of training and requires careful learning rate warmup to converge. The pre-LN convention instead applies layer normalization before each sublayer, which stabilizes gradients and allows training to proceed without warmup, leading to faster convergence. Although the pre-LN convention had been proposed independently several times in 2018, this paper provides the first formal analysis explaining why it works. By the mid-2020s, pre-LN has become the default normalization convention in virtually all large transformer models, including the Llama series and GPT-4.
2020 (October)	Model launch	Dosovitskiy, Beyer, Kolesnikov, and colleagues at Google Brain publish "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale", introducing the Vision Transformer (ViT). The model adapts the standard transformer encoder to computer vision by dividing input images into fixed-size 16×16 pixel patches, linearly embedding each patch into a vector, and treating the resulting sequence of vectors as token embeddings — allowing the transformer's self-attention mechanism to operate over image regions rather than words. When pretrained on sufficiently large datasets, ViT matches or exceeds the performance of state-of-the-art convolutional neural networks (CNNs) on image classification benchmarks while requiring substantially less compute to train. The result challenges the long-held assumption that inductive biases specific to image data — such as translation equivariance built into CNNs — are necessary for strong vision performance. ViT's success in turn stimulates a new wave of CNN architectural improvements, as convolutional network designers incorporate design principles from transformers, and spurs the development of transformer-based approaches across object detection, image segmentation, and video understanding.
2021 (January)	Model launch	OpenAI introduces DALL-E, a decoder-only transformer that generates images from text prompts by treating both text and image data as sequences of tokens. The model first autoregressively generates a text token sequence, followed by a token representation of an image produced by a variational autoencoder (VAE), which is then decoded into a pixel-level image. With 12 billion parameters, DALL-E is trained on 250 million text–image pairs scraped from the internet. Unlike later image generation models based on diffusion, DALL-E is a pure autoregressive transformer, demonstrating that the architecture developed for NLP can be extended to image generation by reframing pixels as tokens. The model can combine unrelated concepts in plausible ways — generating images of "an armchair in the shape of an avocado" or "a snail made of harp" — suggesting that large-scale transformer pretraining on paired data produces flexible compositional representations extending beyond language.
2021 (April)	Research	Su, Lu, Pan, Murtadha, Wen, and Liu introduce RoPE (Rotary Position Embedding), a novel positional encoding method for transformers published in the RoFormer paper. Unlike the original sinusoidal absolute positional encoding, which adds a fixed position vector to each token embedding before the attention mechanism, RoPE encodes position by rotating token embeddings in a high-dimensional space by an angle proportional to their position, applied directly within the attention computation. The key mathematical property is that the dot product between two rotated vectors depends only on their relative positions rather than their absolute positions, allowing the model to generalize to sequence lengths longer than those seen during training. RoPE is initially overlooked relative to other positional encoding methods but is later adopted by the Llama series, PaLM, and most subsequent open-weight large language models, becoming the dominant positional encoding in the field by 2024 and displacing the original sinusoidal encoding used in the 2017 transformer.
2021 (June)	Model launch	Chen, Lu, Rajeswaran, Lee, Grover, Laskin, Abbeel, Srinivas, and Mordatch introduce the Decision Transformer, reframing offline reinforcement learning as a sequence modelling problem solvable by a standard transformer. Rather than training a policy by optimizing a reward function directly, the Decision Transformer conditions on a desired future return, the current state, and the action history, and autoregressively predicts the next action. This formulation allows the transformer to be applied to RL without any modifications to the architecture, using the same pretraining-then-finetuning paradigm established by GPT and BERT. The result demonstrates that transformers are not limited to language and vision but constitute a general-purpose sequence modelling framework applicable across domains, a finding that would motivate subsequent work applying transformers to robotics, protein folding, and other sequential decision-making problems.
2021 (July)	Research (application)	DeepMind publishes AlphaFold 2, a transformer-based system for protein structure prediction that achieves accuracy competitive with experimental methods such as X-ray crystallography and cryo-EM, effectively solving a problem that had challenged structural biologists for fifty years. AlphaFold 2 uses a transformer variant called Evoformer, which operates on pairwise representations of amino acid residues rather than token sequences, applying attention over both sequence and structural dimensions simultaneously. The system is pretrained on the Protein Data Bank and fine-tuned using multiple sequence alignments of related proteins. DeepMind subsequently releases predicted structures for over 200 million proteins — nearly every known protein — through the AlphaFold Protein Structure Database, making the results freely available to the scientific community. AlphaFold 2 is widely regarded as one of the most significant scientific applications of the transformer architecture, demonstrating that the attention mechanism generalizes far beyond its origins in natural language processing to fundamental problems in biology. Its lead author Demis Hassabis and John Jumper are awarded the Nobel Prize in Chemistry in 2024 partly for this work.
2021 (October)	Model launch	NVIDIA and Microsoft jointly introduce Megatron-Turing NLG 530B (MT-NLG), the successor to Microsoft Turing NLG 17B and NVIDIA Megatron-LM 8.3B. With 530 billion parameters — three times larger than GPT-3 — and trained on over 4,000 GPUs, MT-NLG is at the time the largest transformer model ever trained. Its training dataset includes The Pile v1 plus 14 additional sources such as Books3, OpenWebText2, Stack Exchange, PubMed Abstracts, and GitHub, totalling over 825 GB. The model demonstrates state-of-the-art performance on several NLP benchmarks and establishes that collaborative infrastructure between major technology companies can push the scale of transformer training beyond what either could achieve independently.
2022 (April)	Model launch	OpenAI introduces DALL-E 2, a text-to-image generation system that combines a CLIP text encoder — itself a transformer — with a diffusion model decoder, replacing the autoregressive transformer decoder used in the original DALL-E. CLIP (Contrastive Language–Image Pretraining) is a transformer trained on 400 million text–image pairs to align text and image representations in a shared embedding space; DALL-E 2 uses the CLIP text encoder to convert a text prompt into an embedding, which is then used to condition a diffusion model that generates the image. The result produces substantially higher quality and more photorealistic images than the original DALL-E while being more controllable and supporting image editing operations such as inpainting and outpainting. DALL-E 2 demonstrates that transformers need not be the generative backbone of image synthesis — they can instead serve as powerful semantic encoders conditioning non-transformer generative models — and contributes to a broader shift in the field toward multimodal architectures combining transformers with diffusion models. The release triggers significant public and commercial interest in AI image generation, prompting competitive responses including Stable Diffusion and Midjourney.
2022 (June 14)	Model launch	Google releases all Switch Transformer models in T5X/JAX, including the 1.6-trillion-parameter Switch-C and the 395-billion-parameter Switch-XXL. The Switch Transformer replaces the standard feedforward neural network (FFN) layer in the transformer architecture with a mixture of experts layer containing multiple FFNs, routing each token to a single expert. This achieves greater model capacity without a proportional increase in compute per token.
2022 (June)	Research	Dao, Fu, Ermon, Rudra, and Ré at Stanford University introduce FlashAttention, an algorithm that implements the transformer attention mechanism efficiently on GPUs by exploiting the memory hierarchy of modern hardware. Standard attention implementations materialize the full attention matrix in GPU high-bandwidth memory (HBM), which is slow; FlashAttention instead computes attention in blocks sized to fit within the much faster on-chip SRAM cache, minimizing data movement between memory levels. The algorithm is mathematically exact — producing identical results to standard attention — while being significantly faster and more memory-efficient, enabling training on substantially longer context windows than were previously practical. FlashAttention achieves up to 7.6× speedup over the standard PyTorch attention implementation on NVIDIA A100 GPUs. An improved version, FlashAttention-2, released in 2023, achieves up to 2× further speedup over FlashAttention and up to 9× over standard PyTorch attention through better work partitioning and parallelism over the sequence length dimension. FlashAttention is rapidly adopted across the large language model ecosystem and becomes a standard component of transformer training infrastructure.
2022 (November 30)	Model launch	OpenAI releases ChatGPT, a chatbot based on a fine-tuned variant of GPT-3.5, itself a transformer-based large language model. ChatGPT is fine-tuned using reinforcement learning from human feedback (RLHF), a technique that aligns the model's outputs with human preferences by training a reward model on human comparisons and using it to guide further fine-tuning. The release is unexpected in its scale of public impact: ChatGPT reaches one million users within five days and one hundred million users within two months, making it the fastest-growing consumer application in history at the time. The launch triggers a wave of investment and development around large language models, prompts rapid competitive responses from Google, Meta, and Microsoft, and brings transformer-based AI to widespread public attention for the first time. It is widely regarded as a turning point marking the beginning of a new era in public engagement with AI, with transformers at its technological core.
2022 (December)	Model launch	Radford, Kim, Xu, Brockman, McLeavey, and Sutskever at OpenAI introduce Whisper, a transformer-based model for speech recognition trained on 680,000 hours of multilingual and multitask supervised data scraped from the internet — orders of magnitude more than prior speech recognition systems. Whisper follows the same patch-based tokenization strategy as the Vision Transformer: the input audio is converted to a log-Mel spectrogram, which is divided into 30-second chunks and treated as a sequence of patches fed into a transformer encoder, with a transformer decoder generating the output text autoregressively. The large and diverse training set allows Whisper to perform robustly across a wide range of accents, recording conditions, and languages without task-specific fine-tuning, approaching human-level performance on English speech recognition benchmarks. Whisper demonstrates that the transformer's strength in NLP and computer vision extends naturally to audio when the signal is reframed as a visual representation, reinforcing the architecture's generality across modalities.
2022 (December)–2023	Research (application)	Google DeepMind develops a series of transformer-based robotics models that establish a new paradigm for general-purpose robot control. In December 2022, RT-1 (Robotics Transformer 1) is introduced — a compact transformer trained on 130,000 robot demonstrations collected by a fleet of 13 robots over 17 months, achieving significantly better generalisation than prior robotic control systems. In July 2023, RT-2 (Robotics Transformer 2) extends this work by fine-tuning a large vision-language (VLM) — pre-trained on internet-scale image and text data — jointly on robotics demonstration data, producing a single vision-language-action (VLA) model that accepts a camera image and a natural-language instruction and directly outputs robot motor commands. By transferring web-scale knowledge to robotic control, RT-2 nearly doubles performance on novel, unseen tasks compared to RT-1, from 32% to 62%, and can perform rudimentary reasoning — identifying "the extinct animal" among objects on a table without having been explicitly trained for that task. In October 2023, RT-X extends the approach to a cross-embodiment setting covering 22 robot platforms across 34 research institutions, demonstrating that a single transformer trained on diverse robotic data generalises across different robot bodies. The RT series establishes the vision-language-action model as the dominant paradigm for transformer-based robotics, succeeded by Gemini Robotics in 2025.
2022–2024	Research (application)	A wave of transformer-based models substantially accelerates drug discovery and structural biology beyond the protein structure prediction addressed by AlphaFold 2 (2021). In 2022, Meta AI releases ESMFold, a transformer that predicts protein structure from sequence alone — without requiring the computationally expensive multiple sequence alignment step used by AlphaFold 2 — enabling structure prediction in milliseconds rather than minutes and making large-scale screening of novel proteins practical. In September 2023, Google DeepMind publishes AlphaMissense, a transformer derived from AlphaFold's Evoformer architecture that predicts the pathogenicity of all 71 million possible human missense variants — single amino acid changes that may cause disease — providing a comprehensive map of genetic variant effects that had previously required years of experimental work to characterise individually. In May 2024, Google DeepMind releases AlphaFold 3, which extends the AlphaFold transformer architecture beyond proteins to predict the structures of nucleic acids, small molecules, and protein-ligand complexes — interactions directly relevant to drug binding and the primary target of structure-based drug design. Together these systems represent a systematic expansion of transformer-based prediction across the full pipeline of drug discovery, from target identification through structural characterisation to binding prediction.
2023 (February)	Model launch	Touvron, Lavril, Izacard, Martinet, Lachaux, Lacroix, Rozière, Goyal, Hambro, Azhar, Rodriguez, Joulin, Grave, and Lample at Meta AI release LLaMA (Large Language Model Meta AI), a family of decoder-only transformer language models ranging from 7 billion to 65 billion parameters, trained exclusively on publicly available data including Common Crawl, Wikipedia, GitHub, Books, ArXiv, and StackExchange. Unlike contemporaneous large language models such as GPT-4 and PaLM, which are accessible only via API, LLaMA's weights are released to the research community under a non-commercial license, making it the first high-quality large language model widely available for local deployment and fine-tuning. The 13-billion-parameter LLaMA model outperforms GPT-3 on most benchmarks despite being ten times smaller, demonstrating that careful data curation and training procedure can substitute for raw parameter scale. LLaMA incorporates several architectural improvements over the original transformer, including RoPE positional embeddings, RMSNorm normalization, and SwiGLU activation functions. The release triggers an explosion of community fine-tuned variants — including Alpaca, Vicuna, and WizardLM — and establishes open-weight large language models as a viable alternative to closed commercial systems, significantly democratizing access to capable transformer-based AI.
2023 (March 14)	Model launch	OpenAI releases GPT-4, a large multimodal transformer-based language model accepting both text and image inputs and producing text outputs. GPT-4 achieves human-level performance on a wide range of professional and academic benchmarks, scoring in approximately the 90th percentile on the Uniform Bar Examination, the 88th percentile on the LSAT, and passing the United States Medical Licensing Examination, benchmarks on which prior models including GPT-3 performed at or below the 10th percentile. OpenAI does not disclose the model's architecture, parameter count, or training data, marking a significant departure from the research transparency that characterized earlier GPT releases and reflecting intensifying commercial competition in the large language model space. GPT-4 is deployed in ChatGPT and via API, and is integrated into Microsoft's Bing search engine and Microsoft 365 productivity suite, representing one of the largest commercial deployments of a transformer model to date. The release cements the transformer as the dominant architecture underlying frontier AI systems and accelerates regulatory and policy discussions around AI safety and governance worldwide.
2023 (July)	Model launch	Meta AI releases LLaMA 2, the successor to LLaMA, a family of decoder-only transformer language models ranging from 7 billion to 70 billion parameters. Unlike its predecessor, LLaMA 2 is released under a permissive commercial license, making it freely available for most commercial use cases and significantly expanding access beyond the research community. LLaMA 2 is trained on 2 trillion tokens — 40% more data than LLaMA — and incorporates reinforcement learning from human feedback (RLHF) fine-tuning to produce LLaMA 2-Chat, a dialogue-optimised variant. The Chat variant achieves performance competitive with proprietary models including ChatGPT on several benchmarks, demonstrating that openly licensed models can approach the capability of closed commercial systems. LLaMA 2 becomes the most widely adopted open-weight large language model of 2023, spawning a large ecosystem of fine-tuned variants and downstream applications, and cementing Meta's position as the primary driver of open-weight transformer development.
2023 (September)	Model launch	Mistral AI, a French AI startup founded by former DeepMind and Meta AI researchers, releases Mistral 7B, a 7-billion-parameter decoder-only transformer language model. Mistral 7B outperforms Meta's LLaMA 2 13B model on most benchmarks despite having nearly half the parameters, demonstrating that architectural efficiency improvements can substitute for raw scale. The model incorporates two techniques not present in the original transformer: grouped-query attention (GQA), which reduces inference memory requirements by sharing key-value heads across query heads, and sliding window attention (SWA), which allows efficient processing of long sequences by limiting each token's attention to a fixed window of preceding tokens. Mistral 7B is released under the Apache 2.0 open-source license with no usage restrictions, making it one of the most permissively licensed high-performance language models available. The release establishes Mistral AI as a significant new entrant in the open-weight transformer ecosystem and demonstrates that small, well-engineered models can challenge much larger ones — a finding that would influence subsequent model development across the industry.
2023 (October)	Model launch	OpenAI releases GPT-4V (GPT-4 with Vision), extending GPT-4 with the ability to accept image inputs alongside text, making it one of the first widely deployed multimodal large language models accessible to the general public. GPT-4V can describe image contents, answer questions about images, interpret charts and diagrams, read handwritten text, and reason about spatial relationships within images. The release marks a significant expansion of the transformer's role beyond text: rather than requiring separate specialist models for vision and language tasks, a single transformer can now handle both within a unified interface. GPT-4V is deployed in ChatGPT for Plus subscribers and via API, and is used in applications ranging from accessibility tools for visually impaired users to scientific image analysis. The release intensifies competitive pressure on Google, whose Gemini multimodal model is announced two months later partly in response.
2023 (December)	Model launch	Google DeepMind releases Gemini, a family of natively multimodal transformer models trained from the ground up to understand and generate text, images, audio, video, and code simultaneously — in contrast to GPT-4V, which added vision capabilities to a primarily text-based model. Gemini is released in three sizes: Ultra, Pro, and Nano, targeting data centre, general deployment, and on-device use cases respectively. Gemini Ultra becomes the first model to surpass human expert performance on the Massive Multitask Language Understanding (MMLU) benchmark, scoring 90.0% against a human expert score of 89.8%. The natively multimodal training approach, in which the model processes all modalities through a unified transformer architecture rather than stitching together separate encoders, represents a significant architectural departure from prior multimodal systems. Gemini Pro is integrated into Google Search, Google Workspace, and the Bard chatbot, which is subsequently renamed Gemini. The release marks Google's most direct competitive response to ChatGPT and signals the consolidation of Google Brain and DeepMind into a single research organisation focused on transformer-based frontier AI.
2023 (December)	Research	Albert Gu and Tri Dao — the same Tri Dao who developed FlashAttention — introduce Mamba, a state space model (SSM) architecture for sequence modelling based on selective state spaces. Unlike the transformer's attention mechanism, which scales quadratically with sequence length, Mamba processes sequences in linear time by maintaining a compressed hidden state that is updated recurrently — analogous to an RNN but with a hardware-aware parallel scan that makes it as fast to train as a transformer. The key innovation is a selection mechanism that makes the state space parameters input-dependent, allowing the model to selectively remember or forget information based on the content of the input rather than applying fixed dynamics to all tokens. Mamba matches or exceeds transformer performance on language modelling at smaller scales (up to 3 billion parameters) while being significantly more efficient on long sequences. A follow-up paper, Mamba-2 (Dao and Gu, 2024), establishes a formal duality between SSMs and transformers, showing that transformers are a special case of a broader family of structured state space models. Mamba is the first architecture since the transformer's introduction in 2017 to be seriously considered as a viable alternative for large-scale sequence modelling, and its development prompts renewed interest in non-attention architectures as complements or successors to the transformer.
2023 (December)	Research	Ainslie, Lee-Thorp, de Jong, Zemlyanskiy, Lebrón, and Sanghai at Google introduce grouped-query attention (GQA), a generalization of the standard multi-head attention (MHA) mechanism that reduces inference memory requirements without significantly degrading model quality. In standard multi-head attention, each attention head maintains its own separate key and value matrices, requiring the entire set to be cached during autoregressive generation — a memory cost that grows linearly with both sequence length and the number of heads. Multi-query attention (MQA), a prior approach, reduces this cost by sharing a single key-value pair across all query heads, but at the cost of reduced model quality. GQA generalizes both: it partitions attention heads into groups, each of which shares a key-value pair, with MQA and standard MHA as special cases of one group and the maximum number of groups respectively. GQA achieves inference speed close to MQA while maintaining quality close to MHA. The technique is rapidly adopted across the large language model ecosystem — incorporated into Llama 3, Mistral 7B, and most subsequent open-weight transformer models — becoming one of the standard architectural components of deployed transformers by 2024.
2023 (February–May)	Research	Two concurrent papers introduce speculative decoding, a technique that substantially accelerates transformer inference without changing model outputs. Leviathan, Kalman, and Matias at Google and Chen, Borgeaud, Irving, Lespiau, Sifre, and Jumper at DeepMind independently propose the same core idea: a small, fast draft model generates several candidate tokens speculatively, which are then verified in parallel by the larger target model in a single forward pass. If the draft tokens match what the target model would have generated, they are accepted; if not, they are rejected and the target model generates a correction. Because a transformer decoder can verify multiple tokens in parallel faster than it can generate them autoregressively one at a time, speculative decoding achieves substantial speedups — typically 2–3× — with identical output distributions to the original model. The technique becomes widely adopted in production transformer inference systems, reducing the cost and latency of deploying large language models at scale.
2024 (February)	Research (application)	Ruoss, Delétang, Medapati, Grau-Moya, Wenliang, Catt, Reid, and Genewein at Google DeepMind publish "Grandmaster-Level Chess Without Search", demonstrating that a transformer trained purely on chess games via supervised learning achieves an Elo rating of 2895 — placing it at grandmaster level — using only static board evaluation with no minimax tree search. Prior computer chess systems, including Stockfish and AlphaZero, relied heavily on explicit search algorithms to evaluate future positions; the finding that a transformer can reach grandmaster level through pattern recognition alone, without any lookahead, is surprising and suggests that the architecture can implicitly learn strategic reasoning of a depth previously thought to require explicit search. The result contributes to a broader body of evidence that transformers can acquire complex reasoning capabilities from large-scale supervised training across domains far removed from language, including reinforcement learning, protein structure prediction, and now combinatorial game playing.
2024 (March)	Model launch	Anthropic releases the Claude 3 family of transformer-based large language models, comprising three tiers: Haiku (fastest and most compact), Sonnet (balanced), and Opus (most capable). Claude 3 Opus becomes the first model to surpass GPT-4 on multiple standard benchmarks simultaneously, including MMLU, HumanEval, and GSM8K, marking the first time a non-OpenAI model leads across a broad range of capability evaluations. All three models support a 200,000-token context window — at the time among the longest available in a commercially deployed model — enabling processing of book-length documents in a single prompt. The Claude 3 release establishes Anthropic as a leading frontier AI lab alongside OpenAI and Google, and demonstrates that the transformer architecture can be pushed to new capability levels by organisations outside the original GPT lineage. Anthropic emphasises AI safety and constitutional AI training methods as distinguishing features of the Claude model family, reflecting the company's founding mission of developing safe and interpretable AI systems.
2024 (April)	Model launch	Meta AI releases Llama 3, a family of decoder-only transformer language models in 8-billion and 70-billion parameter configurations, with a 405-billion parameter version released subsequently in July 2024. Llama 3 is trained on 15 trillion tokens — more than seven times the training data used for Llama 2 — sourced from a significantly expanded and more carefully curated dataset that includes a higher proportion of code, mathematics, and multilingual content. The 70-billion parameter Llama 3 model achieves performance competitive with GPT-4 on several benchmarks, representing a substantial narrowing of the gap between open-weight and closed proprietary models. Llama 3 incorporates grouped-query attention (GQA) across all model sizes, a 128,000-token vocabulary — four times larger than Llama 2's — and is trained with an improved instruction-tuning pipeline. The release further consolidates the open-weight transformer ecosystem around the Llama architecture, with Llama 3 rapidly becoming the dominant base model for fine-tuning, research, and downstream application development across the AI community.
2024 (May)	Model launch	OpenAI releases GPT-4o ("o" for "omni"), a natively multimodal transformer model that processes and generates text, audio, and images within a single unified architecture — in contrast to prior multimodal systems that stitched together separate specialist models for each modality. GPT-4o accepts any combination of text, audio, and image inputs and can produce any combination of text, audio, and image outputs in a single forward pass, enabling real-time voice conversation with response latencies as low as 232 milliseconds, comparable to human conversational response times. The model matches GPT-4 Turbo performance on text and coding benchmarks while being significantly faster and less expensive to run, and substantially outperforms prior OpenAI voice systems by processing audio directly rather than routing it through a speech-to-text transcription step that discarded tonal and emotional information. GPT-4o is made available to free ChatGPT users as well as paying subscribers, representing the first time a frontier-class multimodal transformer is accessible without a subscription. The release demonstrates that the transformer architecture can unify previously separate modalities — text, vision, and audio — into a single end-to-end model, pointing toward a new generation of truly omnimodal AI systems.
2024 (September)	Model launch	OpenAI releases o1, a transformer-based large language model that introduces a new paradigm of inference-time computation: rather than generating an answer immediately, o1 is trained to spend variable amounts of time reasoning through problems before producing a response, using a process OpenAI calls "chain-of-thought" reasoning at inference time. Unlike prior language models that apply a fixed amount of compute per token regardless of problem difficulty, o1 allocates more computation to harder problems and less to simpler ones, effectively trading inference speed for accuracy on complex tasks. o1 achieves performance in the 89th percentile on competitive programming contests, scores 500 out of 500 on the American Mathematics Competition (AMC) 12, and passes the United States Medical Licensing Examination with high scores — benchmarks on which prior models including GPT-4 performed substantially worse. The release represents a significant departure from the prevailing scaling paradigm, in which capability improvements were primarily driven by larger models and more training data; o1 demonstrates that scaling inference-time computation is a complementary and potentially more efficient path to improved capability. The model establishes reasoning as a new competitive axis in large language model development, prompting rapid responses from Google DeepMind, Anthropic, and Meta AI.
2024 (December)	Model launch	OpenAI releases Sora publicly on December 9, 2024, a text-to-video generation system based on a diffusion transformer (DiT) architecture — combining the latent diffusion approach with a transformer backbone rather than the U-Net architecture used in earlier diffusion models. Sora processes video as sequences of spacetime patches, treating video frames in the same way that vision transformers treat image patches, and can generate videos up to 20 seconds long at 1080p resolution from text prompts. The model had been previewed to select filmmakers and safety testers in February 2024, generating significant public attention for the quality of its outputs — including coherent camera motion, consistent characters across frames, and plausible simulation of physical interactions — before its public release. Sora demonstrates that the transformer architecture, extended to the temporal dimension via spacetime patching, can serve as the backbone of a state-of-the-art video generation system, reinforcing the architecture's generality across yet another modality. The app is subsequently shut down in April 2026 and replaced by successor video generation products.
2024 (June)–2025 (February)	Model launch	Anthropic releases two landmark updates to its Claude model family. In June 2024, Claude 3.5 Sonnet is released as a mid-tier model that outperforms the flagship Claude 3 Opus on most benchmarks at lower cost, establishing that larger models are not necessarily better and prompting a reassessment of cost-performance tradeoffs across the industry. In October 2024, an updated Claude 3.5 Sonnet introduces Computer Use in public beta — the first frontier model to directly control a computer interface by moving a cursor, clicking, and typing, enabling a new category of agentic capability. In February 2025, Anthropic releases Claude 3.7 Sonnet, the first Claude model and one of the first frontier models to offer hybrid reasoning: the same model can produce near-instant responses in standard mode or engage in extended, step-by-step thinking made visible to the user, with API users able to set a fine-grained compute budget for thinking. Unlike OpenAI's o-series, which maintains separate model families for standard generation and reasoning, Anthropic's approach integrates both capabilities into a single model — reflecting a design philosophy that reasoning should be an intrinsic capability of a frontier model rather than a separate product. Claude 3.7 Sonnet is released alongside Claude Code, a command-line agentic coding tool, as a research preview.
2024 (August)–2025 (February)	Research (application)	A wave of transformer-based vision-language-action (VLA) models powers a new generation of commercially deployed humanoid robots, marking the transition of transformer robotics research from laboratory demonstrations to industrial deployment at scale. In August 2024, Figure AI releases Figure 02, a humanoid robot integrating a transformer-based vision-language model developed in partnership with OpenAI, deployed at a BMW manufacturing plant in Spartanburg, South Carolina — one of the first commercial deployments of a transformer-powered humanoid in an industrial setting. In February 2025, Figure AI releases Helix, a generalist VLA model that replaces the OpenAI integration with a single end-to-end transformer controlling the robot's full upper body at 200Hz, enabling zero-shot manipulation of thousands of unseen objects and simultaneous two-robot coordination. Concurrently, Tesla's Optimus humanoid robot and other entrants from Physical Intelligence, 1X Technologies, and Agility Robotics each develop their own transformer-based VLA models for robot control, establishing VLA transformers as the standard AI architecture for humanoid robotics and attracting billions of dollars in investment into the sector across 2024–2025.
2024 (December)–2025 (January)	Model launch	DeepSeek, a Chinese AI research laboratory, releases DeepSeek-V3 in December 2024 and DeepSeek-R1 in January 2025, two transformer-based large language models that attract significant international attention for their combination of frontier-class performance and exceptionally low training cost. DeepSeek-V3 is a mixture of experts (MoE) decoder-only transformer with 671 billion total parameters but only 37 billion active parameters per forward pass, trained on 14.8 trillion tokens at a reported cost of approximately $5.6 million — roughly 50 times cheaper than comparable frontier models from OpenAI and Google DeepMind. DeepSeek-R1 is a reasoning model trained using reinforcement learning without supervised fine-tuning on chain-of-thought data, achieving performance competitive with OpenAI's o1 on mathematics, coding, and reasoning benchmarks. Both models are released as open-weight under a permissive MIT license, making frontier-class reasoning transformers freely available for the first time. The releases cause significant disruption in technology markets and policy discussions, raising questions about the resource requirements for frontier AI development and the effectiveness of export controls on AI hardware. DeepSeek demonstrates that transformer models of frontier capability can be trained outside the United States at a fraction of the cost previously assumed necessary, fundamentally challenging assumptions about the economics and geopolitics of large language model development.
2024 (December)–2025	Model launch	Google DeepMind releases Gemini 2.0 Flash on December 11, 2024, explicitly framed as a model designed for the "agentic era" — a generation of transformer deployment in which models operate autonomously across multi-step tasks rather than responding to single prompts. Gemini 2.0 is the first Gemini generation to support native tool use built directly into the model, allowing it to call Google Search, execute code, and invoke third-party functions without external scaffolding, and to produce native multimodal outputs including text, audio, and images in a single forward pass. Despite being a mid-tier "Flash" model, Gemini 2.0 Flash outperforms the larger Gemini 1.5 Pro on key benchmarks at twice the speed. Alongside the model release, Google DeepMind introduces two agentic prototypes: Project Mariner, a browser agent built on Gemini 2.0 that automates web tasks including shopping, form-filling, and information retrieval directly within Google Chrome; and Jules, a coding agent that integrates into GitHub workflows to autonomously plan and execute solutions to software issues. In 2025, Gemini 2.5 Pro extends the family with thinking budgets — allowing developers to set the number of tokens the model uses for reasoning before responding — and thought summaries that expose the model's reasoning process in structured form. The Gemini 2.0 and 2.5 releases mark a broader industry shift in which transformer deployment moves from single-turn question answering toward extended autonomous task completion across tools, browsers, and codebases.
2025 (March)	Model launch	Google DeepMind releases Gemini 2.5 Pro as an experimental preview on March 25, 2025, the first Gemini generation in which all models are natively thinking models — every response begins with an internal chain-of-thought reasoning pass before any text is shown to the user, rather than offering reasoning as an optional mode. Gemini 2.5 Pro introduces thinking budgets, allowing developers to set a precise token cap controlling how much computation the model spends on internal deliberation before responding, and to disable thinking entirely when low latency is required — a finer-grained control over inference-time compute than the binary on/off offered by prior reasoning models. The model debuts at number one on the LMArena human-preference leaderboard by a margin of approximately 40 Elo points over the previous leader, and leads on mathematics, science, and coding benchmarks including AIME 2025, GPQA, and LiveCodeBench. A Deep Think variant, previewed at Google I/O in May 2025 and released in August 2025, introduces parallel chain-of-thought exploration — generating multiple reasoning paths simultaneously and critiquing them before producing a final answer. A research version of Gemini 2.5 with Deep Think achieves gold-medal standard at the 2025 International Mathematical Olympiad, a benchmark previously considered years away from machine solution. Gemini 2.5 Pro is priced at $1.25 per million input tokens and $10.00 per million output tokens, substantially undercutting comparable frontier models at launch.
2025 (March–June)	Research	Following the release of DeepSeek-R1 in January 2025, inference-time chain-of-thought reasoning rapidly becomes a standard feature of open-weight transformer models across multiple organisations. In March 2025, Alibaba's Qwen team releases QwQ-32B, a 32-billion-parameter open-weight reasoning model that matches DeepSeek-R1 on several benchmarks despite having a fraction of its parameter count, released under an MIT license. In April 2025, Alibaba releases Qwen3, a family of open-weight reasoning models including a flagship Mixture of Experts model (Qwen3-235B-A22B, with 235 billion total and 22 billion active parameters) trained on 36 trillion tokens across 119 languages, incorporating thinking budgets for fine-grained control over reasoning compute, and achieving benchmark performance competitive with OpenAI o1, Google DeepMind Gemini 2.5 Pro, and DeepSeek-R1 under an Apache 2.0 license. In April 2025, Microsoft releases Phi-4-reasoning-plus, a 14-billion-parameter dense transformer that outperforms the much larger DeepSeek-R1-Distill-70B on AIME 2025 mathematics benchmarks, demonstrating that careful data curation and reinforcement learning can substitute for scale in reasoning tasks, under an MIT license. In June 2025, Mistral releases Magistral, its first reasoning model, specialising in multilingual reasoning across European languages — addressing a gap in prior reasoning models which predominantly reasoned in English or Chinese. Together these releases establish open-weight reasoning as a standard capability rather than a frontier differentiator, substantially narrowing the gap between closed commercial reasoning models and freely available alternatives within months of o1's introduction.
2025 (April 5)	Model launch	Meta AI releases Llama 4, a family of natively multimodal transformer language models and the first generation of the Llama series to adopt a mixture of experts (MoE) architecture. The initial release comprises two models: Llama 4 Scout, with 109 billion total parameters across 16 experts and 17 billion active parameters per token, supporting a 10 million-token context window — the longest of any open-weight model at launch — and fitting on a single NVIDIA H100 GPU; and Llama 4 Maverick, with 400 billion total parameters across 128 experts and the same 17 billion active parameters per token. A third model, Llama 4 Behemoth, with an anticipated 2 trillion parameters, is announced but remains in training at the time of release. The adoption of MoE architecture represents a significant departure from the dense transformer architecture used in Llama 1 through 3, directly influenced by DeepSeek-V3's demonstration that MoE could deliver frontier-class performance at reduced inference cost. Llama 4 is also the first Llama generation to process text and images as native input modalities from the ground up, incorporating a MetaCLIP vision encoder. The models are released under a community license that permits free use up to 700 million monthly users, above which a commercial license is required — a more restrictive arrangement than the Apache 2.0 license used by Mistral 7B, reflecting the growing commercial stakes of open-weight model releases.
2025 (April 16)	Model launch	OpenAI releases o3, the successor to o1 and the second generation of OpenAI's inference-time reasoning model family. o3 extends the chain-of-thought reasoning paradigm established by o1, with substantially improved performance: on the Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI) benchmark — a test specifically designed to resist pattern memorisation and measure fluid reasoning — a high-compute configuration of o3 scores 87.5%, compared to approximately 5% for GPT-4o and 0% for GPT-3 in 2020, prompting widespread discussion about proximity to artificial general intelligence. o3 also solves 25.2% of problems on EpochAI's Frontier Math benchmark, where no prior model had exceeded 2%, and achieves 88.9% on AIME 2025. A smaller variant, o3-mini, is released on January 31, 2025, offering three reasoning modes (low, medium, and high compute) allowing users to trade latency for accuracy. o3 is the first o-series model capable of agentically using every tool inside ChatGPT during its own reasoning process, integrating tool use directly into the chain-of-thought rather than as a separate step. The model is named o3 rather than o2 to avoid trademark conflict with the O2 mobile carrier. The release intensifies debate about the limits of inference-time scaling and whether ARC-AGI remains a meaningful benchmark for measuring progress toward AGI.
2025 (May)	Model launch	Anthropic releases Claude Opus 4 and Claude Sonnet 4 on May 22, 2025 at its inaugural developer conference, opening the Claude 4 generation. Claude Opus 4 is positioned as the leading coding model at launch, achieving 72.5% on the SWE-bench Verified benchmark — a test of autonomous software engineering on real GitHub issues — and supporting sustained multi-hour agentic workflows in which the model autonomously executes long-running tasks using tools, memory, and background processes. Claude Sonnet 4 serves as the balanced tier, available to free users, with improved coding and instruction-following over its predecessor Claude 3.7 Sonnet. Both models support hybrid reasoning with extended thinking and thought summaries. Claude Opus 4 is the first Claude model to be assessed under Anthropic's AI Safety Level 3 (ASL-3) standard — a more stringent set of safeguards triggered by internal evaluations indicating the model's potential ability to provide meaningful assistance with dangerous biological or chemical synthesis — reflecting Anthropic's commitment to scaling safety measures alongside capability. Claude Code, Anthropic's command-line agentic coding tool first previewed with Claude 3.7 Sonnet, reaches general availability on the same day, establishing Anthropic as a significant player in the emerging market for AI-powered autonomous software development.
2025 (August)	Model launch	OpenAI releases GPT-5 on August 7, 2025, a multimodal transformer-based large language model that directly integrates the chain-of-thought reasoning capabilities previously available only in the separate o-series models (o1, o3). Rather than maintaining parallel model families — one for general use and one for reasoning — GPT-5 unifies both into a single system, automatically applying more or less reasoning compute depending on the difficulty of the task. GPT-5 becomes the default model in ChatGPT for all free and paid users, replacing GPT-4o entirely, and is also deployed via API and in Microsoft Copilot products. The release marks the consolidation of two previously distinct paradigms — standard next-token prediction and inference-time chain-of-thought reasoning — into a single unified transformer model, a design direction that subsequent releases from Google DeepMind, Anthropic, and Meta AI would follow.

Numerical and visual data

Wikipedia views

The chart below shows the number of views of the Wikipedia article on Transformer (deep learning) per month from 2019 to 2026. Views are negligible until late 2022, when the release of ChatGPT triggers a sharp spike — rising from under 1,000 monthly views to over 50,000 by early 2026 — reflecting the explosion of public interest in transformer-based AI following the ChatGPT launch.

Meta information on the timeline

How the timeline was built

The initial version of the timeline was written by User:Sebastian.

Funding information for this timeline is available.

Feedback and comments

Feedback for the timeline can be provided at the following places:

• FIXME

See also

• Timeline of machine learning
• Timeline of large language models
• Timeline of artificial intelligence