Transformer

Transformer is a neural network architecture introduced in 2017 that has become the foundational building block of modern AI . It was originally developed by Google for machine translation, with two core mechanisms: Attention, which selects relevant information across a sequence, and residual connections, which pass information through deep layers . Researchers at OpenAI, including Ilya Sutskever and Alec Radford, were early to recognize its potential beyond translation; Radford's key modification—training the model to predict the next token in a sequence rather than translating between languages—produced GPT-1 and launched the era of large-scale generative pre-training . The architecture's limitation is its lack of writable memory or recursive mechanisms, making long-horizon reasoning and state management challenging, which has spurred research into alternatives like Mamba and State Space Models .