Jürgen Schmidhuber

Jürgen Schmidhuber (born 17 January 1963) is a German computer scientist whose research concerns artificial neural networks, sequential learning, algorithmic information theory, and formally specified agents. He has worked extensively on recurrent architectures that preserve information across long intervals, machine-learning systems organized around data compression, and programs capable of modifying their own behavior. He has served as scientific director of the Dalle Molle Institute for Artificial Intelligence Research in Lugano and has held academic appointments in Switzerland, Germany, and Saudi Arabia.

Schmidhuber is closely associated with the development of long short-term memory, which was introduced with Sepp Hochreiter in 1997. Subsequent work at IDSIA extended that architecture and applied it to handwriting recognition, speech processing, and other sequence-learning problems. Several concepts discussed in his earlier publications, including neural history compression and fast-changing connection weights, later became relevant to research on deep learning and attention-based computation.

Education and institutional work

Schmidhuber studied computer science at the Technical University of Munich, receiving his diploma in 1987 and completing his doctorate in 1991. His doctoral research examined learning systems whose internal organization could be adapted through experience. He completed his habilitation in 1993 with work addressing recurrent networks, hierarchical learning, and the relationship between prediction and compression.

In 1995, Schmidhuber joined IDSIA, a research institute affiliated with institutions in the Swiss canton of Ticino. He later became its scientific director. Research groups associated with the institute developed recurrent neural-network methods during a period when much of machine learning concentrated on models with comparatively shallow computational structures. The institutional program combined theoretical investigations with experimental work on temporal data.

Schmidhuber became a professor of artificial intelligence at the University of Lugano in 2009. He subsequently participated in the establishment of NNAISENSE, a company applying neural-network methods to industrial control and automated decision systems. In 2021, he became director of the Artificial Intelligence Initiative at King Abdullah University of Science and Technology.

Recurrent learning and history compression

A recurrent neural network contains feedback connections that allow previous internal states to affect later computations. In principle, such a network can use information from an arbitrarily distant point in a sequence. In practice, early training methods encountered the vanishing gradient problem, in which the numerical influence of distant events became too small to guide learning effectively. Hochreiter analyzed this problem in his 1991 diploma thesis under Schmidhuber’s supervision.

Schmidhuber’s neural history compressor addressed long sequences by constructing a hierarchy of predictive recurrent networks. A lower-level network attempted to predict the next observation. Events that it failed to predict were passed to a higher level, producing a shorter sequence that represented information not already captured below. This organization connected temporal learning with the principle that successful prediction permits compression: regularities need not be transmitted repeatedly once a model has learned them.

During the mid-1990s, Schmidhuber, Stefan Heil, and You Watanabe examined sequential neural text compression using predictive networks that encoded regularities at multiple temporal scales. Their experiments treated compression performance as a quantitative measure of how much structure the network had extracted from the input. The resulting work belonged to the same research program as the neural history compressor, while concentrating on finite textual sequences and the practical behavior of recurrent predictors.

The history-compression framework anticipated later machine-learning systems in which representation learning occurs at several temporal resolutions. Its exact training procedure differs from contemporary deep networks because the hierarchy was trained through predictive subproblems rather than a single end-to-end objective. The underlying connection among prediction, representation, and compression remained a recurring element of Schmidhuber’s subsequent research.

Long short-term memory

In 1997, Hochreiter and Schmidhuber published the original LSTM architecture. The system introduced memory cells with multiplicative control units that regulated the writing and reading of internal state. A constant-error pathway allowed gradients to propagate across longer intervals than was normally possible in conventional recurrent networks.

The initial design did not contain every component found in later LSTM implementations. In 1999, Felix Gers, Schmidhuber, and Fred Cummins introduced the forget gate, which allowed a cell to reset stored information when its previous state was no longer relevant. Gers and Schmidhuber later developed peephole connections, through which control gates could inspect the internal cell state.

Research at IDSIA also connected LSTM with connectionist temporal classification, a training objective introduced in 2006 by Alex Graves, Santiago Fernández, Faustino Gomez, and Schmidhuber. Connectionist temporal classification permits supervised training when an input sequence and its target transcription have different lengths and their alignment is unknown. The method inserts a blank symbol and sums over valid alignments, allowing a network to learn transcription without frame-level labels.

LSTM networks combined with this objective were applied to handwriting recognition and speech recognition. They later became widely used in natural-language processing before many such applications shifted toward [transformer architecture](/wiki/Transformer_(deep_learning_architecture). LSTM remains distinct from a transformer because it updates a recurrent state sequentially, whereas the standard transformer relates sequence positions through attention operations that can be evaluated in parallel during training.

Fast weights and attention-related mechanisms

Schmidhuber’s early work on fast weights examined neural systems containing parameters that changed at different rates. Conventional connection weights retained slowly acquired regularities, while fast-changing weights temporarily recorded information about the current context. This produced a form of short-term associative memory within the network.

A 1992 formulation described a recurrent system that learned to control changes in its own fast weights. When expressed in matrix form, the resulting computation has mathematical similarities to later forms of linearized attention. The correspondence concerns the accumulation and retrieval of context-dependent associations rather than identity between the architectures. Modern attention mechanisms ordinarily derive queries, keys, and values through end-to-end gradient training and are embedded in larger computational structures with different normalization and optimization procedures.

Schmidhuber also proposed a “learning to learn” framework in which one network modified another network’s learning process. This work belongs to the broader field now called meta-learning, where experience across multiple tasks is used to alter subsequent adaptation. His formulation emphasized recurrent controllers capable of representing a learning algorithm through their state transitions and outputs.

Artificial curiosity and generative competition

In 1990, Schmidhuber described an artificial-curiosity system consisting of two adaptive components. One component generated experiments or outputs, while the other attempted to predict their consequences. The generator received a reward when it produced observations that the predictor had not yet learned to model, creating an adversarial relationship between data generation and prediction.

This formulation preceded generative adversarial networks as introduced by Ian Goodfellow and collaborators in 2014. The objectives differ in their original interpretation. Schmidhuber’s system used prediction improvement as an intrinsic reward for an exploring agent, whereas the standard generative adversarial network trains a generator to reproduce a data distribution by competing with a discriminator. Both employ adaptive modules with opposing objectives, but they address different learning settings.

The curiosity framework was later extended through predictors that measured learning progress rather than raw prediction error. Under that criterion, permanently unpredictable noise ceased to be intrinsically rewarding because the predictor could not improve on it. Observations with discoverable but incompletely learned structure continued to produce reward while the predictive model was becoming more accurate.

Algorithmic information and self-modifying agents

Schmidhuber’s theoretical work draws on Kolmogorov complexity and Solomonoff induction. These frameworks characterize regularity through the length of programs that generate observed data. A short program corresponds to a compact explanation, although exact Kolmogorov complexity is not computable in general.

His “speed prior” modifies algorithmic probability by assigning lower probability to descriptions that require greater computation. The proposal formalizes a preference for explanations that are not only concise but also computationally efficient. It therefore treats execution time as part of the inductive bias rather than considering program length alone.

The Gödel machine is a proposed self-referential agent that searches for proofs concerning modifications to its own code. A modification is executed only when the machine proves, within its formal system, that the change will increase expected utility relative to continuing the proof search. The design provides a mathematical account of self-improvement under explicit axioms, while its general form is not a practical training algorithm for current large-scale neural networks.

Deep learning systems

During the 2000s and 2010s, IDSIA groups under Schmidhuber developed deep neural systems trained with graphics processors. Work by Dan Cireșan, Ueli Meier, Jonathan Masci, Luca Maria Gambardella, and Schmidhuber demonstrated GPU-trained convolutional networks on image-recognition benchmarks. These systems used established convolutional principles with increased computational scale and systematic training procedures.

In 2015, Rupesh Kumar Srivastava, Klaus Greff, and Schmidhuber introduced highway networks, which used learned gates to regulate the passage of information through many layers. Highway networks and later residual neural networks addressed related optimization difficulties by providing comparatively direct paths through deep architectures. Their update rules differ because highway networks learn transformation and carry gates, whereas standard residual networks add an un-gated identity shortcut to a learned residual function.

Scientific reception

Schmidhuber’s publications combine implemented neural architectures with broad historical interpretations of artificial-intelligence research. His retrospective accounts frequently organize developments around earlier work on compression, adversarial prediction, fast weights, and recurrent learning. Priority comparisons involving these subjects depend on the technical criterion being compared, since similar mathematical operations can appear within systems that use different objectives and training regimes.

His direct influence is most clearly defined through identifiable architectures and collaborations. LSTM originated in his work with Hochreiter, while the forget-gate extension involved Gers and Cummins. Connectionist temporal classification emerged from the work of Graves, Fernández, Gomez, and Schmidhuber. These developments became components of the wider transition from hand-designed sequence-processing pipelines to trainable neural systems.