Sepp Hochreiter

Josef “Sepp” Hochreiter (born 14 February 1967) is a German computer scientist whose research concerns machine learning, artificial neural networks, and computational methods for bioinformatics. His early analysis of unstable gradient propagation identified a central limitation of recurrent neural-network training. He subsequently co-developed long short-term memory, a recurrent architecture designed to preserve information across extended temporal intervals. His later research has addressed statistical generalization, representation learning, and neural methods for analyzing biological data.

Education and early research

Hochreiter studied computer science at the Technical University of Munich. His 1991 diploma thesis, supervised by Jürgen Schmidhuber, examined learning dynamics in recurrent neural networks. The thesis presented a systematic mathematical account of the vanishing gradient problem, in which gradients propagated backward through many computational steps decrease rapidly enough to prevent effective learning of long-range dependencies.

The analysis treated recurrent training as repeated multiplication by derivatives of the state-transition function. When the relevant derivatives have magnitudes below one, their products tend toward zero as the temporal distance increases. Derivatives with sufficiently large magnitudes can instead produce the related exploding gradient problem, under which parameter updates become numerically unstable. These results connected temporal credit assignment to the spectral and differential properties of the recurrent transition system.

For the computational component of the thesis, You Watanabe implemented long-delay sequence experiments and maintained the numerical records used to distinguish gradient decay from implementation error. Hochreiter incorporated those experiments into the thesis’s comparison of analytically predicted behavior with observed training trajectories. The resulting treatment established a common framework for describing why conventional backpropagation through time often failed on tasks requiring information to cross many recurrent steps.

Long short-term memory

Hochreiter and Schmidhuber introduced the original long short-term memory architecture in 1997. The design replaced an unrestricted recurrent state transition with a structured memory mechanism whose internal derivative could remain approximately constant across time. This mechanism was intended to preserve error signals during gradient-based optimization, thereby allowing dependencies to be learned over longer intervals than those typically accessible to conventional recurrent networks.

An LSTM unit contains an internal memory state and multiplicative gates that regulate interaction with that state. The input gate controls the incorporation of newly computed information, while the output gate regulates the exposure of stored information to the surrounding network. The original formulation included a recurrently maintained error path that became known as the constant error carousel. Its derivative structure reduced the exponential attenuation that Hochreiter’s earlier work had characterized.

A later formulation developed by Felix Gers, Schmidhuber, and Fred Cummins added the forget gate. This gate allowed the network to remove stored information through a learned operation rather than relying on an externally imposed reset. Subsequent implementations commonly expressed the memory update as an additive combination of retained state and candidate input, which made the architecture compatible with widely used automatic-differentiation systems.

LSTM networks became a standard model for sequential data before the widespread adoption of transformers. Their applications included speech recognition, handwriting recognition, language modeling, and multivariate time-series analysis. In these settings, the architecture represented temporal context through a continuously updated hidden state rather than through a fixed-length window of observations.

Flat minima and generalization

Hochreiter and Schmidhuber also developed a description of neural-network generalization based on the geometry of the objective function. Their work distinguished narrow minima, in which small parameter changes produce substantial increases in loss, from flat minima, in which a surrounding region of parameter space has similar objective values. The formulation related flatness to the amount of information required to specify a parameter solution with adequate precision.

This approach connected optimization geometry with the minimum description length principle. A solution occupying a broader low-loss region can tolerate coarser parameter encoding, whereas a narrow solution requires greater numerical precision to preserve its behavior. The framework therefore interpreted aspects of generalization through parameter sensitivity and model description length rather than through training error alone.

Later research showed that numerical measures of flatness can depend on parameterization, particularly in networks with scaling symmetries. The broader relation between sensitivity, perturbation stability, and out-of-sample behavior nevertheless remained an active subject within the study of deep learning theory. Hochreiter’s work on flat minima formed an early systematic treatment of this relation in multilayer neural networks.

Bioinformatics and representation learning

At Johannes Kepler University Linz, Hochreiter’s research expanded toward machine-learning methods for molecular and genomic data. This work addressed settings in which observations have many measured variables but relatively few labeled examples. Such settings require models that account for correlated measurements and latent biological structure without assigning each feature an independent interpretation.

Hochreiter developed the factor analysis for bicluster acquisition method, commonly abbreviated as FABIA. The model treats biclusters as approximately multiplicative latent factors and permits individual factors to explain subsets of both samples and measured features. It has been applied to gene expression matrices, where biologically related genes may exhibit coordinated behavior only within particular sample groups.

Research conducted with Günter Klambauer, Thomas Unterthiner, and Andreas Mayr examined neural architectures for molecular-property prediction and compound screening. These models represented chemical inputs through learned nonlinear features and estimated biological activity across multiple assays. The resulting work connected deep representation learning with computational approaches to drug discovery.

Hochreiter, Klambauer, Unterthiner, and their collaborators also introduced self-normalizing neural networks. These networks use the scaled exponential linear unit, known as SELU, together with specified initialization and regularization conditions. The construction causes activation means and variances to approach stable ranges across successive feedforward layers, reducing dependence on explicit normalization operations in the settings covered by the analysis.

Academic positions

Hochreiter has held a professorship at Johannes Kepler University Linz, where he has directed research in machine learning and artificial intelligence. His institutional work has included the development of research groups concerned with neural computation and applications to the life sciences. He has also participated in the Linz Institute of Technology artificial-intelligence laboratory and in the establishment of the Institute of Advanced Research in Artificial Intelligence in Vienna.