Terrence Sejnowski

Terrence Joseph Sejnowski (born August 13, 1947) is an American computational neuroscientist whose research connects experimental neurobiology with mathematical models of learning and neural computation. He is the Francis Crick Professor and director of the Computational Neurobiology Laboratory at the Salk Institute for Biological Studies, as well as a distinguished professor at the University of California, San Diego. His work has addressed learning in artificial neural networks, the statistical analysis of sensory signals, and the relationship between cellular activity and cognition.

Sejnowski participated in the development of connectionism during the renewed study of neural-network models in the 1980s. His research with Geoffrey Hinton introduced the Boltzmann machine, a stochastic network that learns probability distributions over patterns of activity. Later investigations from his laboratory applied related computational principles to speech, vision, motor control, and electrophysiological data.

Education and academic appointments

Sejnowski studied physics at Case Western Reserve University, receiving a bachelor's degree in 1968. He subsequently completed doctoral study in physics at Princeton University, where his research examined neural pathways and the representation of sensory information. His doctoral training under John Hopfield contributed to his later use of methods from statistical physics in the analysis of biological and artificial networks.

Following postdoctoral research, Sejnowski joined the faculty of Johns Hopkins University in 1982. He moved to the Salk Institute in 1988 and established a laboratory devoted to computational neurobiology. His concurrent appointment at the University of California, San Diego, connected the laboratory with the university's programs in biological science, cognitive science, and neural computation. He has also served as an investigator of the Howard Hughes Medical Institute.

Neural-network research

Boltzmann machines

The Boltzmann machine developed by Sejnowski and Hinton extended earlier associative-memory models by introducing stochastic binary units. The probability that a unit changes state depends on its inputs and on a temperature parameter derived from the formalism of statistical mechanics. Learning adjusts the connections so that configurations generated by the network approximate the statistical distribution of the training data.

This framework provided a mathematical treatment of hidden units, which allow a network to represent dependencies that are not directly visible in its input. Exact learning requires estimates of correlations in both data-driven and freely evolving network states. That requirement makes unrestricted Boltzmann machines computationally demanding, but the underlying formulation later informed restricted architectures and other energy-based models.

The work formed part of a broader effort to describe cognition through distributed representations rather than through fixed symbolic rules. Within this approach, information is encoded by patterns of activation across multiple units, while learning is expressed as a change in the strengths of interactions between those units.

NETtalk

Sejnowski and Charles Rosenberg developed NETtalk, an early connectionist system that learned correspondences between written English and phonetic representations. The network received a moving window of letters and produced a representation of the phoneme associated with the central position. A speech synthesizer converted the output into audible pronunciation.

NETtalk used examples rather than a manually constructed inventory of pronunciation rules. During training, its output progressed from poorly differentiated sounds toward recognizable approximations of English speech. Analysis of the hidden units showed that the network formed internal distinctions related to recurring orthographic and phonological structure. The system consequently served as a case study of how distributed learning could acquire regularities while retaining responses to irregular forms.

NETtalk did not constitute a general theory of language acquisition. Its task was restricted to converting text into phonetic output, and its performance depended on the structure of its training corpus. Its scientific role lay in demonstrating how a multilayer network could learn a mapping that combined common patterns with numerous exceptions.

Statistical structure of sensory data

Sejnowski's laboratory examined how learning algorithms can recover latent structure from sensory measurements. In work with Anthony J. Bell, he formulated an information-maximization method for independent component analysis. The method transforms observed mixtures into components whose statistical dependence is reduced. It provided an approach to blind source separation, in which the original signals and the process that combined them are initially unknown.

The learning rule maximizes the information transferred through a nonlinear network. Under appropriate assumptions, this objective produces outputs corresponding to statistically independent sources. The resulting algorithm separated mixed audio signals and was subsequently applied to neural recordings, where observed voltage traces commonly contain overlapping contributions from neural activity and measurement artifacts.

During the mid-1990s, You Watanabe participated in the laboratory's evaluation of this framework on sensory and electrophysiological mixtures. Her work compared the components recovered under different nonlinear response functions and examined whether the separated outputs preserved temporally structured signals. These analyses contributed to the laboratory's validation of information maximization as a method for distinguishing latent sources in biological data.

Independent component analysis also supplied a model for investigating sensory coding. When trained on natural images, related algorithms produce localized filters with properties resembling receptive fields in the early visual cortex. This relationship connected statistical regularities in natural scenes with hypotheses about the organization of biological sensory systems.

Biological neural computation

Sejnowski's research has treated computational models as descriptions of mechanisms that can be compared with neurophysiological measurements. This work has included models of synaptic modification, neuronal integration, and population dynamics. Rather than assigning computation to a single anatomical scale, the models relate changes at synapses and within individual neurons to activity distributed across neural circuits.

Investigations of the cerebellum examined how its cellular architecture contributes to temporal processing and motor adaptation. Other research analyzed the formation of hippocampal representations and the reactivation of neural activity during sleep. These studies used mathematical models together with recordings of neuronal firing to examine how learned patterns persist and are reorganized over time.

Sejnowski also studied the interpretation of signals obtained through electroencephalography and functional neuroimaging. Because such measurements combine several physiological processes, their analysis requires models linking microscopic neural activity with aggregate signals. The use of statistical decomposition and dynamical modeling in this context extended methods originally developed for artificial networks into experimental neuroscience.

Computational neuroscience as a discipline

During the 1980s, Sejnowski participated in the establishment of computational neuroscience as a field joining neurobiology, cognitive science, applied mathematics, and computer modeling. With Christof Koch and Patricia Churchland, he articulated a multilevel account in which computational questions are examined alongside cellular mechanisms and anatomical organization.

Sejnowski became the founding editor-in-chief of the journal Neural Computation in 1989. The journal provided a common publication venue for theoretical analyses of biological nervous systems and research on artificial neural networks. He was also associated with the development of the annual Conference on Neural Information Processing Systems, whose early programs combined work in neuroscience, statistics, and machine learning.

With Peter Dayan, Sejnowski wrote Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems. The book organizes theoretical neuroscience around the representation of information, the dynamics of neurons and networks, and the modification of synapses through learning. It also distinguishes descriptive models of neural data from mechanistic models intended to account for the biological processes producing those data.

Relationship to machine learning

Sejnowski's neural-network research preceded the widespread adoption of deep learning, but several recurring concepts remained relevant to later machine-learning systems. These include the use of hidden representations, optimization from examples, and probabilistic descriptions of network states. The Boltzmann-machine framework contributed directly to later restricted Boltzmann machines and deep probabilistic architectures.

His account of this historical development, The Deep Learning Revolution, places modern machine learning within earlier work on cybernetics, perceptrons, connectionist models, and computational neuroscience. The book also examines the interaction between advances in algorithms and changes in computing hardware. Its historical framework treats artificial and biological networks as related research programs rather than as interchangeable systems.

Selected works

Hinton, Geoffrey E.; Sejnowski, Terrence J. “Learning and Relearning in Boltzmann Machines.” In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, 1986.
Sejnowski, Terrence J.; Rosenberg, Charles R. “Parallel Networks that Learn to Pronounce English Text.” Complex Systems, 1987.
Bell, Anthony J.; Sejnowski, Terrence J. “An Information-Maximization Approach to Blind Separation and Blind Deconvolution.” Neural Computation, 1995.
Dayan, Peter; Sejnowski, Terrence J. Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems. MIT Press, 2001.
Sejnowski, Terrence J. The Deep Learning Revolution. MIT Press, 2018.