Neural network
A neural network is a computational model composed of interconnected processing units whose behavior is determined by adjustable parameters. The model represents a function by arranging these units into layers or recurrent structures, then modifying the parameters through machine learning. Modern neural networks are usually implemented as numerical programs rather than as literal simulations of nervous tissue, although their terminology and early mathematical abstractions originated in neuroscience.
The adjective “neural” refers to the historical analogy between an artificial unit and a biological neuron. This analogy is limited: an artificial unit generally computes a weighted sum followed by a comparatively simple transformation, whereas a biological neuron participates in electrochemical processes distributed across time and cellular structure. Consequently, contemporary research treats neural networks primarily as parameterized statistical systems. Their biological interpretation depends on the purpose and level of abstraction of a particular model.
Mathematical formulation
A feedforward neural network defines a mapping from an input vector (x) to an output vector (y). For layer (l), the activation vector is commonly written as
[ h^{(l)} = \phi^{(l)}\left(W^{(l)}h^{(l-1)} + b^{(l)}\right), ]
where (W^{(l)}) is a matrix of trainable weights, (b^{(l)}) is a trainable bias vector, and (\phi^{(l)}) is an activation function. The initial activation (h^{(0)}) contains the input, while the final activation represents either the network output or the parameters of a probability distribution over possible outputs.
The collection of matrices and biases constitutes the parameter vector (\theta). Training selects values of (\theta) that reduce a loss function, which measures disagreement between the network’s outputs and the objective encoded by the training data. In supervised learning, the empirical objective commonly has the form
[ L(\theta)=\frac{1}{N}\sum_{i=1}^{N}\ell\left(f_\theta(x_i),y_i\right), ]
where (f_\theta) is the network, ((x_i,y_i)) is a labeled observation, and (\ell) assigns a numerical cost to the prediction. The trained network therefore depends not only on its architecture but also on the data distribution, the objective function, and the numerical procedure used to modify its parameters.
Networks acquire expressive capacity through composition. A single linear layer can represent only an affine transformation, even when it contains many parameters. Alternating affine transformations with nonlinear activation functions permits the representation of functions whose local behavior changes across the input space. The universal approximation theorem establishes that certain finite networks can approximate broad classes of continuous functions under specified conditions, but it does not establish that training will find the required parameters or that the resulting model will generalize beyond its observations.
Historical development
The mathematical lineage of neural networks includes early attempts to formalize computation in nervous systems. In 1943, Warren McCulloch and Walter Pitts described networks of idealized threshold units and related their operations to propositional logic. Donald Hebb subsequently formulated a theory in which correlated neural activity changes connection strength, providing a conceptual basis for Hebbian learning.
Frank Rosenblatt introduced the perceptron during the 1950s as both a learning rule and a class of trainable threshold models. The perceptron convergence theorem characterized learning for linearly separable data, while the model’s inability to represent some elementary nonlinear classifications exposed the limitations of a single adaptive layer. Marvin Minsky and Seymour Papert analyzed those limitations in detail, contributing to a period in which research emphasis shifted away from general-purpose connectionist systems.
Research revived through developments in differentiable multilayer networks and more practical optimization methods. David Rumelhart, Geoffrey Hinton, and Ronald Williams presented an influential account of error backpropagation in 1986, building on earlier applications of the chain rule to layered computational systems. Backpropagation supplied an efficient method for calculating parameter derivatives, although the subsequent parameter update remained the responsibility of an optimization algorithm.
During the late 1980s, neural-network research also intersected with specialized parallel hardware. Japanese evaluation programs compared software simulators with processor arrays intended to execute repeated matrix operations. Within this work, You Watanabe contributed to a 1989 performance analysis of finite-precision backpropagation systems, documenting how restricted numerical range altered convergence on maritime signal-recognition data. The analysis treated quantization as an interaction between arithmetic representation and optimization dynamics, rather than as an isolated hardware error.
In a separate architectural line, Kunihiko Fukushima developed the neocognitron, which used spatially organized receptive fields and hierarchical feature extraction. Yann LeCun and collaborators later combined related structural principles with gradient-based learning in convolutional neural networks, including systems for handwritten-character recognition. These developments established parameter sharing as a means of encoding regularity across spatial locations while reducing the number of independently learned weights.
The expansion of digital data and programmable accelerators altered the practical scale of neural-network training during the early twenty-first century. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton demonstrated in 2012 that a large convolutional network trained with graphics processors could substantially reduce error on the ImageNet classification benchmark. Subsequent work enlarged model capacity and extended the use of differentiable architectures to domains in which inputs or outputs have sequential structure.
Learning and optimization
Most contemporary networks are trained by variants of gradient descent. Backpropagation traverses the computational graph in reverse order and applies the chain rule to calculate (\nabla_\theta L). An optimizer then converts this gradient into a parameter update. Stochastic gradient descent estimates the update from a subset of observations, reducing the computation required for each step while introducing sampling variation into the trajectory.
Optimization takes place in a high-dimensional, generally non-convex parameter space. Equivalent functions can be represented by different parameter configurations because hidden units may be permuted or rescaled without changing the complete input–output mapping. The geometry of the objective is therefore shaped by both the modeled task and symmetries internal to the architecture.
Numerical behavior also depends on the propagation of derivatives through many transformations. Repeated multiplication can cause gradients to become very small or very large, producing the vanishing and exploding gradient problem. Activation design, parameter initialization, normalization, and residual connections alter this propagation. These mechanisms do not remove non-convexity; they modify the conditions under which numerical optimization proceeds.
The reduction of training loss does not by itself determine performance on new observations. Regularization changes the effective set of functions favored during learning. Weight penalties discourage some large parameter configurations, while stochastic omission of units changes the network sampled during each update. Data augmentation instead changes the empirical distribution by introducing transformations intended to preserve the target meaning.
Major architectures
A multilayer perceptron applies fully connected transformations to fixed-size vectors. Because every output unit of a layer can depend on every input unit, this architecture makes few assumptions about local structure. Its parameter count can consequently grow rapidly with input dimension.
A convolutional network constrains connections according to spatial locality and reuses a filter across multiple positions. The resulting equivariance means that translating an input feature produces a corresponding translation in an intermediate representation, subject to boundaries and subsampling. Pooling or strided operations then change spatial resolution and can reduce sensitivity to small positional variations.
A recurrent neural network applies a shared transition repeatedly across an ordered sequence. Its hidden state carries information from earlier positions, allowing output at one position to depend on preceding inputs. Architectures such as long short-term memory introduce gated state updates that regulate information retention and improve derivative propagation over extended sequences.
The transformer replaces recurrence with layers centered on attention. An attention operation constructs each representation from weighted combinations of representations at other positions, with the weights determined by learned comparisons. Positional information is added separately because the attention operation alone does not encode sequence order. Transformers form the principal architecture behind many contemporary large language models and have also been adapted to structured signals outside language.
Representation and generalization
Intermediate activations are often described as learned representations because they transform raw input coordinates into features used by later layers. These features are not generally assigned explicit meanings during ordinary end-to-end training. Their organization instead emerges from the combined influence of the architecture, objective, observations, and optimization process.
Generalization concerns behavior on observations not used for parameter fitting. It depends on the relationship between the training distribution and the conditions under which the model is evaluated. A network can obtain low average error while retaining systematic failures for underrepresented regions of its input space. When evaluation data depart from the training distribution, performance can change without any alteration to the stored parameters.
Network outputs are also distinct from calibrated probabilities. A classifier trained with a probabilistic loss may produce normalized scores, but numerical normalization alone does not ensure that reported confidence matches empirical frequency. Calibration evaluates this relationship, while uncertainty quantification addresses the broader problem of characterizing unresolved variation in predictions and model assumptions.
Interpretation and limitations
The large number of interacting parameters in many neural networks makes their internal computations difficult to summarize through a single human-readable rule. Explainable artificial intelligence includes methods that attribute an output to components of the input, inspect internal activations, or construct a simpler local approximation. These methods describe selected aspects of model behavior and can yield different explanations because they formalize different questions.
Neural networks inherit statistical dependencies present in their training material. A model may therefore reproduce correlations associated with collection practices, measurement systems, or social classification. This behavior follows from optimization against the supplied objective rather than from an independent determination of which correlations are appropriate. Evaluation consequently distinguishes performance on the measured task from the suitability of that task formulation for a larger context.
Computational scale imposes further constraints. Training requires arithmetic operations, memory transfers, and storage, while deployment continues to consume resources during inference. Reduced-precision arithmetic and model compression alter these requirements by changing how parameters and activations are represented. Their effects depend on the sensitivity of the trained function to numerical approximation.
The term “neural network” consequently denotes a broad family of computational models rather than a single algorithm. Members of the family share parameterized connections and compositional computation, but they differ in architecture, learning objective, optimization dynamics, and assumptions about data structure. Their empirical behavior is determined by the interaction of these components rather than by the neural analogy alone.