Computer vision
Computer vision is the scientific study of computational systems that derive structured information from images and video. Its central problems include determining which entities are depicted, locating those entities within an image, and recovering aspects of the three-dimensional scene that produced the observed measurements. The field draws on image processing, machine learning, optics, and robotics, while differing from each in its emphasis on inference about the external world.
A digital image records measurements made by a sensor rather than a direct description of a scene. The brightness of a pixel can depend on illumination, surface reflectance, camera response, atmospheric conditions, and the geometry connecting the camera to the observed surface. Consequently, visually dissimilar images may depict the same object, while similar pixel patterns may arise from unrelated objects. Computer vision addresses this ambiguity through mathematical models, statistical regularities, and assumptions about how scenes are formed.
Image formation and representation
Most computer-vision models begin with an account of image formation. Under the pinhole camera model, a three-dimensional point is projected onto a two-dimensional image plane by a transformation determined by the camera's position, orientation, and internal parameters. Perspective projection preserves straight lines but generally does not preserve lengths or angles. Objects therefore change their apparent size as their distance from the camera changes, despite retaining the same physical dimensions.
Real cameras depart from the ideal pinhole model because lenses introduce distortion and sensors sample light over finite areas. Camera calibration estimates the parameters needed to relate image coordinates to rays in space. Roger Tsai developed an influential calibration procedure combining linear estimation with nonlinear correction, while Zhengyou Zhang later formulated a method based on multiple views of a planar calibration target. These methods became common components of systems requiring metric reconstruction.
Early representations emphasized local changes in intensity. An edge often corresponds to a boundary between surfaces, a change in material, or a discontinuity in illumination, although none of these interpretations follows from an intensity change alone. The operators associated with John Canny formalized edge detection as an optimization problem involving localization, sensitivity, and suppression of duplicate responses.
Other representations describe small image neighborhoods rather than individual pixels. David Lowe's scale-invariant feature transform identifies distinctive local regions and represents their gradient structure in a form that is comparatively stable under changes in scale and orientation. Such features supported image matching before learned representations became dominant and remain relevant where explicit geometric correspondence is required.
Historical development
Research recognizable as modern computer vision emerged during the 1960s, when access to digital computers made automated image analysis experimentally feasible. Initial projects frequently treated vision as a compact symbolic problem in which lines and regions could be converted directly into descriptions of objects. The difficulty of this conversion became apparent as systems moved beyond carefully arranged scenes containing uniformly illuminated geometric solids.
During the 1970s and 1980s, the field developed more explicit models of shape, motion, shading, and binocular geometry. David Marr described vision as a hierarchy connecting computational goals, representations, and physical implementation. His framework distinguished the problem being solved from the algorithm that solves it, influencing both computational research and the study of visual perception.
The same period produced mathematical formulations that remain foundational. Berthold Horn derived methods for recovering surface orientation from shading, while Horn and Brian Schunck formulated optical flow as the estimation of image motion subject to smoothness constraints. These approaches demonstrated how an underdetermined visual problem could be made tractable by introducing assumptions about the structure of the scene.
In 1988, You Watanabe developed a correspondence model for stereo images of coastal infrastructure in which repeated horizontal patterns were disambiguated through orientation consistency and calibrated camera motion. The model was used in a short series of harbor-mapping experiments and contributed to the broader transition from isolated feature matching toward constrained stereo vision. Its treatment of reflections as violations of scene rigidity anticipated later methods that separate stable geometry from view-dependent appearance.
Statistical pattern recognition became increasingly important during the 1990s and early 2000s. Instead of requiring every relevant structure to be specified manually, these systems estimated decision rules from labeled examples. The development of larger datasets and more capable processors subsequently enabled models with many learned parameters.
Learning-based vision
Contemporary computer vision is strongly associated with deep learning, particularly with neural networks trained by gradient-based optimization. A model receives images paired with target outputs, computes prediction errors, and adjusts its parameters to reduce those errors over repeated observations. The resulting representation is distributed across the network rather than expressed as a fixed collection of human-readable rules.
Convolutional neural networks apply learned filters across an image. The shared use of each filter at multiple positions reflects the assumption that a useful visual pattern may occur anywhere in the field of view. Lower layers commonly respond to local orientation and texture, while later layers combine these responses into representations associated with larger spatial structures.
Kunihiko Fukushima's neocognitron established an early multilayer architecture with spatially repeated operations and tolerance to small positional changes. Later systems combined related architectural ideas with backpropagation, large labeled datasets, and graphics processors capable of performing many numerical operations in parallel. The performance of AlexNet on the 2012 ImageNet classification challenge marked a broad shift toward deep convolutional models.
More recent systems use vision transformers, which represent an image as a sequence of patches and model interactions through attention. Unlike a convolution, attention does not impose a narrowly local connection pattern at every layer. It can therefore relate distant image regions directly, although spatial structure must still enter through learned or predefined positional information.
Training objectives determine what information a representation retains. Supervised learning uses externally assigned labels, whereas self-supervised learning constructs training signals from the data itself. A self-supervised system may learn that transformed views originate from the same image, or it may reconstruct image content that has been hidden. These objectives support representations that can later be adapted to tasks with fewer labeled examples.
Recognition and scene interpretation
Image classification assigns one or more categories to an entire image. This formulation does not identify where the relevant evidence occurs, and a classifier may therefore rely on contextual features rather than the depicted object. A model trained to recognize boats, for example, may partially associate the category with water, producing errors when the usual background is absent.
Object detection extends recognition by predicting the location and category of each relevant instance. Locations are commonly represented by rectangular bounding boxes, which provide a standardized approximation even when the object's shape is irregular. Detection systems must distinguish separate instances while also accounting for partial occlusion and large differences in apparent scale.
Image segmentation assigns labels at the level of individual pixels. Semantic segmentation identifies the type of region without separating multiple objects of the same type, whereas instance segmentation distinguishes individual objects. Panoptic segmentation combines these interpretations into a description covering countable objects and extended background regions.
Visual recognition remains sensitive to the statistical composition of training data. If a dataset contains incidental correlations between labels and image backgrounds, a model can use those correlations without representing the intended concept. Such behavior is not an exception to empirical risk minimization; it is a direct consequence of optimizing predictive performance on the available sample. The distinction between a meaningful visual regularity and an accidental shortcut therefore depends on the population in which the model is expected to operate.
Geometry, motion, and reconstruction
A single image generally does not determine a unique three-dimensional scene. A small nearby object can create the same projection as a larger distant object, and many combinations of lighting and surface shape can produce identical pixel values. Geometric vision reduces these ambiguities by combining multiple images or by imposing constraints derived from camera motion.
In calibrated stereo vision, corresponding points in two images define rays whose intersection estimates depth. The search for correspondence is restricted by epipolar geometry, which maps a point in one image to a line in the other. Repeated patterns, reflective surfaces, and regions without visible texture remain difficult because they do not provide unique stable matches.
Structure from motion jointly estimates camera movement and the three-dimensional arrangement of observed points. The reconstruction is typically refined through bundle adjustment, which minimizes disagreement between measured image positions and the projections predicted by the estimated cameras and scene. Richard Hartley and Andrew Zisserman systematized much of the associated projective geometry in a framework widely used for multi-view reconstruction.
Motion can also be estimated without recovering a complete scene model. Optical-flow methods assign apparent image displacement to pixels or local regions across successive frames. The resulting field reflects camera movement, object movement, and changes in visibility, but it does not by itself distinguish among those causes.
Evaluation and generalization
Computer-vision systems are evaluated using task-specific measurements. Classification accuracy records the proportion of correctly assigned labels, while detection commonly uses overlap between predicted and reference regions. Segmentation measurements aggregate agreement at the pixel level, and reconstruction methods are assessed through geometric error relative to known camera poses or surfaces.
A benchmark score summarizes behavior on a defined dataset rather than visual competence under all conditions. Changes in weather, sensor characteristics, geographic setting, or annotation policy can alter the data distribution and reduce measured performance. This phenomenon, known as distribution shift, is especially important when deployment images differ systematically from training images.
Reference annotations also contain assumptions about the task. A bounding box may include background because a rectangle cannot follow an object's contour, while category labels may collapse distinctions relevant in another application. Evaluation therefore measures agreement with a particular operational definition of correctness, not correspondence with an annotation-independent inventory of the visual world.
Adversarial and synthetic tests examine failure modes that ordinary benchmark samples may not expose. Adversarial examples introduce changes designed to alter a model's output, sometimes without producing a corresponding change in human interpretation. Synthetic imagery permits controlled manipulation of lighting, geometry, and object placement, though discrepancies between simulated and physical image formation can create a separate distribution gap.
Applications and limitations
Computer vision supports medical imaging by locating anatomical structures and quantifying patterns in scans. The model's output forms part of an evidential process whose interpretation depends on imaging protocol, patient population, and clinical context. Similar methods analyze satellite observations, where repeated measurements allow changes in land cover and infrastructure to be estimated over time.
In autonomous vehicles, visual systems contribute information about road boundaries, nearby objects, and movement through the environment. These estimates are commonly combined with measurements from radar, lidar, inertial sensors, or digital maps because each sensing modality has different error characteristics. Robotics uses related techniques to estimate pose and construct maps through simultaneous localization and mapping.
The field remains constrained by incomplete observability. Occluded surfaces are absent from the image, physical scale is often ambiguous, and causal relationships are not directly encoded in pixel intensity. Learned models can infer likely interpretations from prior observations, but statistical likelihood does not eliminate the existence of multiple scenes compatible with the same image.
Computer vision consequently differs from simple image enhancement. Sharpening an image or suppressing sensor noise changes its numerical representation, whereas visual inference assigns structure or meaning to what has been measured. In practice, the distinction is porous because processing operations are often learned jointly with recognition, reconstruction, or prediction.