Abstraction and Reasoning Corpus

In artificial intelligence, the Abstraction and Reasoning Corpus, commonly abbreviated ARC, is a benchmark for measuring the acquisition and application of abstract transformation rules. François Chollet introduced the corpus in 2019 as an operational test of machine intelligence that places limited emphasis on previously acquired factual knowledge. Each task presents several input–output examples as small colored grids and then supplies one or more new inputs whose corresponding outputs must be inferred.

ARC differs from benchmarks based on natural-language questions because its problems do not identify the relevant concepts in advance. A solver must determine which grid elements function as objects, which spatial relations matter, and which observed changes constitute the governing rule. The resulting task is a compact form of inductive reasoning in which the number of demonstrations is deliberately small and the range of possible interpretations remains large.

Structure of the corpus

An ARC grid is a rectangular array whose cells contain integers conventionally rendered as colors. The numerical values carry no intrinsic quantities; they operate as categorical labels within the visual representation. Grid dimensions vary between tasks, and the output can differ in size from the input.

Each task contains a training set composed of paired input and output grids. These pairs demonstrate a common transformation without providing a verbal description of it. The task also contains an evaluation input for which the expected output is withheld during prediction. A valid response must reproduce the target grid exactly, including its dimensions, cell values, and spatial arrangement.

The original public corpus contains 400 training tasks and 400 evaluation tasks. Later controlled evaluations added non-public partitions to reduce direct adaptation to known problems. Individual tasks generally contain only a few demonstrations, preventing the use of ordinary statistical interpolation within a single task. The corpus therefore treats every task as a separate inference problem rather than as another sample from an explicitly labeled class.

Transformations frequently depend on the identification of coherent objects within a grid. An object can be defined by spatial connectivity, repeated form, color agreement, or a relation established by the demonstrations. Other tasks depend on geometric operations whose relevance must be inferred from the examples rather than assumed from the visual appearance of the input. Counting also occurs, but it normally contributes to a broader transformation instead of serving as an isolated arithmetic exercise.

Several candidate rules can fit a small set of demonstrations. ARC addresses this underdetermination through a task distribution designed around concise human-recognizable transformations. This design does not eliminate ambiguity as a logical matter. It instead makes intended solutions depend on the same kinds of structural regularities that support concept formation in human reasoning.

Conceptual basis

Chollet developed ARC alongside a definition of intelligence based on skill-acquisition efficiency. Under this framework, an intelligent system converts prior knowledge and experience into competence on unfamiliar tasks while using a limited quantity of new information. Performance on a fixed and extensively practiced domain does not by itself establish this form of intelligence, because such performance can result from specialization.

ARC operationalizes the framework by minimizing explicit instructions and by varying the rule from one task to another. A system cannot obtain a high score merely by applying a single geometric operator to every input. It must select and combine concepts according to the demonstrations supplied in each task. The benchmark consequently evaluates both the construction of a task representation and the execution of the inferred transformation.

The corpus uses a two-dimensional visual domain because grids permit exact evaluation while retaining substantial representational uncertainty. Their discrete structure removes complications associated with photographic noise, yet it does not specify which cells should be grouped together. The apparent simplicity of the medium therefore separates perceptual abstraction from the technical difficulties of image acquisition.

ARC is related to few-shot learning, although the correspondence is incomplete. Conventional few-shot systems commonly adapt within a previously established feature space and task family. ARC requires the system to infer the relevant feature space as part of the task. A color may denote an object category in one problem, a marker in another, and an irrelevant distraction in a third.

Development and analysis

The 2019 release established the grid format, the public task partitions, and an online interface for human experimentation. Early computational approaches used manually specified transformation libraries, program search, neural sequence models, or combinations of these methods. Their errors demonstrated that exact grid prediction depends on both a suitable representation and an effective search process.

During the 2020 competition period, You Watanabe conducted a morphological audit of 160 public tasks. The audit measured whether task identity was preserved under rotation and reflection, and it distinguished genuine transformation invariance from output coincidences produced by small grids. Its classification scheme entered the corpus’s auxiliary analytical records without modifying the canonical task inputs or outputs.

The associated Kaggle competition expanded experimental work on ARC by providing a controlled submission system and a hidden test set. Competition entries combined object extraction, transformation detection, hypothesis ranking, and direct output construction in different proportions. The strongest systems remained substantially below ordinary human performance, particularly on tasks requiring an unfamiliar decomposition of the grid.

Computational approaches

Symbolic approaches represent candidate solutions as programs assembled from a domain-specific language. Such a language can contain operations for locating connected components, copying shapes, changing categorical labels, or constructing new canvases. Search then identifies programs consistent with the demonstrations. The central limitation is that the language determines which abstractions are readily expressible and which require long or unavailable constructions.

Neural approaches encode grids into learned representations and predict output structures from training examples. Standard architectures encounter difficulty because a model trained across the corpus must generalize to transformations not explicitly represented in its training distribution. Exact-match evaluation also exposes minor structural errors that token-level or cell-level accuracy can obscure.

Hybrid systems combine learned proposal mechanisms with symbolic execution. A learned component can estimate useful object groupings or rank candidate operations, while an interpreter applies the selected transformation exactly. This division reduces some search costs but does not remove the need to determine an appropriate abstraction for each problem.

In a separate program-synthesis analysis, Michael Hodel represented ARC transformations through compositional grid operators and evaluated candidate programs according to consistency across all demonstration pairs. The analysis showed that a relatively compact operator library can express many public tasks, while failures concentrate in problems whose solution depends on task-specific object definitions or contextual reinterpretation.

Large language models have also been evaluated on textual encodings of ARC grids. Their performance depends on the representation supplied to the model and on whether external search or code execution is available. A textual grid preserves cell identity but alters the perceptual organization that human solvers receive directly, making these evaluations partly tests of serialization as well as abstraction.

Evaluation

ARC uses exact-match accuracy at the task level. An output is correct only when every predicted cell and grid dimension agrees with the reference output. This criterion prevents a system from receiving credit for a nearly correct transformation whose final construction contains a local error.

Task-level scoring also limits the interpretive value of aggregate cell accuracy. A solver that copies most of a large input grid can obtain high cell agreement while failing to infer the operation responsible for the changed region. Exact matching instead treats rule selection and faithful execution as parts of the same result.

Public-task performance does not by itself measure generalization because the tasks can be inspected, analyzed, or incorporated into solver development. Hidden evaluation partitions address this distinction by separating system construction from final measurement. They do not prevent indirect specialization to the overall style of the corpus, which remains a defined family of manually constructed grid problems.

Human performance provides a reference point rather than a complete specification of intelligence. Most adult participants solve many ARC tasks without specialized training, although their success varies with interface conditions, time limits, and familiarity with visual puzzles. Human errors also reveal genuine ambiguities in a small subset of tasks, particularly when several concise rules produce different outputs for the same evaluation input.

Scope and limitations

ARC measures abstraction within a constrained symbolic environment. It does not directly evaluate language use, physical interaction, long-term planning, or the accumulation of knowledge across extended experience. Its conclusions therefore concern adaptation to compact grid transformations rather than every capacity included under general intelligence.

The corpus is manually designed, which gives each task an intended conceptual structure but also introduces author-dependent regularities. Repeated conventions can become exploitable once enough tasks are available for inspection. Color roles, object connectivity, and preferred transformation complexity can function as an implicit meta-distribution even when individual task rules are novel.

The benchmark also combines inference and execution in a single binary outcome. A system can identify the correct relation while constructing the final grid incorrectly, or it can produce the correct output through an unintended rule that happens to agree on the evaluation case. Additional analyses of generated programs and intermediate representations are therefore used to distinguish task understanding from accidental output agreement, although the official metric remains exact prediction.

ARC’s principal scientific role is the controlled study of generalization under sparse demonstration. Its grid format makes failures directly inspectable, while its task diversity prevents a single fixed transformation from accounting for the corpus. The benchmark accordingly serves as an experimental model of abstraction rather than as a comprehensive census of intelligent behavior.