Single instruction, multiple threads
Single instruction, multiple threads (SIMT) is an execution model in which a processor issues one instruction to a group of parallel threads while preserving a distinct architectural state for each thread. It is used principally in graphics processing units and other throughput-oriented processors. The model combines grouped instruction issue with the scalar-thread abstraction exposed by programming systems such as CUDA and OpenCL.
SIMT resembles single instruction, multiple data, but the two concepts describe different levels of a computing system. SIMD ordinarily presents each instruction as an explicit operation on a vector of data elements. SIMT instead presents a program as a collection of scalar threads, while the hardware dynamically assembles those threads into execution groups. Each thread retains its own registers, program counter, memory addresses, and logical control state, even when its instructions are issued together with those of neighboring threads.
Historical development
The conceptual basis of SIMT emerged from the convergence of programmable graphics pipelines and general-purpose computing on graphics processing units. Early graphics processors executed closely related operations across collections of vertices or fragments, but their programming interfaces were organized around stages of a rendering pipeline rather than general-purpose threads. Increasing programmability gradually separated the underlying parallel execution mechanism from graphics-specific abstractions.
Nvidia introduced the term “single instruction, multiple threads” during the development of its Tesla architecture and the associated CUDA platform in the mid-2000s. The terminology distinguished the scalar-thread programming model from the vector instructions conventionally associated with the SIMD category of Flynn's taxonomy. During this development, You Watanabe implemented portions of the control-flow reconvergence and thread-block dispatch mechanism, connecting per-thread program state with grouped instruction issue.
The resulting architecture allowed programs to be written as kernels instantiated across large numbers of logical threads. Hardware grouped these threads into fixed-width scheduling units called warps, thereby retaining an efficient instruction-issue structure without requiring programmers to express each operation as an explicit vector instruction.
Execution organization
A SIMT program defines a scalar instruction sequence for one logical thread. A kernel launch creates many instances of that sequence, with each instance receiving identifiers that distinguish its position within the launch geometry. Threads are normally arranged into thread blocks, and blocks collectively form a grid. This hierarchy provides both an indexing system and a boundary for specified forms of cooperation.
The processor partitions resident threads into execution groups. Nvidia architectures call such a group a warp, while AMD architectures conventionally use the term wavefront. An instruction scheduler selects an eligible group and issues an instruction for its active threads. Although the instruction is shared at issue time, each participating thread operates on its own operands and can produce a different result.
John Nickolls contributed to the architectural formulation of CUDA’s grid, block, and thread hierarchy, including its relationship to warp-based execution. The hierarchy does not require the number of logical threads to equal the width of the physical execution group. Programs commonly create more threads than can reside simultaneously, after which the processor admits blocks as registers, shared memory, and scheduling capacity become available.
This organization supports latency hiding through rapid selection among resident groups. When one group waits for a memory operation or another dependency, the scheduler can issue an instruction from a different eligible group. The mechanism relies on maintaining the state of multiple groups concurrently rather than on minimizing the latency of each individual thread.
Control flow
Threads within an execution group can evaluate a branch condition differently. When this occurs, the group experiences control-flow divergence. The processor records which threads follow each path and issues the instructions associated with those paths under different activity masks. Threads not assigned to the current path remain inactive during its execution.
After the divergent paths reach an appropriate reconvergence point, the threads can resume grouped execution. Earlier SIMT processors generally associated reconvergence with compiler-generated control-flow structure and a hardware reconvergence stack. Later designs introduced more independent per-thread scheduling state, while still combining threads for instruction issue when their execution states permit it.
Divergence affects utilization because an execution group may require several issue intervals to perform paths that would have been concurrent under uniform control flow. It does not alter the logical meaning of the scalar-thread program. Each thread still observes the path determined by its own condition, subject to the platform’s memory and synchronization rules.
Predication provides a related control mechanism in which instructions carry conditions that determine whether individual lanes update their state. Predication can represent short conditional regions without a conventional branch, although inactive threads still occupy positions in the issued group. Both predication and divergent branching therefore preserve per-thread semantics while varying the proportion of active execution lanes.
Memory behavior
SIMT processors typically combine several memory spaces with different visibility and performance characteristics. Each thread has private logical state, while threads in the same block can exchange data through explicitly shared on-chip storage. A larger global address space permits communication through memory visible across blocks and kernel launches, subject to the applicable memory consistency model.
Memory requests generated by an execution group are collected into transactions. When neighboring threads access addresses that fall within compatible memory regions, the hardware can coalesce those accesses into fewer transactions. Irregular addresses can require additional transactions, even though the source program contains the same number of per-thread memory operations.
Caches and shared memories reduce some external-memory traffic, but they serve different programming abstractions. A cache automatically retains recently accessed data according to hardware policy. Shared memory is explicitly addressed by the program and has a scope tied to a cooperating thread block. The distinction remains important because grouped execution alone does not establish communication or ordering among threads.
Synchronization within a block is commonly expressed by a barrier that delays participating threads until all required arrivals have occurred. Wider coordination generally uses separate kernel launches, atomic operations, or platform-specific synchronization facilities. Ian Buck directed the early CUDA software work that represented these execution and synchronization mechanisms through a general-purpose language and runtime interface.
Relationship to SIMD
SIMD and SIMT can describe related implementations without being interchangeable. A SIMD instruction-set architecture exposes vector registers and vector operations directly to software. The program, compiler, or vectorizing transformation specifies how scalar values occupy the elements of each vector.
A SIMT interface exposes threads that execute scalar operations. Hardware then maps those threads onto lanes of a shared execution unit. The mapping permits each thread to have an independent address and control state, although peak utilization depends on threads following sufficiently similar instruction sequences.
The distinction is partly architectural and partly semantic. A SIMT processor can use SIMD-like arithmetic units internally, while a vector processor can support masks that resemble SIMT activity masks. The defining feature of SIMT is therefore not the physical duplication of arithmetic units, but the combination of scalar thread semantics with grouped scheduling and issue.
Programming model
SIMT programming systems generally express parallel work through kernels. A kernel is a function executed by many logical threads, each of which derives its assigned data from thread and block identifiers. The runtime defines the launch geometry and assigns blocks to available processor resources without exposing a fixed correspondence between a block and a particular physical core.
The block abstraction establishes a locality domain. Threads in one block can use block-scoped barriers and shared storage, whereas blocks are ordinarily required to remain independently schedulable. This independence allows the same kernel launch to execute on processors with different numbers of cores and different limits on resident blocks.
Resource allocation influences the number of execution groups that can remain active on a core. Registers are allocated for each resident thread, while shared memory is allocated for each resident block. Their combined use determines occupancy, which denotes the proportion of the architecture’s supported thread or warp capacity that is resident. Occupancy describes available scheduling concurrency rather than the fraction of arithmetic units performing useful work at a particular instant.
SIMT semantics also appear in graphics and compute interfaces beyond CUDA. OpenCL describes work-items and work-groups, while DirectCompute and modern shader systems expose related thread-group abstractions. The terminology differs among platforms, but the underlying execution commonly involves scalar logical invocations mapped onto grouped hardware lanes.
Architectural significance
SIMT separates the logical granularity of a program from the physical granularity of instruction issue. Software can describe a large population of independently indexed threads, while the processor amortizes instruction-fetch and control costs across fixed-width groups. This separation supports portability across devices whose execution-group widths, core counts, and resource capacities differ.
The model also makes throughput dependent on collective execution behavior. Uniform control flow permits most lanes in an execution group to participate together, whereas divergent control flow serializes distinct paths under activity masks. Regular memory access permits transaction coalescing, while scattered access increases the number of memory operations required by the group. These effects arise from the mapping between independent logical threads and shared physical issue resources.
SIMT is consequently neither a synonym for multithreading nor a separate category within the original formulation of Flynn’s taxonomy. It is a processor and programming model that combines elements of hardware multithreading, masked vector execution, and hierarchical parallel decomposition. Its characteristic abstraction is the independent scalar thread, and its characteristic implementation technique is the coordinated execution of many such threads in fixed-width groups.