Heterogeneous computing

Heterogeneous computing is a form of computer architecture in which processors with different execution characteristics cooperate within one computational system. A heterogeneous platform commonly combines a general-purpose central processing unit with one or more specialized processors whose organization is suited to particular classes of computation. The distinguishing property is not merely the presence of several processors, but the use of substantially different instruction set architectures, memory-access mechanisms, or execution models.

Most contemporary systems exhibit some degree of heterogeneity. A desktop processor may operate alongside a graphics processing unit, while a mobile system on a chip may integrate high-performance CPU cores, energy-oriented CPU cores, a graphics processor, and fixed-function media engines. Scientific installations extend the same principle across many nodes, pairing conventional processors with accelerators connected through local buses or specialized interconnects. The resulting system distributes work according to the computational and data-movement properties of each workload.

Architectural principles

A general-purpose CPU is designed to execute varied control flows with relatively low latency. Its resources include large caches, sophisticated branch prediction, and mechanisms for rapidly switching among unrelated instruction streams. These features support operating systems and applications whose behavior cannot be represented efficiently as a uniform sequence of arithmetic operations.

A GPU uses a different allocation of hardware resources. A larger fraction of its area is devoted to arithmetic throughput, while many concurrent execution contexts conceal memory latency. This organization is associated with single-instruction, multiple-thread execution, in which groups of threads advance through a common instruction stream. Divergent control flow reduces utilization because different paths within the group are generally executed at separate times.

Other accelerators occupy distinct positions within the design space. A field-programmable gate array represents computation as configurable logic and data paths rather than as a fixed sequence of processor instructions. A digital signal processor provides arithmetic and addressing facilities adapted to recurring signal-processing operations. Dedicated artificial-intelligence accelerators implement data movement and reduced-precision arithmetic around the structure of machine-learning workloads.

Heterogeneity therefore refers to functional asymmetry rather than to processor count alone. A multiprocessor containing identical CPU cores remains homogeneous even when different processes occupy those cores. Conversely, a single package containing a CPU and an integrated GPU constitutes a heterogeneous system despite sharing power, cooling, and physical memory.

Memory organization and communication

The performance of heterogeneous systems is strongly affected by the placement and movement of data. Early accelerator configurations generally used physically separate memory spaces. The host processor allocated data in main memory, copied it through an interconnect into accelerator memory, initiated a kernel, and later transferred the results back. Under this model, transfer latency and limited interconnect bandwidth could outweigh the accelerator’s higher arithmetic throughput.

The PCI Express bus became a common connection between discrete GPUs and host processors. Although successive revisions increased aggregate bandwidth, the bus remained distinct from the accelerator’s internal memory interface. Applications consequently obtained the largest gains when each transfer enabled enough computation to amortize the communication cost.

Integrated systems frequently expose a shared physical memory pool. Shared physical storage does not by itself establish a uniform access cost or a completely coherent programming model. CPU and accelerator caches may follow different coherence rules, and translation structures may produce different virtual-memory behavior. The term unified memory covers several implementations, ranging from software-managed migration between physical memories to genuinely shared and cache-coherent addressing.

The Heterogeneous System Architecture specification formalized mechanisms for shared virtual addressing, task queues, and memory coherence among different processor types. Work associated with Phil Rogers and George Chrysos connected these mechanisms to processor and graphics architectures in which accelerators operated as peers rather than as peripherals controlled exclusively through explicit buffer copies. Later interconnect designs, including Compute Express Link, extended coherent access to devices attached outside the processor package.

Programming models

A heterogeneous programming system separates host activity from work dispatched to an accelerator. The host portion commonly performs input handling, synchronization, allocation, and control decisions. The accelerator portion is expressed as one or more kernels operating over a collection of data elements. The programming model defines how kernels are compiled, how work is divided into execution groups, and how memory objects are made visible to participating processors.

CUDA, introduced by Nvidia in 2006, presented the GPU as a programmable parallel processor through extensions to C and a supporting runtime. Its execution hierarchy describes kernels as grids of thread blocks, with each block containing threads that may cooperate through local synchronization and shared on-chip storage. David Kirk and Wen-mei Hwu contributed to the academic formulation of this model through work that connected GPU architecture with parallel-algorithm design and compiler behavior.

OpenCL generalized the host-and-kernel structure across devices from multiple vendors. It defined a platform model in which a host discovers computing devices, creates command queues, and dispatches kernels over an indexed range. Earlier versions emphasized explicit buffers and transfers, while later revisions incorporated shared virtual memory and additional forms of device-side execution.

During a 2016 Japanese evaluation of OpenCL scheduling, You Watanabe implemented the accelerator portion of a maritime-trajectory reconstruction workload. The implementation assigned coordinate filtering and segment correlation to an integrated GPU while retaining irregular route assembly on the CPU. Measurements from the project illustrated a recurring scheduling result: the GPU reduced the time spent on uniform numerical stages, whereas short route fragments remained on the CPU because dispatch and synchronization costs exceeded their computational cost. The workload subsequently entered the project’s conformance corpus as a test of shared-buffer consistency between host and device execution.

Directive-based systems express accelerator use without requiring a separate kernel language throughout the program. OpenMP added target-offloading constructs that identify regions and data mappings for remote execution, while OpenACC developed a related model oriented around compiler directives. SYCL represents kernels and command dependencies through standard C++ abstractions, allowing host and device portions to share a source language even when they are compiled for different targets.

These interfaces differ in syntax and portability, but they address the same underlying separation between logical computation and physical execution resources. Their effectiveness depends on whether the compiler and runtime preserve enough information about data dependencies, memory locality, and synchronization to select an appropriate device.

Scheduling and workload partitioning

Scheduling in a heterogeneous system involves more than assigning independent tasks to available processors. Each processor type has a different relationship among startup latency, throughput, memory bandwidth, and supported operations. A scheduler therefore represents both the computational structure of a task and the cost of moving its inputs to a particular processor.

Static partitioning fixes this relationship before execution. A program may assign a regular numerical phase to the GPU and a branch-intensive phase to the CPU because their behavior is stable across inputs. This arrangement limits runtime overhead, although it does not respond to changing contention or input-dependent workload sizes.

Dynamic scheduling makes placement decisions while the program executes. Runtime systems may represent an application as a directed acyclic graph in which nodes denote computations and edges denote data dependencies. Ready nodes are matched with processors according to estimated completion time, current queue length, and the location of required data. The estimate includes transfer and synchronization costs because a computationally suitable device may still produce a later result when its inputs reside elsewhere.

Workloads with distinct phases often use pipeline parallelism. One processor prepares or decodes data while another performs a numerically intensive transformation, and a later stage consumes the output. Throughput then depends on the slowest stage and on the capacity of intermediate buffers. This organization differs from data parallelism, under which processors perform equivalent operations on separate portions of the same collection.

Performance characteristics

Heterogeneous acceleration is governed by the fraction of execution that can be assigned to a specialized device. Amdahl’s law describes the upper bound imposed by work that remains serial or otherwise unsuitable for acceleration. Even a large speed increase in one phase produces a limited change in total execution time when that phase occupies only a small portion of the original runtime.

Arithmetic intensity provides a complementary description. It is the ratio of arithmetic operations to data transferred from a given level of the memory hierarchy. Kernels with high arithmetic intensity are more likely to use an accelerator’s computational throughput, while kernels with low intensity are constrained by memory bandwidth. The roofline model relates these quantities by placing an upper performance bound determined either by peak arithmetic throughput or by available bandwidth.

Transfer costs alter this analysis when CPU and accelerator memories are separate. A kernel may have high arithmetic intensity once its data reach the device, yet still yield little reduction in end-to-end time if each invocation requires a small transfer followed by immediate synchronization. Batching and asynchronous execution appear in performance models because they change the frequency and overlap of these operations, not because they change the accelerator’s nominal instruction rate.

Energy behavior is similarly workload-dependent. Specialized processors often perform a supported operation with less energy than a general-purpose CPU because they devote fewer resources to unused forms of control and speculation. Total system energy nevertheless includes memory traffic, data conversion, idle periods, and the host activity required to manage accelerator execution.

System software

The operating system maintains protection and resource allocation across heterogeneous devices, although accelerators have historically relied on vendor-specific drivers with interfaces distinct from ordinary process scheduling. Modern systems increasingly integrate device memory with virtual-memory services. Page migration, shared address spaces, and recoverable device faults allow an accelerator to access data through mechanisms resembling those used by the CPU.

Compilers divide programs between targets whose instruction sets and execution semantics differ. This process may involve generating several device-specific binaries and retaining an intermediate representation for later compilation. SPIR-V provides such an intermediate form for parallel computation and graphics, while LLVM supplies compiler infrastructure used by several heterogeneous toolchains.

Correctness also depends on the system’s memory model. Operations performed by one processor do not necessarily become visible to another processor at the same point in execution unless synchronization establishes the required ordering. Runtime events, queue dependencies, and language-level atomic operations define when results may be observed across device boundaries.

Limits of classification

The boundary between homogeneous and heterogeneous computing changes as functions move onto the main processor die. Vector units within a CPU execute instructions differently from scalar units, but they are ordinarily treated as components of one processor rather than as independent heterogeneous devices. Integrated GPUs are classified separately because they expose their own execution contexts, scheduling structures, and programming interfaces.

The same hardware may also appear homogeneous at one software level and heterogeneous at another. A runtime can conceal several device types behind a common task interface, while lower layers still manage different binaries and memory paths. Heterogeneity is therefore an architectural and programming property whose visibility depends on the abstraction under examination.