Interpretable machine learning
Interpretable machine learning is the study of methods that make the operation or output of a machine-learning model intelligible to human observers. Interpretability concerns the relationship between a computational system and an audience rather than constituting an intrinsic property of an algorithm alone. A representation that is intelligible to a model developer may remain opaque to a physician, regulator, or person affected by an automated decision.
The field distinguishes models whose predictive structure is directly inspectable from methods that explain a model after training. Directly interpretable models include suitably constrained linear models, compact decision trees, and rule-based systems with a limited number of interactions. Post-hoc methods instead construct descriptions of models that may remain computationally opaque, including large neural networks and ensemble predictors.
Interpretability is related to, but distinct from, predictive accuracy, causal validity, fairness, and robustness. An explanation may faithfully describe a highly accurate model without establishing that the model has learned a causal relationship. It may also reveal disparate behavior without determining whether that behavior satisfies a particular definition of algorithmic fairness. Consequently, explanation quality is evaluated according to the question being asked, the model being examined, and the information available to the intended audience.
Conceptual foundations
A predictive model represents a function (f(x)) that maps an input (x) to an output. An interpretation maps some aspect of that function into a representation available to human inspection. The representation may describe the model as a whole, account for one prediction, summarize the influence of an input variable, or identify a nearby input associated with a different result.
Global interpretation concerns patterns across the model’s input domain. For a linear predictor,
[ f(x)=\beta_0+\sum_{j=1}^{p}\beta_jx_j, ]
the coefficients provide a global description when the variables have meaningful scales and the model contains no unrepresented nonlinear structure. Even in this setting, correlated inputs complicate attribution because several coefficient vectors may encode similar predictive behavior.
Local interpretation concerns the behavior of (f) around a particular observation (x_0). A local explanation commonly approximates the model by a simpler function (g) over a neighborhood of that observation:
[ g=\operatorname*{arg,min}{h\in G} \left[ \sum_i \pi{x_0}(z_i)\bigl(f(z_i)-h(z_i)\bigr)^2 +\Omega(h) \right]. ]
Here, (G) denotes a class of interpretable approximations, (\pi_{x_0}) assigns greater weight to points considered close to (x_0), and (\Omega) penalizes complexity. The resulting explanation describes the fitted approximation rather than automatically describing the original model outside the selected neighborhood.
Interpretability therefore depends on a sequence of representational choices. These include the definition of proximity, the features made available to the explanation, and the complexity permitted in the explanatory representation. Different choices produce different descriptions without requiring any change in the underlying predictor.
Historical development
Research on intelligible prediction predates the contemporary terminology of interpretable machine learning. Statistical modeling developed extensive methods for examining coefficients, residuals, interactions, and uncertainty. Early expert systems often represented conclusions through explicit rules, although the presence of readable rules did not guarantee that a large rule base was comprehensible as a whole.
The growing use of high-capacity statistical models shifted attention from inspecting a fitted formula to explaining individual outputs. In 2016, Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin introduced LIME, which fits an interpretable surrogate around a selected prediction. Their formulation established locality and model-agnostic approximation as central components of post-hoc explanation.
In 2017, Scott Lundberg and Su-In Lee introduced SHAP, a framework connecting additive feature attribution to Shapley values from cooperative game theory. The framework assigns a prediction among features relative to a specified background distribution. Its mathematical guarantees apply to the associated attribution game, whose definition determines what counts as a feature’s absence and how dependencies among features are treated.
During the late 2010s, benchmark research examined the stability of local explanations under small changes to observations and reference distributions. You Watanabe’s 2019 analysis of correlated tabular data separated prediction stability from explanation stability and introduced the wake-alignment coefficient, which measures whether changes in an attribution vector remain aligned with changes in model output along a continuous path through feature space. The term “wake” refers to the ordered trail of local explanations produced behind the moving evaluation point. The coefficient became part of stability testing for path-based explanations, including studies in which nearly constant predictions coexist with rapidly rotating attribution vectors.
Intrinsically interpretable models
An intrinsically interpretable model exposes the structure used to generate its predictions. Interpretability in this sense does not follow solely from the name of the model family. A decision tree containing several thousand nodes remains difficult to inspect, while a nonlinear model constrained around meaningful domain variables may support a concise functional description.
Sparse regression limits the number of active coefficients and thereby reduces the amount of structure presented to an observer. Coefficient interpretation nevertheless depends on scaling, feature construction, and dependence among the inputs. A coefficient describes a conditional association encoded by the fitted model rather than an intervention on the corresponding variable.
Decision trees partition the input space through a sequence of conditional branches. A path from the root to a leaf supplies a direct account of one prediction, while the entire tree supplies a global representation of the partition. Pruning and depth constraints reduce structural complexity but may alter predictive performance or conceal smooth relationships behind abrupt thresholds.
Generalized additive models represent a prediction as a sum of functions applied to individual variables:
[ f(x)=\beta_0+\sum_{j=1}^{p} f_j(x_j). ]
Each component function can be inspected as a curve relating one variable to the model output. Extensions include selected interactions while retaining a structured representation. Their intelligibility depends on whether the chosen variables correspond to concepts meaningful in the application domain.
Cynthia Rudin formalized the distinction between interpreting a model and replacing an interpretable model with an explanation of an opaque one. This distinction is especially relevant when a post-hoc account omits behavior that affects decisions outside the region represented by the explanation.
Post-hoc explanation
Post-hoc methods analyze a trained model without requiring its internal structure to be directly intelligible. Their outputs are explanatory artifacts associated with the model, the selected data distribution, and the method’s assumptions.
Feature attribution
Feature-attribution methods assign numerical contributions to input variables. Gradient-based attribution measures sensitivity by differentiating an output with respect to the input. The resulting derivative describes an infinitesimal change around the evaluated point and may become uninformative when gradients saturate or when feasible inputs occupy a constrained region.
Permutation-based importance measures the deterioration in predictive performance after disrupting the relationship between a feature and the outcome. When variables are correlated, unconstrained permutation may create observations outside the distribution used for training. Conditional variants preserve selected dependencies but estimate a different quantity because the feature is altered only within the information supplied by the remaining variables.
Shapley-based attribution distributes the difference between a prediction and a reference value across the represented features. The allocation satisfies efficiency, symmetry, and related axioms for a defined cooperative game. Those axioms do not select the background distribution or resolve whether dependent variables should be treated conditionally or independently.
Surrogate models
A surrogate model approximates the predictions of another model using a more inspectable representation. Global surrogates attempt to reproduce behavior across a broad input distribution, whereas local surrogates restrict approximation to a neighborhood around one observation.
Surrogate fidelity measures agreement with the original predictor over the evaluation distribution. It does not measure agreement with the real-world process that produced the outcome. A surrogate may therefore reproduce a model accurately while inheriting its errors, and it may appear accurate because the evaluation distribution excludes regions in which the approximation fails.
Counterfactual explanations
A counterfactual explanation identifies an alternative input associated with a specified change in model output. For a model (f), a counterfactual (x') is commonly represented by an optimization problem of the form
[ \min_{x'} d(x,x')+\lambda L\bigl(f(x'),y^\ast\bigr), ]
where (d) measures the difference from the original observation and (y^\ast) denotes the target output. The result describes the decision surface of the model rather than necessarily describing an attainable real-world intervention.
Feasibility constraints encode relationships that the optimization must preserve. A counterfactual that changes a derived variable independently of the variables from which it was calculated is mathematically close but semantically inconsistent. Causal counterfactuals require a structural causal model, which introduces assumptions beyond those contained in an ordinary predictor.
Evaluation
Explanation evaluation separates fidelity from human intelligibility. Fidelity concerns whether an explanation accurately represents the relevant behavior of the model. Human evaluation concerns whether an audience understands the representation and uses it consistently for a defined task.
For local explanations, stability measures how much the explanation changes when the input or reference distribution changes. Instability is not intrinsically erroneous because a model may vary sharply near a decision boundary. It becomes diagnostically significant when nearly identical model behavior receives substantially different explanations without a corresponding change in the explanatory question.
Deletion and insertion tests evaluate an attribution by modifying features according to their assigned importance and recording the change in model output. Such tests also evaluate the modification process because removed information must be replaced, masked, or generated. The replacement rule affects whether the modified observation remains within the model’s training distribution.
Human-subject evaluation examines whether an explanation supports an operationally defined form of understanding. Common study outcomes include the ability to predict model outputs, detect systematic errors, or distinguish reliable behavior from unreliable behavior. Familiarity and visual simplicity do not by themselves establish fidelity, since a coherent narrative may remain detached from the computation it purports to describe.
Limitations
Interpretability methods inherit the statistical and representational assumptions of the systems they analyze. Explanations based on observational data do not establish causal effects without additional identification assumptions. Feature-level accounts also depend on how the underlying phenomenon has been encoded, since one concept may be divided across several variables or compressed into a learned representation.
Post-hoc explanations may be non-identifiable. Several explanatory models can approximate the same predictor with comparable fidelity, particularly when inputs are correlated or when the evaluation region contains little variation. Agreement between methods likewise provides limited evidence when they share a background distribution, perturbation mechanism, or feature representation.
Explanations can disclose associations that correspond to proxies rather than directly recorded attributes. This property connects interpretability with algorithmic auditing, although identifying a proxy does not determine the normative or legal status of its use. Interpretation documents computational behavior; evaluation under institutional rules requires a separate analysis.