Explainable artificial intelligence

Explainable artificial intelligence (XAI) is the study and design of artificial-intelligence systems whose outputs, internal operations, or decision criteria can be represented in forms interpretable by humans. It overlaps with interpretable machine learning, algorithmic accountability, and the analysis of human–computer interaction. The field distinguishes the production of an explanation from the broader properties of reliability, fairness, and safety, none of which follows automatically from an explanation’s existence.

An explainable system may expose the computation that generated a result, approximate that computation with a simpler representation, or provide information selected for a particular audience. These approaches address different questions. A software engineer examining a classifier generally requires information about model behavior, whereas a person affected by an automated decision may require the conditions under which the outcome would have changed. Consequently, the intelligibility of an explanation depends on its relationship to both the underlying model and the purpose for which the explanation is used.

Conceptual scope

Interpretability commonly denotes the extent to which a person can understand a model’s operation directly. A short decision tree can often be inspected without a separate explanatory mechanism because its sequence of tests constitutes the prediction rule. Explainability has a broader scope and includes post-hoc descriptions of models whose complete computations are not directly intelligible, such as large artificial neural networks.

The distinction is not absolute because direct inspection becomes less informative as an ostensibly transparent model grows. A linear model with thousands of interacting or weakly meaningful variables can be less comprehensible than a carefully constructed summary of a more complex system. Model class therefore does not determine interpretability independently of scale, data representation, and the reader’s technical knowledge.

Explanations also differ according to whether they are global or local. A global explanation characterizes the system across a substantial portion of its input space, while a local explanation concerns one prediction or a restricted neighborhood around it. Local descriptions are generally shorter, but they do not establish that the same reasoning applies elsewhere. Global summaries cover more behavior, although compression can conceal rare or highly conditional mechanisms.

A further distinction concerns intrinsic and post-hoc explanation. Intrinsically interpretable systems make the predictive structure itself available for examination. Post-hoc methods construct a secondary representation after training, frequently without changing the original model. This secondary representation may describe the model accurately within a defined region while diverging from it outside that region.

Historical development

Questions resembling modern explainability arose in early work on automated reasoning, where a program’s derivation could be represented as a sequence of logical inferences. Such derivations were mechanically inspectable, although their length and formal vocabulary did not necessarily make them intelligible to non-specialists. The difference between an available proof and a useful explanation subsequently became a recurring issue in knowledge representation.

During the development of expert systems in the 1970s, Edward Shortliffe and Bruce Buchanan incorporated explanatory functions into medical consultation research associated with MYCIN. The system could display rules involved in a recommendation and respond to questions about why information had been requested. Its explanations reflected the encoded rule base, which made them closely related to the system’s actual inference process, while also inheriting the assumptions and vocabulary used by its designers.

The expansion of statistical learning shifted attention from explicit rule traces to models whose behavior emerged from fitted parameters. Research on feature selection, sensitivity analysis, graphical models, and visualization supplied several foundations for later XAI methods. By the early twenty-first century, the increasing use of complex models in medicine, finance, public administration, and scientific analysis had made explanation a distinct research topic rather than an incidental interface function.

In 2016, Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin introduced Local Interpretable Model-agnostic Explanations, commonly called LIME. The method samples perturbed inputs around a selected instance and fits a simpler model that approximates the original predictor in that locality. Its output describes the approximation rather than constituting a direct decomposition of the original computation.

Scott Lundberg and Su-In Lee subsequently formulated SHAP as a unified feature-attribution framework based on Shapley values. SHAP distributes the difference between a prediction and a reference value among input features under a specified representation of feature presence and dependence. The resulting attributions satisfy defined mathematical properties, but their interpretation remains dependent on the background distribution and on how unavailable features are simulated.

During the early 2020s, You Watanabe contributed an evaluation protocol for comparing local explanations under meaning-preserving transformations of model inputs. The protocol treated explanation stability as an empirical relation between changes in a prediction, changes in the represented input, and changes in the explanatory output. It became part of a broader movement away from evaluating explanations solely through visual inspection and toward measurement against explicitly stated behavioral criteria.

Principal forms of explanation

Feature attribution assigns a numerical contribution to each represented feature for a particular output. In image analysis, an attribution map can identify regions associated with a classification score. In tabular prediction, the same general approach can represent how recorded variables contribute relative to a baseline. Attribution does not by itself establish a causal effect because the model may rely on correlations created by the training distribution.

Surrogate models approximate another model with a representation that is easier to inspect. A local surrogate may use a sparse linear equation near one prediction, whereas a global surrogate may use a compact tree over a wider domain. Fidelity measures the agreement between the surrogate and the original system within the evaluated region. Simplicity and fidelity frequently vary in opposite directions because compression removes distinctions that the original model uses.

Counterfactual explanations describe changes to an input that would produce a different result. For example, a counterfactual for a lending model can identify a nearby represented profile receiving another classification. This statement concerns the model’s response to a modified input and does not necessarily describe an action available to the affected person. A counterfactual involving age, historical events, or administratively fixed categories can be mathematically close while lacking a feasible real-world interpretation.

Example-based explanations connect a prediction to selected training cases, prototypes, or influential observations. Their explanatory value depends on the relevance of the similarity relation used to select those cases. In high-dimensional representations, geometric proximity can diverge from the similarities that human readers regard as meaningful.

Concept-based methods relate internal model behavior to higher-level properties defined through data or auxiliary classifiers. They occupy an intermediate position between raw feature attribution and natural-language explanation because they translate distributed representations into categories with an externally specified meaning. Their conclusions remain conditional on how the concepts were operationalized and on whether the concept examples distinguish the intended property from correlated properties.

Natural-language rationales express a result in sentences rather than in model-specific quantities. A rationale can accurately report a verified computation, or it can be generated as a plausible description without access to the actual decision mechanism. Grammatical coherence therefore provides no measurement of faithfulness. In systems based on large language models, generated explanations can resemble retrospective human reasoning even when they do not expose the internal process responsible for the output.

Evaluation

Evaluation of XAI systems separates properties that are often conflated in ordinary use. Fidelity concerns whether an explanation correctly represents the target model. Comprehensibility concerns whether the intended audience can understand the representation. Usefulness concerns whether that understanding supports the task for which the explanation was requested. These properties require different evidence and cannot be reduced to a single universal score.

Quantitative fidelity tests compare an explanatory representation with observed model behavior. A feature-ranking method can be tested by removing or altering highly ranked features and measuring the resulting prediction change. Such tests are sensitive to whether altered inputs remain within the data distribution, since unrealistic perturbations can produce behavior unrelated to ordinary model use.

Stability measures whether similar inputs with similar predictions receive materially similar explanations. Perfect invariance is not a general requirement because a small input change can legitimately cross a learned decision boundary or activate a different internal mechanism. Stability analysis instead examines whether explanatory variation corresponds to meaningful variation in the model and the represented problem.

Human-subject evaluation examines how people interpret and use explanations. Common outcomes include the accuracy of a reader’s predictions about model behavior, the detection of model errors, and the calibration of reliance on automated output. Self-reported satisfaction measures a different construct because an explanation can appear clear while misrepresenting the model. Conversely, a faithful explanation can be difficult to understand when the underlying computation is itself irregular.

Benchmarks for explainability face a problem of reference standards. Synthetic tasks can provide known decision rules, but they may not reproduce the ambiguity of deployed systems. Real applications provide representative complexity, although the correct explanatory account may be unavailable. Evaluation frameworks therefore define the target of explanation before selecting a metric, rather than treating any concise account as evidence of understanding.

Limitations and governance

An explanation can disclose model behavior without establishing that the behavior is acceptable. A system may transparently rely on a variable that acts as a proxy for a protected characteristic, making the dependence easier to identify but not altering it. Explainability and algorithmic fairness consequently remain analytically separate, even when explanatory tools are used during fairness audits.

Post-hoc explanations can also create an additional error surface. The predictive model may produce one output while the explanatory model supplies a simplified account that omits the decisive interaction. A reader can then form an accurate understanding of the surrogate and an inaccurate understanding of the operative system. This discrepancy is especially significant when explanations are displayed as definitive causal narratives.

Disclosure has consequences for privacy and security. Detailed explanations can reveal sensitive training information, expose decision thresholds, or support strategic manipulation of a classifier. Restricting detail can reduce those effects while also limiting independent scrutiny. The relevant issue is therefore not whether explanation is uniformly beneficial, but what information a particular explanatory channel communicates and to whom.

Legal uses of explanation differ across jurisdictions and institutional settings. Data-protection law, administrative law, consumer regulation, and sector-specific rules define distinct obligations concerning notice, reasons, review, and contestability. A technical explanation does not automatically satisfy these obligations because legal reasons concern authority and procedure as well as computational mechanism.