Skip to content
Novandri.

Model Explainability: SHAP, LIME, and Interpreting Black Boxes

Post-hoc explanation is not explanation, and the strongest critique is mathematical. Read what SHAP and LIME actually compute and when to trust them.

Rachmadwipa Novandri9 min read

"Explainable AI" became a category when practitioners discovered that a black box's output could not be defended to a customer, a regulator, or a colleague. The category filled with methods that produce a plausible-looking attribution for any prediction. The interesting question is not which method is best but what any of them can legitimately claim — and the answer is narrower than the marketing.

Direct answer: Post-hoc explanation methods such as LIME and SHAP estimate how features contributed to a particular prediction, under assumptions that are rarely stated and rarely hold. SHAP derives its values from Shapley values, which gives it a principled definition of feature contribution and a well-specified set of conditions under which that definition is the right one. LIME fits a local surrogate, which is fast and flexible but approximate in a way that is easy to under-estimate. And the strongest argument in the literature is not that these methods are unusable — it is that in high-stakes settings, designing an interpretable model should be preferred over explaining an opaque one.

Post-hoc explanation methods such as LIME and SHAP estimate how features contributed to a particular prediction, under assumptions that are rarely stated and rarely hold.

A four-step flow: pick a prediction, choose the method, state assumptions, then report the approximation.
Conceptual: an attribution for one prediction, under assumptions. Every SHAP number is an approximation whose quality is rarely reported.

The tension the framework names

Lundberg and Lee open with the problem statement. "Understanding why a model makes a certain prediction can be as crucial as the prediction's accuracy in many applications." But they note that "the highest accuracy for large modern datasets is often achieved by complex models that even experts struggle to interpret, such as ensemble or deep learning models, creating a tension between accuracy and interpretability."

Their contribution is to make the field less ad hoc. The stated problem is that "various methods have recently been proposed to help users interpret the predictions of complex models, but it is often unclear how these methods are related and when one method is preferable over another". SHAP's answer is a unified framework: it "assigns each feature an importance value for a particular prediction", with "the identification of a new class of additive feature importance measures" and a theoretical grounding.

The Shapley connection is the substantive part, and it is worth understanding what it buys. Shapley values come from cooperative game theory: a prediction is treated as a value produced by a set of features, and each feature's contribution is the average marginal effect of including it across all possible orderings of the coalition. That definition has properties — consistency, missingness, local accuracy — that most ad-hoc attributions do not, and the SHAP paper's contribution is showing that a specific class of additive feature-importance measures can satisfy all of them simultaneously.

That is a real theoretical advance, and it is also exactly where the limitations live: the definition is sound, but computing it is exponential in the number of features. The practical implementations approximate.

What LIME actually does, and why it is fast

LIME takes the opposite approach. Ribeiro, Singh and Guestrin describe the problem it solves: to explain a single prediction, "fit a local interpretable model (surrogate) approximating the black-box model's behavior locally". The surrogate is weighted so that the points near the prediction of interest dominate, and it is fitted to the model's outputs rather than to the labels.

The advantage is speed and flexibility: LIME works with any black-box model, including ones with no internal structure, and it needs no retraining. The disadvantage is inherent to the approach — a local linear approximation is an approximation, and the quality of the explanation depends on a neighbourhood choice and a sampling procedure that are usually hyperparameters with defaults nobody validated for their data.

The honest way to compare the two is not "SHAP is more principled so use SHAP". It is: SHAP's quantity is defined and the approximation error can be measured; LIME's surrogate is local and the error is harder to bound. If you need a defensible number, that asymmetry matters. If you need an explanation for a model you cannot modify, LIME's flexibility may still win.

The critique that should change your design decisions

Rudin makes the strongest case in the literature, and it is not a critique of any particular method. She argues that "trying to explain black box models, rather than creating models that are interpretable in the first place, is likely to perpetuate bad practices and can potentially cause catastrophic harm to society". Her paper "clarifies the chasm between explaining black boxes and using inherently interpretable models, outlines several key reasons why explainable black boxes should be avoided in high-stakes decisions, identifies challenges to interpretable machine learning, and provides several example applications where interpretable models could potentially replace black box models in criminal justice, healthcare, and computer vision".

The argument's force is in the last clause. She is not saying interpretable models are always as accurate. She is saying that the accuracy gap is frequently smaller than assumed, and that where it is not, the decision should be made explicitly rather than by default.

There is also a mathematical objection that has become standard. Zhang et al. observe that "model explanations based on pure observational data cannot compute the effects of features reliably, due to their inability to estimate how each factor alteration could affect the rest". A feature attribution computed from a static dataset answers "which features co-varyred with this prediction", which is not the same question as "which features caused this prediction". For a decision where you intend to act on the explanation — change the feature and expect the outcome to change — the distinction is not academic.

Choosing an explanation method, and choosing a model

A workable decision order:

  1. Can you use an interpretable model at all? If the task permits it — and for many tabular tasks it does — that dominates any explanation technique. Start there and measure the accuracy you give up.
  2. If not, decide whether you need a local or a global explanation. A local explanation answers "why this prediction"; a global one answers "what does this model use". These need different methods, and a SHAP summary plot is not a global explanation.
  3. If you need local attributions, use a method with a defined target quantity, and check the approximation. SHAP gives you Shapley values under stated assumptions; LIME gives you a local fit. Know which one you are quoting.
  4. State the assumptions when you report. The most common misuse is reporting a number with no statement of what it approximates and how well.
  5. Test the explanation against something you can check. Where possible, perturb the input in the way the explanation says matters and confirm the model's output moves as predicted. An explanation you cannot test is a story.
  6. For high-stakes decisions, treat the black box itself as the finding. If the explanation cannot support a defensible decision, the answer is a different model, not a better attribution.

One practical warning about any of these tools: explanations are computed on the data you have, so they inherit its pathologies, and a model that learned a spurious correlation will produce a confident explanation pointing at the spurious feature. An explanation is not a validation.

What to do with the explanation

The most valuable use of a local explanation is not the report; it is the debugging. An attribution that says a prediction was driven by a token that turned out to be a formatting artefact, a postcode, or a null value tells you where the bug is. That is a genuinely better use of explainability than reassuring a stakeholder, and it is the use that survives contact with an incident.

For the reverse direction — building a model that is explainable by construction rather than by annotation — see Data Strategy for the organisational context, and The Limits of AI for why the surrounding process matters as much as the technique.

A worked example

The following is hypothetical and illustrative. Consider a team building a model to flag loan applications, where a reviewer asks why one application was declined. Three checks follow the decision order. First, the team tests whether an inherently interpretable model can do the tabular task at all, measuring the accuracy given up rather than assuming a gap. Second, because a local explanation is needed, it uses SHAP for its defined Shapley-value target and reports the approximation, rather than quoting a LIME surrogate whose neighbourhood was never validated. Third, the explanation is tested against something checkable: the input is perturbed in the direction the attribution says matters, and the model's output is confirmed to move. In this imagined case the attribution points at a formatting artefact, exposing a bug rather than reassuring the reviewer, and the team recognises that a co-varying feature is not a cause it can act on. For a high-stakes decision, the black box itself becomes the finding and an interpretable model replaces it.

A counterargument

The article leans on Rudin's call for interpretable models in high-stakes settings, but a critic can note the claim outruns its evidence: the accuracy gap is called "frequently smaller than assumed," yet the source's examples come from specific domains, not a general proof. The piece also recommends testing an explanation by perturbing the input, yet Zhang et al.'s objection - that observational data cannot estimate the effect of altering a feature - applies to that test too, since a perturbation can move the input off the manifold where the explanation held. Having endorsed explanation as a debugging tool at the end, the article never reconciles that use with its warning that explanations inherit the data's pathologies.

The objection is that it scrutinises SHAP and LIME but not the interpretable-model alternative it recommends.

References & Further Reading