What Vulnerability Detectors Actually Learn

a code model that flags vulnerable functions might be reading the property that makes them unsafe, or it might be reading tokens that happen to correlate with the label. the pair of a vulnerable function and its own fix is the test that tells those apart, because the two versions differ only by the patch, so length, identifiers, api surface and idiom are the same on both sides and anything above chance has to come from the change itself. Risse and Bohme built that test (vulnpatchpairs, 26k c functions paired through their fix commits) and measured detectors at roughly chance, with the accuracy an augmentation buys in-distribution falling 30.2 percent, and f1 77.5 percent, once the test perturbation differs from the one seen in training (Risse and Bohme 2024). Ding et al. gave it a number: over 5,480 pairs that share at least 80 percent of their text, count a pair correct only when both sides are classified right, and gpt-4 with chain of thought reaches 12.94 percent against a 22.70 percent baseline for guessing, class-aware contrastive learning takes unixcoder from 1.60 to 2.30 percent, and the fine-tuned open models sit at one or two percent, mostly by assigning both sides of a pair the same label (Ding et al. 2025).

a pair score near chance does not tell you which failure produced it, and several are consistent with the same number. the information that separates the two sides might not be in the input at all, so the function on its own does not fix the label. it might be in the input and simply not learned. it might be learned and decodable in the activations yet outside the part of them the model reads its verdict from. or it might be there and tangled up with surface features tightly enough that the model cannot use it. those four want four different responses, and the loss curve cannot choose between them. you have to look at the activations, which is the thing mechanistic interpretability does.

figure 1. a pair of a vulnerable function and its patch goes to a model, whose verdict separates them at close to chance. that single number is consistent with several internal states, and each interpretability tool rules one out. a probe asks whether the distinction is decodable from the activations at all; a bag-of-tokens baseline asks whether what the probe found is only the surface difference between the two versions; a jacobian lens asks whether the distinction lies in the subspace the model can report from; activation patching asks whether inserting it changes the verdict, which is the difference between a representation that is present and one that is used.

the primitives

a transformer keeps a running vector per token, the residual stream. writing h^L for that vector after layer L, each block computes h^L = h^(L-1) + attn(h^(L-1)) + mlp(...), reading a linear view of it and adding its output back into the same space (Elhage et al. 2021). the intermediate state of the computation lives there, not in the token that comes out. the working assumption about that space is the linear representation hypothesis: a feature is a direction whose coefficient says how present it is, so you read it off by projection and change it by addition (Park et al. 2023; Mikolov et al. 2013), which is what lets both probing and steering work at all.

a probe is the cheap way to ask whether a property is there. you fit a linear classifier to minimise a loss over the activations and read its held-out accuracy, and if it succeeds the property is linearly decodable. decodable is not the same as used, and this is where most probing claims go wrong: the accuracy is an upper bound on what some linear readout could pull out, and not evidence the model itself pulls it out. Hewitt and Liang's selectivity, the real-label accuracy minus the accuracy of the same probe on a random-label control task, is the minimum needed to rule out a probe that has only memorised, and in this setting a bag-of-tokens classifier on the raw text is that same guard by another name (Alain and Bengio 2017; Hewitt and Liang 2019; Belinkov 2022).

you cannot read the model one neuron at a time because of superposition. under sparsity a layer of width d can hold many more than d features as an overcomplete set of nearly orthogonal directions and absorb the interference, so a single neuron answers to several unrelated things and the feature you want is mixed in with others (Elhage et al. 2022). sparse autoencoders are the current way around it: you learn a sparse code z = ReLU(W_e h + b_e) with reconstruction h_hat = W_d z, minimising the reconstruction error plus an L1 penalty on z, so the columns of W_d approximate the underlying directions and z is their sparse activation, and you judge the result by its L0, the mean number of features that fire, and by how much of the variance or cross-entropy the reconstruction recovers (Bricken et al. 2023; Cunningham et al. 2023; Gao et al. 2024), with dictionaries of this kind now trained on production and open models (Templeton et al. 2024; Lieberum et al. 2024).

none of that settles whether a feature is used, and the test that does is activation patching, causal mediation applied to a network. you run the model on a clean input and a corrupted one, cache the activations, move one component's activation from one run into the other, and read how a target quantity such as the logit difference changes (Vig et al. 2020; Meng et al. 2022; Wang et al. 2022; Heimersheim and Nanda 2024). it has a failure mode worth taking seriously: a subspace can carry a signal that patching moves without that subspace being the mechanism the model actually uses, so a patching result needs controls of its own (Makelov et al. 2023). the logit lens and the tuned lens, which decode the intermediate stream through the output head, tell you the depth at which the prediction is already fixed (nostalgebraist 2020; Belrose et al. 2023). a jacobian-based refinement of that readout, which corrects for how the coordinate frame shifts between layers, additionally picks out the subspace of representations a model can report from and separates it from the processing that never surfaces (Gurnee et al. 2026).

what it buys

the use is diagnostic. a linear probe will often recover the vulnerable-versus-patched label from a model's mid-layer activations well above what the model's own verdict manages, and on its own that reads as an encoded-but-unread gap, the distinction present in the representation but never surfaced. the control that decides it is a surface baseline. a bag-of-tokens classifier on the raw text tends to recover most of the same signal, because the two sides of a pair differ only by the tokens the fix touches and those are already in plain view, and the word-count classifier that matches most of the state of the art makes the same point for the whole task (Risse, Liu and Bohme 2025). a probe reported without that baseline claims a semantic representation that a bag of words has too, which is why the baseline is not optional and why a probe result is a hypothesis rather than a finding (Hewitt and Liang 2019).

the experiments that follow are specific. you can train a sparse autoencoder on a code model and look for a feature that fires on a particular kind of bug, the dereference in an out-of-bounds read or the use of a freed pointer, then check whether that feature activates differently on a function and on its patch. you can take the activation that separates the vulnerable side, patch it into the patched side on the verdict logit, and watch whether the verdict flips, which is the causal test that training on the verdict never performs and the one that would show a distinguishing representation is actually used. and a rewrite ladder, every rung recompiled and re-run so the fault is confirmed to still fire, lets you read a probe across increasing obfuscation: a probe that leaned on surface should decay as identifiers and control flow go, a probe that found structure should hold, as long as the scaffolding the rewrite does not touch is controlled for.

the input may not hold the answer

there is a fact that limits all of this, and it belongs before the experiments because it may make some of them pointless. Risse, Liu and Bohme show that a classifier over word counts alone lands within 4.4 to 10 percent of the state of the art (62.2 percent f1 on devign, 86 percent on bigvul), that labels taken from fix commits are correct for only 39 to 65 percent of the functions they mark, and that every one of the 151 vulnerable functions they audited by hand needed context outside the function body to be judged (Risse, Liu and Bohme 2025). if the label is not determined by the function, there is no stable representation of it for a probe to find, and the honest form of the experiment first gives the model the thing that does determine the label, an execution-verified pair together with the path that reaches the fault, before asking what the activations contain.

that gap is also the opening. vulnllm-r is built partly against this wall and does hold execution-grounded data: on its arvo subset it runs a proof-of-concept, takes the crash, and keeps the pre-fix and post-fix functions with the stack-trace context (Nie et al. 2025). it then scores arvo as project-level recall, counting a vulnerability as found if at least one of its functions is flagged, and its headline f1 is measured over a benign set that mixes the patched twins in with unrelated standalone-benign functions. section 4.3 says plainly that they tried using only patched functions as the negatives, which is the pair setup, and dropped it because the patch differs by a line or two and is sometimes not even a real fix, so the model learned shortcuts. an execution-grounded pair scored pairwise, and then probed, is a thing nobody has done.

the open questions are concrete. with the lexical features and the label noise controlled, is there any non-surface signal left in the pair, or does it go to the floor. do sparse-autoencoder features for particular bug classes exist in code models at all, and if they do, are they causal in the verdict or only correlated. can a causal edit, patching or steering, move the pair separation that fine-tuning and reinforcement could not, and does whatever it moves survive on projects the edit never saw. is the failure an artefact of granularity, so that feeding inter-procedural or execution context produces a representation a probe can find and patching can show in use. when a property is decodable but never spoken, is that because it sits outside the reportable subspace the model reads its answer from, the workspace a jacobian lens identifies (Gurnee et al. 2026), and can that subspace be widened by training. that test also separates a real readout gap from the duller case where a probe only read the surface diff. does a verified rewrite ladder actually separate a surface probe from a structural one once the scaffolding is held fixed. and can any of it be run on code the model has not already seen, so that a probe or a flipped verdict measures reasoning and not recall of a public cve.

interpretability will not make the model pass this test. what it does is tell you which of the four failures you are looking at, and that is worth more than another training run against a wall the literature has already documented, because the four point at four different pieces of work: the data and the oracle that labels it, the readout, the disentangling of the feature, or giving up on the function as the unit. the discipline it forces is simple. a probe result is a hypothesis until its baseline has been run, and whether the model uses what it found is settled by intervening rather than by decoding.

references

Alain, G. and Bengio, Y. (2017). Understanding intermediate layers using linear classifier probes. ICLR Workshop. https://arxiv.org/abs/1610.01644

Belinkov, Y. (2022). Probing classifiers: promises, shortcomings, and advances. Computational Linguistics. https://arxiv.org/abs/2102.12452

Belrose, N., Furman, Z., Smith, L. et al. (2023). Eliciting latent predictions from transformers with the tuned lens. arXiv:2303.08112. https://arxiv.org/abs/2303.08112

Bricken, T., Templeton, A., Batson, J. et al. (2023). Towards monosemanticity: decomposing language models with dictionary learning. Transformer Circuits Thread. https://transformer-circuits.pub/2023/monosemantic-features/index.html

Cunningham, H., Ewart, A., Riggs, L., Huben, R. and Sharkey, L. (2023). Sparse autoencoders find highly interpretable features in language models. arXiv:2309.08600. https://arxiv.org/abs/2309.08600

Ding, Y., Fu, Y., Ibrahim, O. et al. (2025). Vulnerability detection with code language models: how far are we? ICSE. https://arxiv.org/abs/2403.18624

Elhage, N., Nanda, N., Olsson, C. et al. (2021). A mathematical framework for transformer circuits. Transformer Circuits Thread. https://transformer-circuits.pub/2021/framework/index.html

Elhage, N., Hume, T., Olsson, C. et al. (2022). Toy models of superposition. Transformer Circuits Thread. https://transformer-circuits.pub/2022/toy_model/index.html

Gao, L., la Tour, T. D., Tillman, H. et al. (2024). Scaling and evaluating sparse autoencoders. arXiv:2406.04093. https://arxiv.org/abs/2406.04093

Gurnee, W., Sofroniew, N., Pearce, A. et al. (2026). Verbalizable representations form a global workspace in language models. Transformer Circuits Thread. https://transformer-circuits.pub/2026/workspace/index.html

Heimersheim, S. and Nanda, N. (2024). How to use and interpret activation patching. arXiv:2404.15255. https://arxiv.org/abs/2404.15255

Hewitt, J. and Liang, P. (2019). Designing and interpreting probes with control tasks. EMNLP. https://arxiv.org/abs/1909.03368

Lieberum, T., Rajamanoharan, S., Conmy, A. et al. (2024). Gemma Scope: open sparse autoencoders everywhere all at once on Gemma 2. arXiv:2408.05147. https://arxiv.org/abs/2408.05147

Makelov, A., Lange, G. and Nanda, N. (2023). Is this the subspace you are looking for? An interpretability illusion for subspace activation patching. arXiv:2311.17030. https://arxiv.org/abs/2311.17030

Meng, K., Bau, D., Andonian, A. and Belinkov, Y. (2022). Locating and editing factual associations in GPT. NeurIPS. https://arxiv.org/abs/2202.05262

Mikolov, T., Yih, W. and Zweig, G. (2013). Linguistic regularities in continuous space word representations. NAACL-HLT. https://aclanthology.org/N13-1090

Nie, Y., Li, H., Guo, C. et al. (2025). VulnLLM-R: specialized reasoning LLM with agent scaffold for vulnerability detection. arXiv:2512.07533. https://arxiv.org/abs/2512.07533

nostalgebraist (2020). Interpreting GPT: the logit lens. LessWrong. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

Park, K., Choe, Y. J. and Veitch, V. (2023). The linear representation hypothesis and the geometry of large language models. arXiv:2311.03658. https://arxiv.org/abs/2311.03658

Risse, N. and Bohme, M. (2024). Uncovering the limits of machine learning for automatic vulnerability detection. USENIX Security Symposium. https://arxiv.org/abs/2306.17193

Risse, N., Liu, J. and Bohme, M. (2025). Top score on the wrong exam: on benchmarking in machine learning for vulnerability detection. arXiv:2408.12986. https://arxiv.org/abs/2408.12986

Templeton, A., Conerly, T., Marcus, J. et al. (2024). Scaling monosemanticity: extracting interpretable features from Claude 3 Sonnet. Transformer Circuits Thread. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html

Vig, J., Gehrmann, S., Belinkov, Y. et al. (2020). Investigating gender bias in language models using causal mediation analysis. NeurIPS. https://arxiv.org/abs/2004.12265

Wang, K., Variengien, A., Conmy, A., Shlegeris, B. and Steinhardt, J. (2022). Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. arXiv:2211.00593. https://arxiv.org/abs/2211.00593