Reads the code but not the vulnerability

figure 1. the check on one pair. one 8B model is used twice: it reads a function as text with its adapter off and its layer-18 activations are cached, then those activations are injected back into the same model wearing a released adapter, which answers, about the activations rather than the text, whether the code is vulnerable. the yes/no verdict is read from the answer-token logits, and a pair counts as separated when the vulnerable side outscores its patch. nothing is fitted to the pairs. the finding is that the oracle reads the code and the fix off the activations while its vulnerability verdict stays a near-constant prior, one that is a property of the injection channel, not of the adapter.

a model's yes-or-no verdict cannot tell a vulnerable function from its own fix. detectors score near chance on such pairs, and the accuracy augmentation buys in-distribution disappears under transformations not seen in training (Risse and Bohme 2024); gpt-4 with chain of thought reaches 12.94 percent pair-correct against a 22.70 percent guessing baseline (Ding et al. 2025); a classifier over word counts lands within a few points of the state of the art, and the function alone often does not even determine the label (Risse, Liu and Bohme 2025). it is tempting to read this as the model being unable to represent the distinction. recent work says the failure is better described as a prior. models carry a stable directional lean on this task, some toward calling code vulnerable and some toward calling it safe, and fine-tuning mostly moves the threshold of that lean without changing what the model comprehends (Zibaeirad and Vieira 2026). the lean is roughly a coin flip across models and it is not a general property: on neutral yes-or-no questions models tend to answer no (Braun 2025), and the over-flagging seen specifically on vulnerability is traced to surface-cue shortcuts rather than to a safety instinct (Dahiya et al. 2026; Chen et al. 2026).

so the question worth asking is not only whether a reader can separate the pair but what the verdict is actually reading. a linear probe fitted to the model's mid-layer activations separates the two sides at 0.823 on held-out projects, where the model's own verdict is at chance, and a bag-of-words classifier on the raw text reaches 0.764, so the distinction is present in the activations but most of the linearly-decodable part is the visible token diff. the probe is fitted, on 1,490 labeled pairs, and a fitted classifier learns whatever separates its classes, semantic or surface, so its number needs a control beside it and is not evidence the model uses anything (Hewitt and Liang 2019). an activation oracle is fitted to none of these pairs, which is why it is the instrument for asking what an unfitted reader of the same activations does with them.

what an activation oracle is

an activation oracle (Karvonen et al. 2025) is a language model that takes another model's residual-stream activations as part of its input and answers natural-language questions about them. placeholder tokens in the prompt mark where the activations go; each injected vector is normalized, rescaled to the norm of the oracle's own residual at that position, and added into the stream at an early layer, with the source layer named in the prompt. the oracle is the target model itself with a small trained adapter, so the two share weights and the space the activations live in. a residual-stream vector is a point in the specific space that model reads and writes at every layer, and under the linear representation hypothesis its features are directions the model already acts on (Park et al. 2024), so the oracle inherits the machinery for free. the training, following latentqa (Pan et al. 2024), adds only the habit of treating an injected vector as something to describe, on generic tasks, none of them the ones it is later tested on.

it earns its use by generalizing past those training tasks. Karvonen et al. finetune a target to hint at a secret word without ever stating it, then recover the word from the target's activations alone, beating black-box prompting on two of three such tasks; because the word never appears in the output, recovering it shows the oracle reads internal state rather than paraphrasing text. the same work is clear that oracles also confabulate, which is why later work measures when their answers hold up (Bauer et al. 2026; Torrielli et al. 2026), and why the check below reads scored yes/no answers with controls rather than trusting prose. released adapters cover a dozen models, among them the 8b one the probe above comes from.

setup

the data is vulnerable/patched function pairs anchored to public fix commits, the vulnerable side confirmed by execution: it faults under asan or ubsan on a triggering input, the patched side runs clean. 1,795 pairs over 191 projects, split by project; layer, window, and question wording fixed on 300 training-side pairs, then 305 held-out pairs scored once. P(yes) is read from the yes/no answer-token logits rather than the oracle's stated confidence, which is not trustworthy (Torrielli et al. 2026); the model's actual first token is one of those two words on every pair and agrees with the logit read, so the verdict is where it is scored, not a tail artifact. injection works: asked what language the code is in, the oracle answers c for the libssh2 functions and c++ for the harfbuzz ones from the activations alone, and it gives a plausible one-line summary of what each function does.

the verdict is a prior, and it is the wrong one

on the held-out pairs the oracle scores paired accuracy 0.439. its verdict is not undecided, it is collapsed: asked yes or no, it answers no to nearly every function, calling the vulnerable side vulnerable 9 percent of the time and clearing the patched side 86 percent, so pairs come out both-cleared and neither side is ordered above the other. this is the mirror image of the same model reading the code as text, which comes out at 0.439 as well but by the opposite collapse: yes on nearly everything, recall 0.948, specificity 0.039. two readers of the same functions, two constant verdicts, opposite directions, both failing the pair. the fitted readers on the identical pairs separate it, tf-idf 0.764 and the probe 0.823.

figure 2. paired accuracy on the same 305 held-out pairs, with 95 percent bootstrap intervals. the two unfitted readers, the model given the code as text and the oracle given its activations, sit at chance; the two fitted readers separate the pairs, and most of what the probe finds a bag of tokens finds too.

two constant verdicts pointing opposite ways is exactly the directional-prior picture: a skeptical lean and a paranoid lean, the two poles of the spread that dominates model accuracy on this task (Zibaeirad and Vieira 2026). one caution before leaning on that comparison. the directional-prior work is measured with surrounding code context preserved, and our reader sees the function alone, which is the axis the pair problem is most sensitive to, so the two sit in different regimes and the alignment is in direction, not in numbers. what our pair adds is that the split is not about the model at all here: it is one set of weights, and which pole it lands on is set by whether the code arrives as text or as activations.

figure 3. the same base weights, on the same 305 pairs, in three conditions. reading the code as text, with or without the adapter, the model over-flags: it calls almost everything vulnerable. only when the code arrives as injected activations does the verdict collapse the other way, to almost always safe. so the always-safe verdict is a property of the activation channel, not of the adapter.

but the reading is intact

the collapse is not blindness. three things show the oracle reads the code and the fix off the same activations its verdict ignores.

figure 4. the oracle's yes-rate against the true rate for eight yes/no questions with ground truth computed from the code, over 200 functions. it tracks the true prevalence, from features present in almost every function to ones present in almost none; the one outlier is a memory-allocation cue it over-fires on. vulnerability is the property it does not track: its yes-rate barely differs between code that was and was not vulnerable.

first, the readout is calibrated on features it can read. asked eight yes/no questions with ground truth taken from the code, the oracle's yes-rate follows the true prevalence: about 0.98 where the answer is almost always yes, about 0.03 where it is almost always no, tracking the middle. the one exception is a memory-allocation cue it over-fires on, the same surface-trigger behavior seen in prompted detectors, and it does not spill into the verdict. against that backdrop vulnerability is the odd property out: the oracle's yes-rate on functions that were vulnerable is 0.19 against 0.14 on functions that never were, an AUROC of 0.54, barely above chance. so structural facts come through the channel; the vulnerability judgment does not.

second, its free text is side-sensitive and on the libheif pairs names the change. asked what is wrong, the vulnerable side gets the bug, a 32-bit read used as 64-bit, an unchecked tile index, and the patched side gets what the fix put in, the 0xFFFFFFFF value now rejected, the tile index now tested against bounds. it is not uniform, other pairs differ only in the guessed consequence or get the same sentence on both sides, and it confabulates elsewhere, but where the prose moves it moves toward the fix. third, a different scored question separates the pair where the vulnerability question does not: asked whether the code ends with a bounds or null check, the oracle scores the patched side higher, within-pair P(yes) difference -0.053 against 0.000 for the vulnerability question on the same windows. it reads that the fix added a check; it will not conclude the fixed version is safe.

figure 5. within-pair P(yes) difference, the vulnerable minus the patched side, for two questions over the same injected windows. the vulnerability question puts the two sides level; asked instead whether the code ends with a bounds or null check, the oracle scores the patched side higher, where the fix put one. it reads the added check off the activations without reading vulnerability.

it is the channel, not the window, not the adapter

two controls rule out the easy explanations. the window is not the problem: injecting every token position of the whole function, the paper's own default mode and a span its adapter generalizes to, leaves the verdict where it was, still always-no, if anything a touch worse on the pair. and text inversion does not rescue it. the oracle can partially reconstruct the source from the full-sequence activations, on more than half the functions with substantial overlap, so it has recoverable text to work from, yet it does not then behave like the text reader and over-flag. recoverable code is available and the verdict still defaults to safe.

the adapter is not the cause either, and this is the control that names the mechanism. run the same adapter on the code as text rather than as activations and it over-flags exactly like the base model, recall 0.65 against the base's 0.63. the always-safe verdict appears only through the activation channel; hand the adapter the text and its lean flips to the paranoid pole. so the oracle is not a more honest reader that the latentqa training taught to resist over-flagging. the model over-flags vulnerability whenever it can read the code as text, and the activation channel gates that judgment out. injecting the real activations moves the verdict essentially not at all from injecting nothing, 0.25 against a no-injection 0.27, while the same channel carries the structural features cleanly. the channel is selective: features pass, the vulnerability judgment does not, and the verdict falls to its base-rate default, no.

what it means

the tidy version, that an unfitted reader confirms the distinction is encoded but unused, does not survive its own controls. what the ladder actually shows is narrower and, to us, more interesting. the distinction is weakly present in the activations, mostly as surface. an unfitted reader of those activations recovers the code, tracks concrete features by their true rate, and describes the fix in prose, so its reading is intact. its vulnerability verdict is nonetheless a constant, and that constant is the base-rate default of a channel that carries structural facts but not this particular judgment, a default the same weights abandon the moment the code arrives as text instead. so the failure sits at the verdict, and specifically at what the injection channel lets reach it.

this lines up with, and sharpens, two threads in the literature. prompted coder models already show the gap between representation and output: linear probes read secure-versus-vulnerable far better than prompting, and the models lean secure (Yu et al. 2025); we add the oracle modality and locate the leak, the channel gates the judgment out of the verbalized verdict. and the directional-prior account (Zibaeirad and Vieira 2026) gets a cleaner demonstration than a cross-model survey can give, because here the two directions come from one set of weights under a change of input modality, not from different models.

the honest limit is the one the pair literature has always pointed at. our functions are seen alone, and the vulnerability of a function often is not determined by its body (Risse, Liu and Bohme 2025). so a verdict that will not commit can be a channel failing to carry the judgment or an input that does not contain it, and our controls cannot fully separate the two. both readings put the failure at the verdict rather than the representation, but telling them apart needs the callers and the reachable dataflow in the input, which a function-level setup does not have. that, more than a stronger oracle or more injected tokens, is the next thing to change.

limits

one model at 8b, one released adapter; a stronger multi-layer oracle exists (Bauer et al. 2026) and has not been run. the probe reads a mean-pooled whole function and the oracle a windowed or full injection, so those numbers compare instruments as configured, not two readers of one object. the vulnerable functions are public cves and plausibly memorized, which should raise an unfitted reader's score rather than lower it. the corpus carries semantics-preserving rewrites, each recompiled and re-run so the same fault still fires, which puts the same questions to code that cannot have been memorized and is the measurement to run next, alongside adding context to the input.

references

Bauer, J., De Schamphelaere, C., Karvonen, A., Luick, N. and Nanda, N. (2026). Building better activation oracles. arXiv:2606.02609. https://arxiv.org/abs/2606.02609

Braun, D. (2025). Acquiescence bias in large language models. Findings of the ACL: EMNLP. https://arxiv.org/abs/2509.08480

Chen, M., Wang, L., Qin, Y., Wang, Y., Wu, B. and Liu, D. (2026). How code representation shapes false-positive dynamics in cross-language LLM vulnerability detection. arXiv:2604.27714. https://arxiv.org/abs/2604.27714

Dahiya, V., Nehra, S., Dholariya, V., Shangari, B. and Khatri, C. (2026). Are frontier LLMs ready for cybersecurity? evidence for vertical foundation models from dual-mode vulnerability benchmarks. arXiv:2605.23243. https://arxiv.org/abs/2605.23243

Ding, Y., Fu, Y., Ibrahim, O. et al. (2025). Vulnerability detection with code language models: how far are we? ICSE. https://arxiv.org/abs/2403.18624

Hewitt, J. and Liang, P. (2019). Designing and interpreting probes with control tasks. EMNLP. https://arxiv.org/abs/1909.03368

Karvonen, A., Chua, J., Dumas, C. et al. (2025). Activation oracles: training and evaluating LLMs as general-purpose activation explainers. arXiv:2512.15674. https://arxiv.org/abs/2512.15674

Pan, A., Chen, L. and Steinhardt, J. (2024). LatentQA: teaching LLMs to decode activations into natural language. arXiv:2412.08686. https://arxiv.org/abs/2412.08686

Park, K., Choe, Y. J. and Veitch, V. (2024). The linear representation hypothesis and the geometry of large language models. arXiv:2311.03658. https://arxiv.org/abs/2311.03658

Risse, N. and Bohme, M. (2024). Uncovering the limits of machine learning for automatic vulnerability detection. USENIX Security Symposium. https://arxiv.org/abs/2306.17193

Risse, N., Liu, J. and Bohme, M. (2025). Top score on the wrong exam: on benchmarking in machine learning for vulnerability detection. ISSTA. https://arxiv.org/abs/2408.12986

Torrielli, F., Schneider-Kamp, P. and Galke Poech, L. (2026). Confidence and calibration of activation oracles for reliable interpretation of language model internals. arXiv:2605.26045. https://arxiv.org/abs/2605.26045

Yu, W., Mangal, R., Zhuo, T., Fredrikson, M. and Pasareanu, C. S. (2025). A mixture of linear corrections generates secure code. arXiv:2507.09508. https://arxiv.org/abs/2507.09508

Zibaeirad, A. and Vieira, M. (2026). Calibration without comprehension: diagnosing the limits of fine-tuning LLMs for vulnerability detection in systems software. arXiv:2606.20502. https://arxiv.org/abs/2606.20502