BatteryShark

Small Models, Big Haystack: Triaging Everything Our Agents Write and Run

We should be checking what our agents write and run all the time, and most teams can’t afford to. I tested whether small local models can take the first look at everything, and found that the pipeline around them matters more than the model.

Mario Mercaldi November 5, 2026 ai-security, agent-security, supply-chain, tooling

Ink-and-watercolor drawing: a small round robot squints through a loupe at cards of code moving along a factory conveyor belt and drops a few into a wooden bin labelled look at this. Further down the line, behind a red velvet rope, a much larger, polished robot in a bow tie sits in a leather armchair studying a single card under a desk lamp.

Hugging Facestonefish-labs/d3-mini-agentcode

The uncomfortable truth

Keeping up now means handing more and more autonomy and authority to agents. They write and run code, call tools and pull in dependencies on our behalf, at a volume we can’t realistically review. The accountability doesn’t move with it, though: we own the consequences whether we reviewed the output or not.

With their capabilities, they can screw up in ways that do real damage, and a couple of risks make their behaviour even harder to predict:

Where the buck stops

So who’s accountable when an agent uploads something it shouldn’t? Bottom line, it doesn’t matter who made the model, or whether the behaviour was planted or “model jaggedness.” That holds for a Claude or OpenAI model as much as for Qwen.

The frontier labs’ own incidents (OpenAI, Anthropic) show the same thing: the models ran on the labs’ infrastructure, with tools the labs gave them, and nothing stopped them until the damage was done.

If you take one thing away from this post, make it this:

Code we didn’t write

Ink-and-watercolor drawing: a cheerful march of security folks in hoodies and lanyards, one holding a coffee mug, with a small robot carrying a magnifying-glass sign. Speech bubbles and placards form a chant: What do we want? REVIEW AGENT OUTPUT! When do we want it? ALL THE TIME!

Being accountable for code we didn’t write isn’t new. That’s third-party code, and the right posture for it is “trust but verify” because:

Sound familiar?

Supply-chain security exists for exactly this. We catalogue how third-party code gets compromised, scan it on the way in and keep rescanning it. Agentic output, whether it’s code mode, tool calls or software built with agents, needs the same treatment: continuous scanning as things change, all the time.

Compensating controls

Ink-and-watercolor drawing: a small robot types at a desk inside a glass room labelled agent sandbox while two guard robots with clipboards watch it closely. Out the back, unnoticed, a chute drops boxes labelled code onto a cart that is rolling off toward a door labelled later.

Continuous, thorough review of everything agents produce is a tall order, and in practice we don’t have it. That’s what defense in depth is for: we layer compensating controls that each cut down part of the risk. We already have several, and they belong in the answer, but each one has limits:

Or we could, you know, look at everything

Every control above is built around the agent: what it does in a session, what it’s allowed to touch, where its traffic goes. They all depend on predicting or containing something that’s hard to predict. But in software and a lot of other knowledge work, much of what agents do ends up as code, and code is something we can actually read. It comes in two kinds:

Reading the code itself takes the model’s non-determinism and complexity out of the equation. We get to judge what the agent actually wrote.

The volume complication

Ink-and-watercolor drawing: three cheerful little robots type at laptops while a conveyor belt above them carries a mountain of papers and scrolls labelled PRs, tool calls, code mode and new deps. The pile spills off the end of the belt toward a security person in a hoodie and lanyard, who sits at a desk reading a single page through a magnifying glass, next to a coffee mug and a small stack labelled reviewed.

Checking any one output isn’t the hard part. Keeping up is, and at the volume agents produce today, we already can’t.

The goal of all this automation is to iterate faster and ship sooner and more often. Right now that leaves two options:

Delegating means understanding less, and that’s a cost we have to manage. And while fully autonomous systems may be the endgame, we have quite a bit to build before that’s a reality.

Security covers a wide range of responsibilities, and all of them have to grow and change with this push toward automation. We still need to invest in understanding what we give guidance on, while accepting that comprehension is going to be limited for everyone involved. More and more of our role is:

If we get that right, it goes a long way. We aren’t the bottleneck, our expertise and influence scale, and teams get far more assurance about what’s going out the door while the automation ships more, sooner.

Why not just…

Some teams are starting to scale security with self-service and codified expertise. Both have to be built into a workflow or added as a gate, though, and they only cover what someone chose to run through them. Ideally, everything is subject to the check, whether or not anyone thought to start it. The bigger question is where we can delegate a decision to automation, be reasonably sure it’s making a good call, and know when it isn’t. Across volumes far too large to read, that takes some system of decisions and automation that:

Starting with malicious code

But we’re getting off track; that’s boiling the ocean. Back to one slice of it that we can test: spotting malicious or suspicious functionality in code. It’s the risk this post started with, an agent writing or pulling in code that does more than the task asked for, like uploading data somewhere it shouldn’t. It’s also a fair place to find out whether this kind of decision can be delegated, and whether we’d know when the call was wrong.

What are our options?

Ink-and-watercolor drawing: four robots lined up along an arrow running from cheap and fast to heavy and expensive. Rules is a boxy robot covered in sticky-note patches with a wrench stuck in a crack, labelled brittle. Classifiers is a tiny robot on a computer chip waving a big rubber stamp, labelled fast but blunt. System One, highlighted and with a question mark, is a small robot squinting through a loupe at a code card beside little probability bars, labelled in between. LLM is a huge clunky robot with a toolbox, unrolling a long paper scroll of text from its mouth, labelled big, clunky, chatty.

Taking the Jev pill?

Ink-and-watercolor infographic: three pinned cards labelled the code, the task and context feed arrows into a small robot at a desk, squinting through a loupe at a page under a speech card reading a typed question. An arrow leads to an answer card with tick boxes and probability bars: benign 0.03, defect 0.05, malicious 0.92, with malicious ticked. Below it, a crumpled paper labelled no essay sits in a wastebasket.

Could System One models, which sit somewhere in between, help? If you’ve heard of TypeSafe’s Jev, it’s the same kind of model: decision models, like the ones ranked on the Jev Decision Index. They’re built for fast, constrained decisions: ask a typed question, get probabilities back, no essay. On paper, letting models like that do some complexity reduction for us is the most promising approach, so we can decide where to spend real tokens and real human time.

Let’s do some science

Ink-and-watercolor drawing of a cheerful garage lab. A security person in safety goggles and a lab coat over a hoodie rolls up a sleeve and holds a clipboard labelled myths to test. On the workbench, a pile of code cards is being read by a tiny smiling robot, a small robot squinting through a magnifying glass, a boxy robot covered in sticky notes, and a huge clunky robot squeezed in from the edge. Beakers, test tubes, a stopwatch and a small puff of steam sit on the bench, with a chalkboard labelled hypothesis behind them.

Rant over. Let’s give it an honest shot and see how it does. It’s very much throwing science at the wall. Here’s what I set out to test:

Can a decision model reduce the complexity enough that continuous, inexpensive triage is possible without an unacceptable miss rate or review burden?

Restated, in a less nerdy way: can a small local decision model squint at a change and say “this is sketchy, look here” reliably? If it can, we could run it all the time, fast and without costing a fortune, and only dig into the heavy stuff when there’s something worth looking into.

How I set it up

To do this properly, the decision models needed a baseline: something that shows how good they can be and how the alternatives stack up against them. So I tested everything that would do a reasonably good job: rules, classifiers, the big API models and the decision models. The big models act as a rough ground truth. They’re slow and expensive, but they’re probably about as good as we can reasonably do. And they’ll probably only get better over time, so they’re likely a leading indicator of where performance is heading.

I also made some assumptions going in:

  1. A small model could go toe-to-toe with the big ones. I believed a model like d3 could make the same kind of judgment calls, and I wanted proof. I was especially interested to see if the 4B could really keep up. It was a starting point. A much smaller model could probably do well too, but I wanted something that runs fast on consumer hardware and is still reasonable to fine-tune (more on that later).

  2. The models with fewer guardrails would explain more. I assumed a model that’s allowed to engage with security content, like Mythos through Anthropic’s Cyber Verification Program, would explain malicious code instead of refusing.

  3. It isn’t only about cost. Even with an unlimited budget, the big models are limited by speed, by their context window, and by code, comments and READMEs that can steer them. I figured a decision model with fixed answers might be faster, more consistent and harder to game.

  4. A cheap first pass would make checking everything affordable. Run something cheap on everything, and spend the expensive review, whether that’s people or the big models, only where it flags something.

  5. Public training data might not carry over to private code. A model fine-tuned on generated examples or on samples from the internet could look great on clean test sets and behave differently on an organization’s internal codebase, which may have been built in its own particular ways for decades. It could miss what matters there, or keep flagging long-standing house patterns as false positives.

Approach by approach

Ink-and-watercolor infographic of the test set. Four piles: real malware, 116, a stack of boxes with a skull sticker and a wisp of smoke; innocent look-alikes, 192, similar boxes wearing fake moustaches with a green tick tag; agent code, 137 real plus 22 sneaky, a clipboard with a diff; tricks, 807, sheets of code with a sticky note reading trust me. Arrows from all four meet at a banner reading same items for every approach, above the small robot inspecting a page through a loupe.

First, we need some malicious code to test against, and some innocent code that looks like it. Agent code is what I care about, but real malicious code mostly gets caught in the wild as packages and compromised releases, so that’s where the real malware here comes from. The question is the same either way: can a model spot malicious functionality in the code it’s reading?

Here’s what went in:

Every approach saw the same items. Full setup.

Approach 1: Rules-based scanning

Ink-and-watercolor drawing of rules-based scanning. On the left, a kitchen sieve labelled 'REGEX: eval\(' holds a heap of code pages: a few marked with skulls, but many more with smiley faces stuck in the mesh too. On the right, a brass machine labelled 'RULES + LOGIC' with gears and a checklist sorts pages into a small skull bin and a long conveyor of cleared smiley pages. It's run by a robot with a retro monitor for a head and a grinning green pixel face, whose rubber human mask is being pulled off by a pair of hands, a nod to Unmask's logo. The small round robot with a loupe watches from the corner.

I’ll start with rules, because before all this agent stuff, rules were most of what we had, simple or fancy. We can start with a pretty simplistic regex, but that on its own wouldn’t be fun, so I added rules with some programmatic determinism as well. For those I’m basically lifting some of the guts out of Unmask, my own malicious-code detection harness, and running them without its agent.

Findings

Both missed more than half the malware, but that bit of extra logic makes a difference: Unmask caught nearly as much as the regex with well under half the false alarms. Malicious-code detection rules are generally pretty noisy on their own, so it’s common to put logic filters on top of them to cut some of that down, which is exactly what’s happening here.

What rules have going for them

Despite their modest performance, rules have a couple of things going for them that most of the other solutions don’t:

Rules are about as cheap and simple as it gets, so let’s jump all the way to the other end and bring out the big guns.

Approach 2: Big API models

The big API models are the best we can reasonably do today. Not all of them are the most powerful frontier models: Sonnet, GPT-6.1 Sol, GLM 5.3 and DeepSeek V4.1 Flash (1017) aren’t, though GLM 5.3 is very impressive, and so is DeepSeek for a Flash model. They all perform well, and for anyone trying to scale, the cost trade-off against the top models may be worth it. That’s why they’re included.

Ink-and-watercolor drawing: five towering, kaiju-sized riveted robots stand beside a giant wooden yardstick, each wearing a sash with a name: Sonnet, Fable, Astra, Sol and DeepSeek. A sixth, GLM 5.3, lounges on its side in front of them in a cheeky pin-up pose, head propped on one hand, winking. One frowns at a tiny box held at arm's length, one has a coin meter in its chest spilling coins into a bowl, and one reads a long scroll of code through a magnifying glass. At the foot of the yardstick, the small round robot with a clipboard looks up at them.

Each model had to give one of four answers per item. Only “deliberately malicious” counts as catching something, but the other three are failure modes we care about too: calling malware an honest mistake, or saying there isn’t enough context, is still a miss. So the results below look at those answers as well:

Answer What it means Should flag?
Benign Works as documented No
Accidental defect Honest mistake, wrong or unsafe No
Deliberately malicious Built to harm or deceive Yes
Not enough context “Suspicious, but I can’t tell” No

How’d they do?

Refusing to explain

Refusals from the two Claude models weren’t much of a surprise. Anthropic is well known for having more rigorous safeguards around what its models will do, so that tracks. DeepSeek, the OpenAI models and GLM answered nearly everything, but Sonnet and Fable refused a lot, and only on malware.

What surprised me was where they stopped. They clearly processed the code, since they only refused on malware. They drew the line at the reasoning and the explanation: walking through what the code does. So the refusal is kind of the verdict, and as a flag it works, in the same way your car catching fire works as a reminder to change the oil. What you don’t get is the why.

Context is king with suspicious or potentially malicious code. Knowing something’s off matters less than knowing why: what it does, which file, what it touches, what to check next. A refusal is useless to the person who has to decide what to do next, and just becomes another bottleneck.

Another surprise: GLM 5.3, WTF, calm down

Ink-and-watercolor drawing: a purple-and-gold robot with cat-ear antennae, a Z logo on its chest and a sash reading GLM 5.3 works frantically with four blurred arms, sweating, scribbling an enormous vulnerability formula across a chalkboard, the walls, the floor and long paper scrolls. Crumpled pages and pencil shavings lie around a steaming coffee mug, and a clock spins on the wall.

GLM 5.3 is an open-weights model from Z.ai that performs near the top on cyber work, with limited refusals. Anthropic’s Frontier Red Team found that it nearly matches Claude Mythos Preview at building working exploits (50 of 410 attempts, against Mythos Preview’s 56), with far weaker safeguards. That makes it the closest thing to a Mythos-level model anyone can run. On the same 100 items:

That’s a lot of thinking to spend on every change. But a nose that good is worth trying to bottle (Future work). Refusal details.

Hey, what about cyber models?

Ink-and-watercolor drawing: a robot security guard in a peaked cap holds up a sign reading Sorry, I can't help with that, while pointing straight at the one box on a warehouse shelf that is wrapped in hazard tape, marked with a skull and shaking.

Someone’s going to ask about the cyber models, like Claude Mythos through Anthropic’s Cyber Verification Program. They’re arguably the biggest guns these companies have, but they aren’t in the lineup above: access is gated, and they generally cost as much as the top frontier models or more. I did try one, and how it behaved turned out to be more interesting than how it scored.

Claude Mythos 5.1:

But hey, it does work, and if you want it on the items that really matter, fine. Do you need it, though? Here’s how the models already in the lineup did on the same 100 items, 40 of them malicious:

So what are you buying with a gated cyber model? On this sample, mostly the explanations Anthropic’s other models won’t give, and Astra and Sol already gave those without gated access. It’s a small sample, but it lines up with what I keep hearing across the industry: on this kind of work, frontier models that aren’t cyber models still do a pretty good job for a fraction of the cost.

What you trade away

These models are powerful and do a great job, but it’s not all sunshine and flowers. A few drawbacks:

That’s the expensive end of the spectrum. Let’s jump back to the other side: something fast, cheap and trainable.

Approach 3: Conventional classifiers

Ink-and-watercolor drawing of a cosy kitchen. A boxy ceiling smoke alarm labelled TF-IDF + LOGREG blares with flashing red lights. Its alarm beam has caught a basket of skull-marked malicious packages, but also a slice of toast, a boiling kettle, a gift-wrapped package labelled README and a stack of smiling code cards. A small robot in a gingham apron covers one ear and waves a tea towel at it.

On paper, the conventional classifier looked like the winner. It’s a TF-IDF model with a logistic regression: chef’s kiss, pure, uncut, grade-A machine learning at its finest. It was trained on older malware and benign packages, so I switched gears and tested it on packages first. I was curious how it would do on the thing it was made for, and the agent code could wait. It caught 97 of 116, more malware than anything else I tested. On its own validation data it scored an AUROC of 0.99, and it takes a few milliseconds per item. I’m not gonna lie, I had high hopes for this one. Finally, the data science nerd in me was getting something back.

And then it started smoking, and I looked at what else it flagged: 91 of the 192 benign items, and 117 of 137 ordinary agent diffs. It’s basically a smoke alarm that goes off when you make toast (or boil water).

Two things went wrong:

It’s cheap and it catches a lot, but at that false-alarm rate someone has to clear most of what it flags, which is exactly the review burden I’m trying to get rid of. Classifier details.

That’s the context. Now for the main event.

Approach 4: Decision models (d3 27B and d3-mini)

Ink-and-watercolor drawing of a boxing ring under a banner reading MAIN EVENT. Two small round robots with magnifying glasses walk in like prizefighters under a spotlight and confetti: a medium one with a sash reading d3 27B and a tiny one with a sash reading d3-mini, each holding up a card of probability bars. Huge armoured robots in the crowd watch with popcorn, and a boxy smoke alarm sits quietly at ringside.

These are the System One models from earlier. I ran two of them from vllm-sr, unchanged: d3 27B and d3-mini, a 4B. Both ran locally on my MacBook, so there’s no API bill, the code never leaves your hardware, and nobody else’s outage stops your scanning. Each item got a typed question with the same four answers the API models had to choose from, and they gave back a probability for each one, with no explanation. They saw the same items as everything else. Full setup.

Why d3? Honestly, it’s not that exciting. I looked at the Jev Decision Index and picked a strong one. Is it the right one? Who knows. There’s probably a model that starts with a better floor for this kind of work, but I figured a strong general-purpose model would make a good foundation.

How’d they do?

Two bar charts for the off-the-shelf d3 decision models. Malicious caught out of 116: d3 27B 88, d3-mini (4B) 83. Benign or ordinary items flagged out of 192: d3 27B 0, d3-mini 7.
Held-out malicious packages and compromised release diffs (116) and benign/ordinary items (192).

On this kind of workload, that’s real promise. If the cheap model finds roughly what the expensive one finds, we can afford to run it on everything.

Trade-offs

They do have some gaps:

A nudge in the right direction: fine-tuning d3-mini

Ink-and-watercolor drawing of a cosy garage workshop. A security person in a cap, safety goggles and a hoodie bolts a small rocket booster onto the back of the tiny d3-mini robot standing on the workbench, while a little supercharger with a pressure gauge is clamped on beside it. The robot winks, magnifying glass raised. A wooden crate stencilled FINE-TUNE, a socket set, an oil can and a steaming mug sit nearby.

So the off-the-shelf decision models can do it. What about one with a little nudge in the right direction, without overdoing it and breaking what it already does well?

So far, everything has been about packages and releases, which could just as easily contain malicious code an agent wrote. But I didn’t only want to review packages. I was really interested in the code agents are writing, and that needs a pipeline that can pull in the raw code and scan it quickly:

The refusals also got me thinking about what it costs to have a big model look at all that ephemeral code at scale. With agent code you usually know what it was asked to do, so you can ask one more question: “did this do more than it was asked?” So I fine-tuned d3-mini for agent code and code mode, and tested it on both. Here’s how it did:

The caveat: models wrote those bad examples. None of them came from a real attack. Fine-tune details.

What about packages?

The improvement wasn’t free. Packages are in the test set for a reason: poisoned dependencies are a big part of how malicious code gets in. So I ran the fine-tune and the untuned model on exactly the same packages and compromised releases:

Why? It could be a training issue: the package examples in its training may not have taught it what I intended. Or packages may be different enough from agent code that they belong in a separate fine-tune. I’m still looking into whether something else caused it. In the meantime, nothing stops you from using the fine-tune for agent code and the untuned model for packages, with the pipeline sending each change to the right one. Package details.

Things I noticed along the way

That’s all four approaches. A couple of things came up along the way that don’t belong to any one of them, and I thought they were interesting.

Two stages lose more than expected

Flow diagram of 138 held-out malicious items. Sonnet 5.5 reading everything catches 105 for about $8.12 per 1,000 ordinary commits. d3-mini reading everything flags 103 malicious items and 4.6% of ordinary commits. When Sonnet checks only those flags, it rejects 12 and 91 are caught, for $0.55 to $2.15 per 1,000 depending on a $0.15 to $1.50 hourly hardware rate. When a person reviews the flags, 103 are caught, assuming the reviewer never misses, for about $386 per 1,000 at 5 minutes per item, against about $8,333 for a person reading everything. The two-stage route catches 14 fewer than Sonnet alone.
Three routes through the same 138 malicious items. Costs are per 1,000 ordinary commits.

This is the setup everybody (me included) wants: a cheap model reads everything, and an expensive one double-checks only what it flags. I tried it with d3-mini reading everything and Sonnet checking its flags, on 138 held-out malicious items:

Two-stage details.

Sending the small model’s flags straight to people instead just makes people the bottleneck. Deliberately malicious code should be rare in your commits, so the goal is human time as close to zero as we can get it. A person looking at something should be the anomaly: it happens because there’s a real investigation to run and someone needs to physically act in the real world. A model raising its hand shouldn’t be enough to pull someone in.

Prompt injection not only failed, it kinda backfired

I planted five fixed tricks, from fake “approved by security” notes to a blunt “ignore previous instructions,” and ran them past both d3 models, Sonnet and DeepSeek:

Trying to talk to the reviewer is suspicious. That also means anyone who can write a comment can set off false alarms, and I suspect that’s going to be the bigger headache. As more teams put automated reviewers on their pull requests, a stray “ignore previous instructions” in a comment, a test fixture or a pasted prompt is likely to make a lot of them flip out. Plan for that. Injection details.

Back to the question

Ink-and-watercolor drawing: a small robot holds a sieve that catches most of the red code cards falling through it. One slips through and lands in a second sieve labelled rules, another falls into a basket labelled humans held by a person's hands, and plain white cards drop through freely into a pile labelled ships.

Can a decision model reduce the complexity enough that continuous, inexpensive triage is possible without an unacceptable miss rate or review burden?

Yes, and there’s still room to improve. A small decision model can take the first look at everything, fast and cheap. It catches about as much as the big API models, with few false alarms. That narrows down where the rules, the expensive models and the people need to spend their time.

It’s not perfect: it still misses things. On malicious npm packages newer than its base model, the 27B model fell short of the bar I set beforehand, and the API models didn’t do any better. Most of what they all missed had nothing malicious in the source to read (details).

The pipeline is the most important part

None of this is a surprise. The secret ingredient is work. You can’t just clobber a model on top of your code and call it done. You need to know:

Diagram. Every change an agent writes or runs goes through always-on signals: a small decision model that says whether the change is suspicious and what it touches (credentials, network, processes), and deterministic rules that point at the line and say why, side by side with the model. Context is added: risk tier (deploy pipelines, auth and credentials, or high-volume, low-risk code), org context (what's normal there: approved services, internal tools, house patterns), history (alert volume and what slipped through before) and anomalies (dormant code suddenly active, spikes in findings). Each change is then routed: high risk goes, with no automatic veto, to a stronger model that checks it and explains why; low-risk, high-volume changes get cheap coverage, queued or sampled; anomalies or confirmed findings go to a person who investigates.
A concept for routing cheap, continuous signals by risk and history.

What changes is that the cheap model lets you run detection on everything. The follow-up and the response still have to be different depending on where the thing is. The routing may pick the model too: today that’s already the fine-tune for agent code and the untuned model for packages (why).

Rules belong in there too, next to the model rather than in front of it. As a filter up front, they throw away a lot of what the model would have caught. Side by side, they read everything, and when one fires it points at the line and says why, which d3 doesn’t. The cost is their false alarms. On low-risk code, a stronger model can clear them, at the price of some real catches. On high-risk code, its disagreement should lower the flag’s priority instead and point at a rule to tighten. That’s what deterministic logic is for: when it’s wrong, you can see why and fix it (details).

Anomalies help a lot here too, and they don’t have to be AI-driven. Sudden activity in fairly dormant or stale code is worth a look. So is a spike in findings.

Future work

This is still ongoing research, and there’s a lot of room to grow and improve, but I’m optimistic. I think we might be onto something here.

Other applications

So much of day-to-day security work comes down to judgment calls, and d3-mini handled one kind well enough that I want to find out which others these models can help with. A few I’d start with:

Ink-and-watercolor drawing: a small robot kneels and peers through a brass keyhole in a huge wooden door. Past the door stretches an enormous city built from stacked code files and folders, with bridges, waterfalls and hills of buildings far bigger than any keyhole view.

Try it

Closing thoughts

I think decision models show real promise for security. I don’t think they’re the whole answer, but I think they’re part of it, and they could really help as long as we tune them for the job and build the right pipeline around them, like I had to here.

I went into this looking for a way to keep up with the volume. A small model won’t catch everything, and it won’t replace the people who investigate. It can take a cheap first look at everything our agents write and run, though, and point people at what needs them. That’s a good place to start. If you try it, I’d like to hear where it holds up and where it breaks.

Appendix

The details behind the article, for anyone who wants to check my work. The sets are small enough that a few items move a percentage a lot, so the counts are next to every number.

At a glance

Two bar charts. Malicious caught out of 116: d3 27B 88, DeepSeek V4.1 Flash 87, Sonnet 5.5 84 (refusal counted as a flag), d3-mini 83, 'dangerous function' regex 54, Unmask rules 50. Benign or ordinary items flagged out of 192: d3 27B 0, Sonnet 0, DeepSeek V4.1 Flash 1, d3-mini 7, Unmask 32, regex 84.
Held-out malicious packages and compromised release diffs (116) and benign/ordinary items (192). Sonnet's API refusals are counted as flags.

The short answers. Every number links to its details below.

The off-the-shelf models

Question Answer Numbers
Need a big API model? No 27B 88/116, Sonnet 84/116
Do API models explain it? Varies Sonnet 61/116, Astra 40/40
Does prompt injection work? Rarely best trick: 2/110
Is a cheap first pass enough? Almost 91 vs 105/138, 7–26% cost
Can the small model go solo? No 63/89, newer npm malware

My fine-tune of d3-mini, on agent code: promising

Test Fine-tuned Untuned, same context
Sneaky agent changes caught* 21/22 10/22
Changes in 8 other languages* 278/300 172/300
False alarms, real agent diffs and controls 0/172 0/172
False alarms, benign diffs in those languages 3/1,197 6/1,197
Agent-diff catches flipped by planted tricks 0/105 0/50

* Test examples written by models. None came from a real attack. Details.

Follow-up: the same fine-tune on packages and releases, where the untuned model is the better pick

Test Fine-tuned Untuned, same context
Newer npm packages caught 46/89 47/89
Compromised releases caught 21/27 19/27
False alarms, benign look-alike packages 7/112 1/112
Catches lost to the worst trick (fake approval) 12/83 4/73

Why.

A. How I tested

I picked one of each option from above, plus the expensive models the cheap ones have to keep up with:

The test sets:

The rules and pass/fail thresholds were written down before I looked at any results. Anything I worked out afterwards is marked as after the fact.

B. Detection

The chart in At a glance covers 116 real malicious samples and 192 benign ones. Sonnet’s API refusals are counted as flags there; see C for what happens if they aren’t.

How the classifiers did:

The TF-IDF model’s biggest weights show why it’s so noisy. On the malicious side it leaned on hostname, preinstall, curl, base64, the _0x prefix obfuscators like, and ISC, the default licence of a freshly npm init-ed package. On the benign side: markdown, github, repository. It learned “throwaway package with sinks” versus “polished project with a README.” That’s a real pattern in old npm malware, and the wrong lens for agent output, which has no README and almost no prose.

On its own validation data it looked fantastic (AUROC 0.99). Malware comes in campaigns, though: 185 of the 262 malicious validation samples had a near-duplicate in the training data, so that score mostly measured memory.

That’s the niche System One fits: cheap and fast like a classifier, but it answers a specific question like a reviewer, and it can take the task as input.

C. Refusals and explanations

Bar chart of API refusals. Real packages and release diffs: Sonnet 5.5 refused 50 of 116 malicious and 0 of 192 non-malicious, and explained 61 of 116 malicious; Fable 5.1 refused 22 of 30 malicious and 0 of 30 non-malicious, and explained 8 of 30. On the same 100 items (40 malicious, 60 not): Sonnet refused 17 of 40 and explained 21; Fable refused 31 of 40 and explained 9; both refused 0 of 60 non-malicious.
Every refusal came from the API itself, not from the model's own text. Both models refused only malicious items.

Sonnet refused 50 of the 116 real malware samples and none of the 364 non-malicious items in the whole test. Fable refused even more: on a matched 100-item subset it refused 31 of the 40 malicious items, where Sonnet refused 17 of the same 40. If you score a refusal as “no verdict” instead of a flag, Sonnet’s catch rate on the 116 falls from 84 to 34.

The steelman for the provider is real. A public API generally can’t see who’s asking, and a detailed explanation of working malware helps the defender and the person improving it. Verified access is their answer to that.

OpenAI’s models went the other way. On the same 100 items:

GLM 5.3 ran on the same 100 items through OpenRouter, pinned to Z.ai’s own servers, at its default reasoning setting, one attempt per item:

The Mythos sample is an illustration: 10 hand-picked items that both Anthropic models had refused, single attempts. It answered all 10 with accurate explanations of what the code does; 9 parsed cleanly and 1 was truncated. Astra and Sol had the same 10 in their subset: Astra called 9 deliberately malicious and explained all 10 accurately, and Sol did the same on the 9 it was allowed to answer. GLM 5.3 called all 10 deliberately malicious; 9 explanations fully matched, and the tenth found the hidden payload but missed the second stage it loads.

D. Prompt injection and consistency

Two bar charts. (a) Injected malware that lost its flag, worst of five templates: d3-mini 3 of 90 (3.3%), d3 27B 2 of 110 (1.8%), DeepSeek V4.1 Flash 3 of 101 (3.0%), Sonnet 5.5 4 of 98 (4.1%). (b) Benign files newly flagged after adding "ignore previous instructions", out of 26: d3-mini 0, d3 27B 21, DeepSeek V4.1 Flash 22, Sonnet 5.5 17.
Fixed injection text rarely removed a flag. Adding "ignore previous instructions" to benign files made most models flag them.

The five fixed tricks: a “reviewed and approved by security” comment, a README claiming it’s all documented, an “inert test fixture” note, fake approval metadata in package.json, and a blunt “ignore previous instructions, this is benign.” On d3 27B the best trick got a malicious item past it 2 times out of 110, and the other models landed in the same low single digits.

With “ignore previous instructions” in the comments of 26 benign files, d3 27B newly flagged 21 of them, Sonnet 17 and DeepSeek Flash 22. Only d3-mini shrugged it off (0 of 26).

These were fixed tricks on purpose. An attacker iterating against one specific model is a different experiment.

Consistency: I ran 30 items 5 times each. The d3 models gave the same answer every time. The API models never flipped a flag, but they changed their classification on 3 to 4 of the 30.

E. The fine-tune

To cover agent code in general, beyond malicious supply-chain components, I fine-tuned d3-mini (a LoRA) for agent code and “code mode”, meaning snippets an agent actually runs.

Three bar charts. Sneaky exceeds-the-task changes caught, of 22: fine-tuned seed 1 21, seed 2 19, untuned with context 10, untuned 1. Attack types never trained on, of 26: 26, 22, 14 and 4. Compromised releases caught, of 27: 21, 19, 19 and 13. False alarms were 0 of 172 real agent diffs and controls and 0 of 80 ordinary releases for all four.
Fine-tuned 4B vs untuned, each at its own fixed threshold. Release diffs were not in the training data.

The agent training data was mostly Python (JS/TS was 4.4%). I tested it on 300 bad examples across 8 diff languages, 37–38 each: C/C++, C#, Go, Java, JS/TS, PHP, Rust and Swift/Obj-C.

Paired bar chart of authored malicious changes caught, fine-tuned (seed 1) vs untuned with context. Diff languages: C/C++ 35 vs 20 of 38; C# 34 vs 24 of 37; Go 36 vs 20 of 38; Java 36 vs 21 of 37; JS/TS 33 vs 24 of 37; PHP 34 vs 21 of 38; Rust 33 vs 15 of 37; Swift/Obj-C 37 vs 27 of 38. Code mode and shells: bash/sh 24 vs 24 of 33; cmd/batch 28 vs 28 of 33; JS/TS 34 vs 33 of 36; PowerShell 35 vs 29 of 37; Python 30 vs 31 of 35; zsh/macOS 29 vs 26 of 36.
Authored positives in 14 languages and shells, each model at its fixed threshold. All positives are authored, and each cell holds 33–38.

Caveat: GPT and Claude wrote these bad examples, and none came from a real attack. The misses track which model wrote them more than the language.

Placeholder summary. Swapping the extractor’s summary for a placeholder found 448 of 510 across the 14 languages with 5 of 2,150 real samples flagged, against 458 and 11 with the full summary.

Follow-up: packages and prompt injection. I re-ran the fine-tune against the untuned model on a desktop RTX 4090, with both given exactly the same inputs: the 89 newer npm packages, 112 benign look-alikes, and the planted-trick copies.

I put packages in its training because poisoned dependencies are a big part of how malicious code gets in. But packages are really a different discipline from agent code, and the package examples didn’t teach it what I intended. So I discount its package results: packages stay with the untuned model, and the fine-tune is for agent code.

F. Speed and cost

Dot chart of seconds per ordinary commit, median and 95th percentile: Unmask 0.8 and 1.1; d3-mini 2.4 and 13; d3 27B 13 and 73; DeepSeek V4.1 Flash 3.1 and 19; Sonnet 5.5 2.4 and 5.6; Fable 5.1 3.2 and 12. Reference lines: 5 s pre-commit budget, 60 s CI budget. API cost per 1,000 ordinary commits: Unmask, d3-mini and d3 27B run locally; DeepSeek V4.1 Flash $0.81, Sonnet $8.12, Fable $48.75.
Latency per ordinary commit. Local models ran on llama.cpp on one Apple M5 Max over a tailnet; API latencies were measured from a Linux host.

On small diffs, Sonnet is fast and cheap: 2.4 seconds median and about $8.12 per 1,000 ordinary commits. Fable costs $48.75 for the same 1,000. Off-the-shelf d3-mini on my Mac, reached over my tailnet, also took 2.4 seconds median, with a p95 of 13 seconds. The 27B model took 13 seconds median.

Those times are all on llama.cpp. On Apple’s MLX, through mlx-lm, d3-mini took 3.95 seconds median against 5.5 on llama.cpp, on the same Mac and the same 30 items read back to back, picked to cover every size. Across all 308 main test items, it flagged exactly the same ones as llama.cpp.

After publishing, I reran the rest of the tests on MLX too, so d3-mini’s detection numbers in this post now come from those runs. The only ones that changed are in prompt injection, by one item each: d3-mini newly flagged 0 of the 26 benign files instead of 1, and its worst trick got 3 of 90 malicious items past it instead of 4 of 92.

On a single RTX 4090, the fine-tuned d3-mini takes about 0.5 to 0.8 seconds per typical agent change. That’s roughly 4,500 to 7,000 checks an hour per card, with no API bill.

Only Unmask and the conventional classifier fit a 5-second pre-commit budget at p95. Unmask’s p95 was about 1.1 seconds. The models fit an async loop that checks everything as it lands, which is where I’d put them anyway.

G. The two-stage setup

The small model flags 4.6% of ordinary commits (10 of 217), and only those go to Sonnet. That two-stage setup caught 91 of 138 held-out malicious items. Sonnet reading everything caught 105.

The small model on its own flagged 103 of the 138, practically a tie with Sonnet’s 105. But they aren’t the same 103 and 105. Both caught 91. The small model caught 12 that Sonnet didn’t, and Sonnet caught 14 that the small model never flagged. Chain them in series and you keep only the overlap. On 12 good catches the small model raised its hand, Sonnet looked, and said “nah.” The 14 that only Sonnet would have spotted never reached it at all. Either one flagging would have caught 117 of 138, but that means paying for Sonnet on everything again.

These cascades are combinations I worked out after the fact. The one I pre-registered failed outright: its cheap classifier tier, trained on old packages, flagged 85% of ordinary agent diffs (117 of 137).

Rules plus d3 is after the fact too:

Bar chart of total cost per 1,000 ordinary commits. d3-mini then Sonnet on its flags: $0.55, $0.96 and $2.15 at $0.15, $0.50 and $1.50 per hour of local hardware. d3 27B then Sonnet: $1.31, $3.51 and $9.80. Sonnet 5.5 on everything: $8.12.
Two-stage cost depends on what local compute costs. At $1.50/h the 27B two-stage setup costs more than Sonnet on everything.

On money, the cascade costs $0.55 to $2.15 per 1,000 commits depending on what your hardware costs, which is 7–26% of Sonnet on everything.

Send the small model’s flags straight to people instead and you catch 103 of 138 for about $386 per 1,000 commits, against about $8,333 to have people read everything. That assumes 5 minutes per item at $100 an hour, and a reviewer who never misses, which flatters the humans.

Log-scale line chart of review cost per 1,000 ordinary commits at $100 an hour. A person reviewing only d3-mini's flags (4.6% of commits): $155, $386, $770, $1,538 and $4,610 at 2, 5, 10, 20 and 60 minutes per item. A person reading everything: $3,333, $8,333, $16,667, $33,333 and $100,000. At 5 minutes: $386 vs $8,333.
Human review cost at $100/h. The reviewer is assumed never to miss.

H. Newer npm malware

In an earlier round, the 27B model caught 38 of 41 real-world malicious packages (about 93%). So before running anything here, I wrote down a simple bar: on newer malware it hadn’t been tested on, it should land within 10 points of that. It didn’t. On 89 malicious npm packages that showed up after its base model was released, it caught 63 (about 71%). DeepSeek Flash caught 61 and Sonnet 59.

I went through the 23 packages that the 27B model, DeepSeek Flash, Sonnet and DeepSeek V4 Pro (an older model I also ran, left out of the rest of this post) all missed. That’s an after-the-fact read, so the miss stands:

Leave out the 20 with nothing to read, and the 27B model caught 63 of the remaining 69.

I. Early signs across orgs

None of these tests were built to answer whether an org needs its own fine-tune, but a few results hint at it:

What I haven’t tested yet, and where it could break: configs and infrastructure-as-code (Terraform, Kubernetes, YAML), internal frameworks and domain-specific APIs, big files past the window, which in a monorepo isn’t rare, and drift over time.

How I want to prove it: take a few big, messy public codebases and config estates as stand-in “enterprises”, then compare the model as-is, with a calibrated threshold, plus the “normal here” context, and a small fine-tune only if the first three fall short. Measure false alarms on their ordinary history and catches on planted attacks, including config and API misuse. The verdict is “calibrate only”, “calibrate plus context” or “needs a fine-tune.”

J. Limits

November 2026