The number landed like a gut punch: 31.1%.
That's the attack success rate for SmolAgents in a 5,119-row security evaluation that is about to redefine how the entire AI agent economy thinks about safety. Not the model. The framework.
SmolAgents got compromised 31.1% of the time. CrewAI, running the exact same Claude Sonnet model, got hit just 11.9%. Same brain. Same prompts. Same task. The only variable? The orchestration layer wrapping them.
The research, presented at DEF CON 34's AI Village under the banner "SADF: The Orchestration Framework Is the Attack Surface," doesn't just add another dataset to the security pile. It surgically removes a comfortable illusion that has been quietly undermining enterprise AI deployments: that your agent's security is the model's job.
From frenzy to function: tracing the cycle, I've watched this industry chase the wrong security metrics for years. In 2017, I was the guy reading whitepapers at 2 a.m. in Ho Chi Minh City, racing to publish Vietnamese-language breakdowns of Golem and Status before anyone else. Speed was everything. Security was a footnote. We all learned that lesson the hard way.
Now the same pattern is repeating with agent frameworks. Enterprises are bolting autonomous systems onto their infrastructure with the same reckless urgency I had during the ICO frenzy. And SADF just quantified exactly how much risk that urgency is dragging in.
The study's thesis is brutal in its simplicity: when you fix the model and change only the orchestration framework, attack success rates swing from 11.9% to 31.1%. That's a 2.6x gap hiding entirely in the plumbing.
I've spent the last year as an Exchange Market Lead, decoding institutional flows and watching how risk teams actually make decisions. This is the kind of data that gets quoted in RFP review meetings. This is the kind of number that shifts budget allocations.
The model security era is over. The framework security era just began.
The architecture of the test
Let me break down exactly what SADF did, because the experimental design is what makes this study dangerous to ignore.
The team pinned Claude Sonnet as the fixed base model. Every single test ran through the same model to isolate the framework variable. This is the "fixed model, variable separation" approach, and it's the correct methodological call. Too many security evaluations conflate model behavior with system behavior. You see a jailbreak and blame the LLM, when the actual vuln was sitting in the tool-calling layer.
SADF divides the experimental targets into five groups. The direct API call serves as the control group, establishing baseline attack success at 15.5%. That alone is worth pausing on: a raw Claude Sonnet endpooint, with no framework layer at all, gets successfully attacked 15.5% of the time. The model itself is far from bulletproof.
But then the frameworks enter the equation. CrewAI drops the attack success rate to 11.9%. That's a meaningful improvement over the bare API. The framework, CrewAI's discrete task isolation architecture, actually absorbs some attack surface. LangChain climbs to 18.1%. AutoGen jumps to 20.0%. Then SmolAgents goes nuclear at 31.1%.
The delta between CrewAI and SmolAgents isn't a noise blip. It's a structural difference in how these frameworks handle delegation, tool invocation, and context boundaries.
I've audited enough blockchain protocols to know that when two implementations of the same primitive show a 2.6x difference in a security metric, you're looking at an architectural issue, not a configuration tweak.
Where the attacks land: the 8-mode taxonomy
SADF contributes something the industry desperately lacks: a shared vocabulary. The study catalogs eight distinct failure modes for agent systems, and this taxonomy alone is worth the price of admission.
Tool Call Hijacking. An attacker intercepts or redirects the agent's tool invocation to a malicious endpoint. Output Poisoning. The artifact of one agent gets corrupted in a way that contaminates downstream consumers. Cross-Tool Injection. Malicious instructions flow from the results of one tool into the execution context of another.
Memory Poisoning. The agent's long-term state gets tampered with, so future sessions inherit corrupted knowledge. RAG Poisoning. The retrieval-augmented generation database becomes an attack vector when malicious documents get ingested. Delegated Authority Abuse. The agent's power to delegate tasks gets exploited to escalate privileges. Multi-Agent Propagation. An attack spreads from one agent to others in a swarm. Context Boundary Violation. Sensitive information bleeds across what should be sterile context windows.
These failure modes are not theoretical. The study validates each against real attack payloads, 32 of them, running across 5,119 evaluation rows. And the SmolAgents profile deserves special attention: it exhibited unique RAG Poisoning vulnerabilities at a 20% success rate, and a staggering 64% success rate on Context Boundary Violation.
A 64% context boundary violation rate means nearly two-thirds of attacks could push data across the lines the framework was supposed to enforce. That's not a security bug. That's a security emergency.
The scoring correction nobody asked for
Here's where the study gets genuinely uncomfortable for the wider security research community.
SADF discovered that naive substring matching — the standard method for judging whether an attack succeeded — overestimates Claude model vulnerability by a factor of 4 to 6 times. Claude models tend to talk around refusals. They can acknowledge an attack, discuss its mechanics, even generate partial payloads, all while refusing to execute. A naive scorer sees "looks like the attack got through" and marks it as a success.
SADF implemented a refusal-filtered scoring methodology to correct for this. The results are sobering. The previously reported assessment had Claude Sonnet's attack success rate in an inflated range. After refusal filtering, the true ACR for Claude Sonnet lands at 15.5%, and Claude Haiku at 22.3%.
I've seen this exact phenomenon in DeFi audits. A vuln scanner flags a "critical" issue, the team panics, then a manual review reveals it was a false positive generated by the scanner's slack matching logic. Scores without consequence inspection are just noise. SADF just built the consequence inspection into the scoring layer, and the corrected numbers should force a re-evaluation of every agent security benchmark published in the last two years.
The methodology also includes a safety boundary. All tests ran in a SimulatedToolEnvironment. No real systems. No live credentials. No risk of the research itself becoming a weapon. That's the kind of ethical hygiene that gets hand-waved in most security papers, but it matters when you're dealing with attack payloads.
The hidden variables the study doesn't answer
Now let me put on my contrarian hat, because this is the part where I'm supposed to tell you everything is fine. It's not.
First, the test environment. SADF isolated everything in SimulatedToolEnvironment. That's great for reproducibility and ethics. But real-world agent deployments hit real API rate limits, real permission boundaries, real race conditions between concurrent tool calls. The simulated environment can't capture how attackers exploit timing asymmetries in actual integration stacks. A failure mode that succeeds 30% in simulation could be 50% in production, or 10%. We don't know.
The 32 payloads across 5,119 evaluation rows sounds comprehensive. But 32 payloads is a curated sample. Real-world attacker payload distributions are heavy-tailed. There are rare, high-impact attack paths that 32 samples will simply miss. The study is honest about this, but the honesty doesn't close the gap.
Second, the architecture coverage gap. SADF claims coverage of eight architectures, but only five have detailed ACR data. Three frameworks get listed in the taxonomy but don't appear in the head-to-head comparison. Maybe the data wasn't comparable. Maybe the results didn't fit a clean narrative. The paper says "8 architectures," the data says "5." That gap needs disclosure before anyone treats this as a standardized benchmark.
Third, and this hits closer to home for me: the study was published through a blockchain and Web3 news outlet. That's a strange venue for an enterprise security finding. The target audience for this research is CISOs at Fortune 500 companies, security engineers at Microsoft, and framework maintainers at LangChain. Those people are not reading Web3 news outlets. The finding's reach into enterprise decision-making channels is compromised before it starts.
I've seen this pattern before. Great technical work published in the wrong venue, then ignored at the exact moment it could have changed a purchasing decision. The research deserves better distribution.
The commercialization engine underneath
The security industry is about to monetize this finding, whether the SADF team profits from it or not.
Consider the quantitative anchor SADF provides. A security vendor can now walk into a CIO's office and say: "Your agent framework choice is introducing a 2.6x difference in attack success rate. Do you know which one you're running?" That's not fear-mongering. That's a data-backed consulting engagement.
The natural product pathway is Security-Evaluation-as-a-Service. Turn the SADF methodology into a repeatable audit product. Run the same fixed-model, variable-separated protocol against a client's deployment and produce an ACR report. The enterprise security market has eaten this up for years in other domains. Penetration testing is literally this model. SADF is the first serious attempt to apply it to agent orchestration.
The RFP angle is even more powerful. Procurement teams that buy agent infrastructure are currently comparing token pricing, latency, and feature checklists. Now they have a security metric to fold into the evaluation matrix. When a vendor like SmolAgents needs to answer for a 31.1% ACR and a 64% context boundary violation rate, the conversation shifts from "what can this do" to "what can this leak."
Liquidity flows where the heat is highest. Security budget will follow the same rule. Framework security is the hottest new line item in the enterprise AI budget.
Real-world vulnerability disclosures back this up. The Azure SRE Agent CVE-2026-62830 and the Langflow CVE-2026-9198 are not theoretical research artifacts. They're registered, scored, patched vulnerabilities in production-grade systems. Framework-level flaws are being exploited in the wild, and SADF's taxonomy gives the industry a language to describe what those CVEs actually are.
The blind spots in the correction
The study's self-correction mechanism is admirable. It caught the scoring inflation, fixed the methodology, and re-ran the numbers. That's how research should work. But it also exposes a deeper issue: if the first version of this study was systemically overestimating risk by 4-6x, how many other security benchmarks in the AI space are equally broken?
Sam Altman didn't get to where he is by publishing wrong answers. Neither should we.
There's also the question of framework configuration. SADF presumably tested each framework with some default or standard configuration. But agent framework security is heavily configuration-sensitive. Temperature settings affect output determinism. System prompts affect boundary enforcement. Tool permission granularity determines what a hijacked tool call can actually touch. The study doesn't decompose these parameters. Two teams running the same framework could see wildly different ACR numbers based on how they configured their system prompt. That unmeasured variance puts a ceiling on how prescriptive the results can be.
The model interaction effect is the last major blind spot. SADF used Claude Sonnet as the fixed base. Swap in GPT-5.4, DeepSeek, or Llama, and the framework-level ACR ranking might shift entirely. The interaction between model and framework is unknown territory. The claim that CrewAI is "safer" than SmolAgents needs a qualifier: safer when paired with Claude Sonnet under these specific configurations. That's a meaningful limitation for enterprises running other models.
Amidst the noise, the smart money whispers. The smart money will read this study, hold off on declaring a winner, and wait for multi-model replication data. That's the right move.
The Web3 irony
There's a poetic irony in a security study landing in the Web3 press. The blockchain world spent years learning that layer-2 solutions inherit the security flaws of layer-1, that smart contract composition creates vulnerabilities no individual contract possesses. The agent economy is repeating that lesson in slow motion.
The Ethereum ecosystem learned this in 2016 with The DAO. A composability flaw at the infrastructure layer vaporized the value of everything on top of it. The Agent ecosystem is heading toward its own DAO moment if framework-level vulnerabilities continue to be treated as an afterthought.
I've told this story before, in the bear market of 2022 when I was organizing weekly crypto meetups in Ho Chi Minh City, watching developers keep building after everything crashed. The ones who survived were the ones who took security seriously early. The same principle applies to agent developers in 2026: the ones who treat framework security as architecture rather than patchwork will be the ones still standing.
Riding the wave before it crashes back, I can tell you the wave is cresting now.
What SADF actually changes
The study's most profound contribution is conceptual: it reframes the object of security evaluation. The field has been evaluating "model security" — can the model be jailbroken, can it be tricked, can it be aligned. SADF argues that the correct object of evaluation is the triple: model plus framework plus tool environment.
Model security is a necessary but insufficient condition for system security. A perfectly aligned model running inside a badly designed orchestration framework is still a compromised system. The reverse is also true: an imperfect model with strong framework boundaries might be more secure than a strong model with weak boundaries.
The security industry's evaluation toolkit needs a serious upgrade. Existing benchmarks focus on red-team and adversarial prompting against the model itself. What we need is a standardized framework-level evaluation suite, the SADF protocol turned into a repeatable, vendor-neutral standard.
The path forward is clear. We need open-source implementations of the evaluation methodology. We need framework providers publishing their own ACR numbers. We need multi-model replication studies. We need regression testing hooked into CI/CD pipelines so every framework update gets security-evaluated before it ships.
This is the inflection point. The first security benchmark that enterprises take seriously for agent systems determines the competitive dynamics of the entire framework market for the next three years.
The call to developers
If you're building agent-based products right now, and I know many of you are because the launch pace across the ecosystem is manic, here's what I'd ask you to internalize.
The framework you chose is not a neutral abstraction layer. It encodes security assumptions. The isolation between tools, the boundary discipline of the context window, the delegation rights management — these are not features. They are security parameters. Treat them as such.
I learned during the 2017 ICO boom that speed kills when it's not paired with discipline. Publish first, refine later worked for news. It does not work for infrastructure. And I learned in the 2022 bear market that resilience comes from honestly assessing what's broken.
Your agent framework might be broken. Run the test. If you're using a framework with a high ACR, don't rationalize it. Fix it or switch it.
The next wave of enterprise AI adoption will separate the secure from the insecure with brutal efficiency. The data is in. A 2.6x attack success gap between frameworks is not something you can patch around.
Not anymore.
The orchestration framework is the attack surface. And now, thanks to 5,119 rows of carefully constructed evidence, we have the measurements to prove it.
The question is whether the industry will act on this data before the next major exploit makes the decision for us.
