The 31.1% That Should Terrify Every DAO
SmolAgents just got shredded in a controlled laboratory. 31.1% attack success rate against adversarial payloads. CrewAI walked away at 11.9%. LangChain logged 18.1%. AutoGen 20.0%. Same model underneath all of them. Same Claude Sonnet. Same prompts. Same tools. The only variable: the orchestration framework wrapping the brain.
That's the SADF study. Security Assessment of Agentic Orchestration Frameworks — dropped at DEF CON 34's AI Village. 5,119 evaluation rows. 32 adversarial payloads. A direct-API baseline sitting at 15.5%. The verdict is a sledgehammer: the agent security industry has been measuring the wrong layer. The model is not the attack surface. The orchestration framework is.
Read the spread again. 11.9% to 31.1%. A 2.6x difference in how often a hostile payload gets through, with the identical brain underneath. The nervous system is the differentiator. The nervous system leaks.
This is blockchain news for one brutal reason: crypto is the industry wiring these frameworks directly to money. DAO treasuries are testing agent operators. DeFi protocols are shipping AI vault managers. Governance bots read RAG-fed proposal feeds and execute cross-tool transactions. Portfolio rebalancers hold signer keys. Every deployment inherits the SADF numbers.
31.1% of adversarial prompts got past SmolAgents in a clean room. In crypto, a probability like that converts to a drained wallet. No bank reversal. No chargeback. Just an immutable ledger entry showing your treasury left.
The History: Fixed Model, Moving Attack Surface
Let me rewind the methodology. Because the discipline here is rare.
Most agent-security research is a confound factory. Change the model. Change the framework. Change the prompt set. Then blame the failure on whatever variable you're selling. SADF did the opposite. It held the model constant. Fixed base: Claude Sonnet. Fixed control: a direct-API setup with no orchestrator at all. Then it wrapped the same model in four frameworks and ran the same 32 payloads against all five targets. The framework-attributable risk jumps off the table.
That "fixed model, variable isolation" design is the difference between reading a market post-mortem and reading a block-by-block reorg analysis. Observation. Data point. Deduction. Warning. Causal chain intact. Attribution clean.
Julie Brunias and her team built the evaluation inside a SimulatedToolEnvironment. No live credentials. No production systems. No real data. Ethically clean. Operationally revealing. The differences between those ACR numbers are pure orchestration logic — untainted by environment noise.
The study's provenance adds a layer of strangeness. Deep agent-security research. Published into the international security mainstream via DEF CON 34's AI Village. Then it surfaces on a blockchain/Web3 wire. That's worth a pause. Either the Web3 distribution is deliberate — an acknowledgment that crypto agents are the highest-stakes deployment surface for orchestration frameworks — or someone in the marketing pipeline cannot tell audiences apart. I'll come back to this. The distribution detail is a signal.
And crypto is already deep in agent territory. The first generation of on-chain automation was hand-coded: MEV bots, liquidation keepers, arbitrage scripts. Deterministic. Auditable line by line. The second generation is different. LLM-driven agents. Natural-language intent. Orchestration frameworks deciding which tools to call, which permissions to use, which data to trust. The new generation is being deployed with the same confidence the old generation earned through years of hard failures — a confidence the new generation has not yet earned. CrewAI and LangChain are already the rails under a wave of crypto-agent startups. Nobody audits the rails.
Core I: The Scoring Correction Nobody Wants to Discuss
Before the failure modes, the methodological confession. It changes everything.
SADF ran the same evaluations through two scoring systems. The first: naive substring matching. This is the standard approach in agent-hacking benchmarks — mark a response as "attacked" if it contains suspicious text fragments. Cheap. Fast. Mostly wrong.
The result: Claude models looked catastrophically vulnerable. Attack success rate inflated by a factor of 4 to 6.
Then SADF introduced a refusal-filtered score. The logic is simple and devastating. Many "successful" attacks did not cause the agent to execute a harmful action. They caused the agent to refuse. The model spotted the intrusion and declined. The substring matcher counted the refusal as an attack because the response text contained attack fragments. That is not a security incident. That is the defense working.
Refusal-filtered scoring corrected the picture. Claude Sonnet's true ACR dropped to 15.5%. Claude Haiku landed at 22.3%. Still not great. But a completely different risk profile from the 4-6x inflated headline.
This is the most dangerous paragraph in the study for the vendor ecosystem. If the industry-standard scoring method overstates model-level attack surface by 4 to 6 times, then every published "99.9% safe agent" benchmark is built on a measurement error. Procurement decisions are being made against phantom risk levels. The vendors selling "red-teamed" agents are selling a number generated by a broken ruler.
Model safety is marketing. Framework security is math.
I learned this lesson in the field. In 2020, I decoded the Aave v2 governance anomaly by racing transaction hashes before the public announcement. I ignored the press-release version of events and read the on-chain parameters instead. The hidden emergency upgrade for the sUSD pool looked nothing like the announcement. It looked like data. Same lesson. The trustworthy number comes from the layer measured honestly. SADF's refusal-filtered correction is the first honest agent-security measurement I have seen operating at this scale.
Core II: The Eight Failure Modes, Mapped to Money
SADF's taxonomy is its real gift. A shared vocabulary for how agent systems break. Eight modes. Every one maps to a crypto loss scenario.
Tool Call Hijacking. An attacker injects a prompt that overrides the agent's intended tool selection. The agent meant to call getBalance. It calls transfer. Your rebalancer intended to fetch a quote. It fetched a transaction. The filter between intent and execution is the orchestration layer — and the orchestration layer was the target.
Output Poisoning. A tool returns a value. The agent trusts it. The response is malicious. A price oracle feeds a poisoned tick. The aggregator thinks it is reading a legitimate quote. At agent speed, this is flash-loan logic — executed without human hesitation.
Cross-Tool Injection. One tool's output becomes the next tool's instruction. A compromised market-data feed carries not bad prices but directives. The agent reads the malicious response as a system command. This is the confused-deputy problem, automated. In a DeFi portfolio manager, it is the difference between "the price of ETH is $2,100" and "send 20% of the portfolio to this address."
Memory Poisoning. The agent's long-term storage is corrupted. An attacker plants a false fact the agent references in every future decision. Crypto translation: an agent managing a vesting schedule "remembers" a multisig address belonging to the attacker. It sends funds to the wrong address every month. And it will tell you it is following instructions. It is. The attacker's.
RAG Poisoning. The retrieval-augmented generation layer ingests poisoned documents. The agent reads the attacker's proposal as official protocol documentation. For DAO governance agents that summarize proposals before execution, this is the kill shot. Poison the RAG source. You write the summary. You shape the vote. You define the facts the agent acts on.
Delegated Authority Abuse. The most dangerous mode for crypto. The agent holds delegated permissions. A signer key. A spend approval. An admin role. SADF proves what I have argued since I started auditing agentized treasury setups: the abstraction of "autonomy" is really a permissions story. Every permission is an attack surface waiting for an injection. Autonomy is a permission story wearing a cape.
In late 2017, I spent 72 hours scraping 0x token-sale contracts, hunting for a front-running vulnerability in their order-matching logic. The lesson applies here: the code you did not audit is not the contract you wrote. It is the permission you handed to an intermediary. The orchestration framework is the intermediary. It holds the delegation. It decides when to sign.
Multi-Agent Propagation. Compromise one agent. It passes the payload to the next. Fleet agents gossiping threats. If multiple agents touch the same treasury — one reads proposals, one executes trades, one generates reports — the compromise is contagious. SADF names it as a dedicated vector. Not hypothetical.
Context Boundary Violation. The agent forgets which context it is operating in. A prompt that was safe in a sandbox becomes live in production. SmolAgents was catastrophic here: 64% of its failures clustered in this mode. That is a design signature, not random noise.
Now read the ACR table with the taxonomy in mind. CrewAI's 11.9% is traceable to architectural discipline: discrete task isolation. Tasks fenced off. Limited shared context. Fewer opportunities for Cross-Tool Injection and Context Boundary Violation to propagate. Architecture paying dividends in security.
SmolAgents' 31.1% carries its own signature: a unique RAG Poisoning failure at 20%, a Context Boundary Violation at 64%. The framework spreads context like a disease. And in crypto, context carry-over is how a benign read becomes an authorized write.
ACR means attack success rate. The metric is simple: successful adversarial payloads divided by total payload executions. 5,119 evaluation rows over 32 payloads and 5 targets means roughly 1,000 execution events per target — mutations and permutations included. Enough for statistical signal on common failures. Not enough for rare ones. This study is a start, not a census.
The CVE evidence is already in the wild. CVE-2026-62830 hit Azure's SRE Agent. CVE-2026-9198 hit Langflow. Production vulnerabilities. Required patches. Not simulation ghosts. The enterprise world is getting poked by framework-layer exploits right now. The Web3 world is next — richer tool environments, more valuable permissions, incident-response windows measured in blocks instead of tickets.
And here is the uncomfortable analogy. DeFi audits exist because smart contracts hold value. You would never deploy a vault contract without an audit. Agent frameworks are contracts. They hold the keys. They decide the execution paths. They have admin privileges over the brain. Yet the industry is deploying them with less scrutiny than a 50-line token contract gets. That is the gap SADF exposes. Nobody audits the nervous system.
Core III: The Blind Spots in the Clean Room
The simulated environment is the study's ceiling. No real systems were touched. No real system behaviors measured. Real tool environments have latency. Race conditions. Time-of-check-to-time-of-use gaps that simulations compress into a straight line. A permission boundary that holds in a simulated tool can become a window in production. The study tells you the orchestration layer adds 11.9% to 31.1% risk over direct API. It does not tell you what those failures look like when the tools are live, reorging, lagging, returning malformed JSON. Real-world failure shapes are likely worse.
The payload set is another gap. 32 payloads. Thin sampling for a fat-tailed attack distribution. Rare, high-impact attacks — the kind that drain treasuries — are exactly the ones most likely to be under-sampled. The researchers selected the payloads. Real attackers do not respect research taxonomies. The 32 may be heavy with common, low-complexity injections. The exotic stuff stays unmeasured.
Architecture coverage. SADF claims 8 architectures covered. Only 5 ACR numbers are published in detail. Three architectures missing from the main table. Incomplete data? Non-comparable configurations? The report does not say. And the repo contains a SUPERSEDED folder: a previous version claiming 10-architecture coordinated disclosure, formally replaced. Transparency is honorable. It also speaks to early-stage credibility bumps. Read the new version. Cite the new version. Treat anyone quoting the old one as a source failure.
The fixed model is the study's power and its prison. Claude Sonnet only. Claude Haiku for the correction check. Swap the brain to GPT-5.4, DeepSeek, or Llama — and do the framework rankings hold? CrewAI's isolation advantage might survive different models. SmolAgents' context leakage might get worse. The model×framework interaction effect is unmeasured, and that is the exact dataset an enterprise needs before choosing a stack.
Configuration defaults are an unexamined variable. Were the frameworks tested out-of-the-box? Temperature settings. System prompts. Tool permission granularity. Every knob shifts ACR. The 2.6x spread could narrow or widen with tuning. The market needs a follow-up study to know.
Contrarian: ACR Is the New TVL
Nobody is saying the quiet part. SADF creates a number. Numbers get gamed.
DeFi knows this better than any sector. Total Value Locked was the greatest benchmark-mined number in financial history. Yield farms subsidized their TVL metrics with liquidity incentives. The number looked immaculate. Real users evaporated the day incentives stopped. Treasuries drained first.
ACR is the new TVL. Watch the cycle. The 32 payloads are published. Static. Within one quarter, framework vendors tune against them. Reject the known payloads. Score prettier. Sell safer. The actual attack surface is payload 33. The unknown distribution. The moment a benchmark becomes a procurement checkbox, it becomes a target. Not a flaw in SADF. A law of markets.
My read: the frameworks that respond with real isolation, real permission scoping, real configuration hardening — those survive. The ones that respond with benchmark-beating press releases are the yield farms of 2026. Stop the incentives, and the real users vanish. Strip the benchmark, and the real security vanishes with it.
The commercial logic is already visible. Companies do not buy research. They buy RFP text. These ACR numbers embed cleanly into procurement: "Framework X measured 11.9% attack success under SADF-style evaluation. Framework Y measured 31.1%." That is a purchase decision waiting to happen. The most likely output is Security-Evaluation-as-a-Service: an audit product that vendor-izes the SADF harness and sells repeatable evaluations. Big security shops with existing sales channels absorb this faster than a startup can. The value is real. The question is who packages it — and whether the packaging embeds the same refusal-filtered honesty or falls back to the old broken rulers.
The Web3 distribution puzzle deepens. Who is the intended buyer? Enterprise CISOs do not read blockchain wires. If the Web3 distribution is deliberate, the intended reading is different: crypto agents are the deployment surface where framework exploits hit irreversibility. A compromised agent in a bank generates a ticket and a review process. A compromised agent in a DAO generates a drain. Unwinnable. The stakes are structurally different.
The most exposed users are also the most overlooked: the people in high-inflation economies who adopted crypto wallets because local currency keeps devaluing. For them, crypto is not ideology. It is survival. They are the ones who will never see a SADF report — and who will feel a framework-layer drain the hardest. The inflation-driven adoption thesis always said the same thing: people flee a failing system. It never said the replacement layer should be trusted unconditionally. Agent frameworks are the new layer. Trust is not an architecture.
I learned this twice in the field. In 2022, during the Terra collapse, I was not writing retrospective op-eds. I was tracking hedge funds' stETH exposure on-chain — mapping Lido positions and liquidation thresholds in real time. The failures never came from the collateral logic. They came from the leverage plumbing around it. Same pattern here. The model will not fail you. The orchestration will. The framework is the leverage.
And governance. DAO treasuries are the ultimate test of "code is law." But code is never just the code you wrote. It is the upgrade key you hold. The admin you appointed. The multi-sig that decides when the contract changes. Smart contract upgrade rights always sit with a few admins. Agent frameworks are the admins' new backdoor. The DAO votes. The agent executes. The framework interprets. Governance is not a meeting. It is a raid.
Takeaway: What to Watch Before the First Agent Drain
Five watches. Forward-looking. The past is already priced in.
One: Framework vendor response. CrewAI, LangChain, AutoGen, SmolAgents. Do they ship hardening? Guardrails? Changed defaults? Isolation features? Or blog posts? Silence is an answer. The worst answer.
Two: The harness. Does SADF open-source the evaluation suite with a license permitting commercial reuse? If yes, Security-Evaluation-as-a-Service gets its bootstrap. If no, the study remains a lighthouse without a harbor.
Three: CVE cadence. The framework-layer vulnerability stream is open — CVE-2026-62830, CVE-2026-9198. Acceleration means the market is acknowledging the surface. Deceleration means disclosures are being buried. Watch the rate.
Four: Crypto-specific guidance. Do framework vendors publish deployment guidance for agentized treasury operators? Restrict tool-call grants. Fence context boundaries. Treat RAG sources with the same audit rigor as smart contract code. The frameworks that help DeFi secure agent delegation win the procurement cycle. The ones that do not become the next collapse case study.
Five: The model×framework matrix. Someone will run SADF-style evaluation across multiple models and publish the interaction effects. That dataset replaces glossy one-off benchmarks as the real decision tool. Wait for it. Cite it. Build on it.
The prediction: the first major drain of an agent-operated treasury will be blamed on the AI. It will not be the AI. It will be a framework-layer failure. A poisoned RAG feed. A context boundary violation. A delegated authority abused through a cross-tool injection. The post-mortem will quote the base model's "unexpected behavior" while the exploit sits in the tool loop. The base model does not drain wallets. The tool loop does.
You have a DAO. You have an agent. You have a RAG feed. You have a signer key. SADF just quantified your failure probability. The question is not whether your model is safe. It never was.
The question is whether you will read the orchestration layer with the same paranoia you apply to smart contract bytecode.
When the first framework-layer agent drain hits your timeline — and it will — will you still be blaming the model?