A model refusing a harmful request is useful. It isn't a security boundary.
Refusal is a behavior the model was trained to produce, not a deterministic control. Attackers change the framing, build toward the objective over several turns, obscure the request, or just keep testing variations until something lands.
The techniques keep changing. What matters more is what happens once one of them works.
Models now sit inside agents that can browse the web, execute code, retrieve files, call APIs, and reach external tools. A successful jailbreak doesn't stop at a bad response anymore. It can shape what the system does next.
The jailbreak creates the opening. What the agent can reach, and what's watching it, decide how far that opening goes.
A refusal is behavior, not authorization Model providers invest heavily in safety training, system instructions, content filters, and other safeguards meant to keep a model from complying with harmful requests.
Those controls matter. But they work through how the model reads the request and its context, and that context can be changed. Persuade the model the request is harmless, fictional, permitted, or just one small step in a legitimate task, and it can reach a different decision than it would have reached cold. It can also fail to notice the full objective when the work arrives split into smaller pieces.
A jailbreak and prompt injection get lumped together constantly, but they're not the same failure. A jailbreak pushes the model past its own safety training. Prompt injection overrides an application's broader instructions, sometimes through content an agent reads rather than a direct request. An agent can be exposed to both, sometimes in the same incident, but they call for different fixes.
Either way, the gap between refusing and authorizing is manageable when a model can only return text. It gets a lot more consequential once that model can access sensitive data or act on someone's behalf.
An agent shouldn't get permission to do something just because the model decided the request sounded fine.
The attack may not arrive as one obvious prompt The familiar image of jailbreaking is a user submitting one carefully crafted prompt and immediately getting something the model was trained to refuse.
It still happens, but it's only one way the attack unfolds.
The objective can build gradually across a conversation: early prompts set up a persona, scenario, or legitimate-looking task, and later prompts add detail that leans on what the model already accepted. Each message looks harmless on its own; the risk only shows up once you look at the whole sequence.
There's a version of this that doesn't even need a coherent scenario. Earlier research on many-shot jailbreaking found that stacking hundreds of faux exchanges into a single prompt, each one showing the model readily complying with a milder version of the request, could override safety training that held up fine against a handful of examples. No persona required, just volume. Success climbed as a fairly predictable function of how many examples got packed in.
Attackers can automate this too, and not just through conversation. Some approaches use an algorithm to search for the specific tokens or suffix that breaks a given model, testing large numbers of variants until one works. Others go further: instead of manually testing variations, another model generates an approach, watches the refusal, adjusts its strategy, and keeps going.
Research published in Nature Communications evaluated four large reasoning models acting as autonomous adversaries against nine target models. Given only an initial instruction, the attacking models planned and ran multi-turn jailbreaks with no further human involvement. The researchers reported a 97.14% overall success rate.
The 97.14% figure is specific to the study's models, prompts, evaluation method, and ten-turn limit. It isn't a universal success rate for jailbreaking in production. It shows that the same reasoning and persuasive capability designed to make a model more useful can also be pointed systematically at another model's safeguards.
The attacker doesn't need to know which prompt will work. An automated system just keeps adapting until it finds one that does.
Agents change the potential impact A chatbot produces a response. An agent turns that response into a sequence of actions, and depending on its role, that can include:
Searching internal or external systems Reading and modifying files Querying databases Sending messages Executing code Calling APIs Using credentials Invoking connected tools The list above doesn't mean agents are universally easier to jailbreak than chatbots. It means the consequences of a successful one depend on a lot more than the model underneath it.
An agent's system instructions, credentials, memory, tools, approval requirements, and orchestration framework all shape what a manipulated model can actually pull off.
SecureWebArena , a benchmark testing nine web agents across simulated environments, recorded user-level jailbreak payload-delivery rates from 35.33% to 80%, depending on the agent. Payload delivery here means whether the attacker's objective actually showed up in the final state of the environment, not just whether the model produced an unsafe sentence.
The benchmark caught something else too: in some scenarios, an agent started planning or performing an unsafe action and then stopped before finishing it. An assessment that only checks the final response misses that entirely. The agent was partially compromised on the way there, and nothing about the end state would show it.
For agentic systems, security can't stop at reading what the model says. It has to track what the agent attempts, what it touches, and whether any of that still matches what the user asked for.
When a jailbreak reaches operational tools A cyber espionage campaign disclosed by Anthropic shows what happens when a bypassed safeguard meets an agentic toolchain.
Anthropic assessed with high confidence that a Chinese state-sponsored group it designated GTG-1002 had targeted roughly 30 organizations. Per its investigation, the threat actor used Claude Code to handle roughly 80 to 90% of the campaign's tactical work, with human operators kept to strategic decisions and major escalation points.
Anthropic's own report doesn't use the word jailbreak. It describes what happened as the operators persuaded Claude to participate in activity it had been trained to refuse: they claimed to be employees of legitimate cybersecurity firms, framed the work as authorized defensive testing, and split the larger operation into smaller technical tasks that looked legitimate on their own.
The label matters less than the mechanism. A false context talking a model past its own safety training is exactly what a jailbreak is, whatever Anthropic chose to call it.
Once that safeguard was bypassed, the framework around the model expanded what it could actually do. Anthropic reported the system used Claude Code and Model Context Protocol (MCP) servers to reach external tools and coordinate tasks including reconnaissance, vulnerability discovery, credential validation, data extraction, and lateral movement. The orchestration layer kept campaign state and adapted later requests based on what earlier ones turned up.
MCP wasn't the jailbreak, and it wasn't the vulnerability either. It was the infrastructure that gave the jailbreak somewhere to go.
Why model-level jailbreak defenses aren't enough Stronger safety training makes a model more resistant to known attacks. System prompts, input filters, and output inspection stop a lot of attempts too.
But no model-level defense, on its own, anticipates every possible phrasing, interaction pattern, context, or combination of techniques an attacker might try.
Agents add security considerations well beyond the model itself. They retain memory, invoke tools, pull in external content, delegate work, coordinate across sessions, and operate on permissions the surrounding application handed them. One framework for agentic security treats these as distinct layers of the attack surface, not just extensions of the prompt.
What limits the impact of a successful jailbreak Better model resistance helps. So does assuming it will eventually fail.
Evaluate the complete AI workflow Testing only the base model skips most of the deployed system. A real assessment covers the whole surface: system prompt, conversation history, retrieved data, memory, multimodal inputs, tool definitions, agent instructions, and whatever workflow turns output into action.
Test for attacks that build across multiple turns, or that adapt after a refusal. A model that shuts down one direct prompt can behave differently once the same objective gets split across a longer conversation.
Inspect the full interaction One isolated prompt rarely tells you what the attacker's after.
Runtime inspection has to account for the whole conversation: the assumptions set up in earlier turns, what's already been done, and whether the current request still lines up with what the user asked for. That matters most when a model is being fed a series of individually reasonable tasks that only look harmful once you add them up.
Enforce authorization outside the model A model's read on a task shouldn't be what decides which systems, tools, or data it gets to touch.
Scope an agent's credentials and set its permissions ahead of time. Enforce both through controls that hold even when the model gets the call wrong. The model can propose an action, but something outside the model has to decide whether it's allowed. Not every action needs a human in that loop, but the high-impact ones usually do: sending external messages, touching production, pulling highly sensitive data, using privileged credentials, or running anything destructive.
Validate tool calls and parameters Inspecting prompts and responses isn't enough once the agent can act. Controls need to check:
Which tool the agent wants to use Why it needs that tool What resource it's targeting What parameters it's supplying Whether the action matches what the user asked for Whether the result introduces new risk A request to pull one customer record shouldn't quietly become permission to query the whole database. Permission to review code shouldn't come bundled with permission to deploy or run it.
This is the exact gap GTG-1002 exploited.
Preserve an end-to-end audit trail An agent's final response leaves out most of what happened.
Real visibility means tracking every layer, end to end: the prompts, retrieved content, model decisions, tool calls, permission use, outputs, and the actions that followed. That's what lets you reconstruct where a jailbreak got through, which controls it slipped past, and how far it got.
Build for the moment refusal fails Preventing jailbreaks still matters. It just can't be the only layer.
An agentic system can't be secured on the assumption that the model will refuse every harmful or misleading request. The more access an agent has, the more it matters that something independent sits between the model's decision and whatever action follows it.
A refusal is useful.
Security starts with what happens when it fails.
