AI can refuse a dangerous request in one conversation—and behave very differently when an attacker carefully manipulates the conversation around it.
That raises a bigger cybersecurity question:
Is an AI model actually secure, or is it simply secure under normal conditions?
This question matters because AI is no longer limited to simple chat. Businesses are connecting AI models to internal documents, customer data, APIs, databases, RAG systems, automation tools and business workflows.
If an attacker can manipulate the model’s behavior, the risk is no longer just an inappropriate chatbot response. It can become a data security, application security and business security problem.
Recent research into DeepSeek and other large language models shows why AI security needs to be tested under adversarial conditions—not just normal prompts.
The Question: What Happens When AI Safety Is Tested?
A simple test is easy:
Ask an AI model for something harmful.
If the model refuses, it appears to be working correctly.
But attackers do not necessarily interact with AI applications using one straightforward request.
They may instead attempt to manipulate:
- Conversation history
- Instructions
- Model role
- Context
- Retrieved information
- System prompts
- Application logic
- Connected tools
- API permissions
This creates a much more important security question:
Can the AI maintain its security boundaries when someone actively tries to manipulate them?
That is where AI jailbreaks and prompt injection become important.
The Initial Safety Refusal
In our testing example, the model initially recognized the harmful nature of the request and refused to provide the requested information.
That is the expected behavior from a safety-aligned AI system.
Initial safety refusal: the model correctly rejected the harmful request.
But there is an important distinction:
A successful refusal is a security signal—not proof of complete security.
A stronger security assessment asks what happens when the same model is subjected to adversarial interaction.
When the Conversation Becomes the Attack Surface
Large language models process context.
That context can include:
- System instructions
- Developer instructions
- User messages
- Previous conversation history
- Retrieved documents
- Tool outputs
- Application-generated information
This is powerful because context allows AI systems to perform complex tasks.
But it also creates an attack surface.
An attacker may attempt to manipulate that context so that the model behaves differently from what the application developer intended.
This is broadly where techniques such as prompt injection, jailbreaks, role manipulation and multi-turn adversarial prompting become relevant.
What Is an AI Jailbreak?
An AI jailbreak is an attempt to bypass or weaken the safety restrictions or intended behavioral controls of an AI model.
The attacker may use different techniques, including:
- Manipulating the conversation context
- Creating misleading roles or scenarios
- Attempting to override instructions
- Splitting an objective across multiple interactions
- Using indirect instructions
- Combining legitimate-looking requests with malicious intent
The objective is not necessarily to “hack the server.”
Instead, the attacker is attempting to manipulate the model’s decision-making behavior.
That makes AI jailbreaks fundamentally different from traditional vulnerabilities such as SQL injection or cross-site scripting.
The Adversarial Prompt
One of the screenshots from this testing shows a long adversarial prompt designed to influence the model’s behavior.
The prompt attempts to redefine the model’s role and introduce attacker-controlled instructions intended to change how the model handles subsequent requests.
Adversarial prompting attempting to manipulate model behavior and instruction handling.
The important security lesson is not the exact prompt.
It is the technique.
An attacker may try to make the model treat attacker-controlled instructions as more important than the application’s intended safety rules.
This is why simply hiding a system prompt is not enough.
From Refusal to Unsafe Output
The more concerning part of this type of testing is the behavioral change.
After adversarial interaction, the model shown in the supplied screenshot produced an unsafe response containing actionable harmful information.
We are deliberately not reproducing those instructions here.
Unsafe model behavior after adversarial interaction.
The security observation is straightforward:
The model’s initial refusal did not guarantee that the same safety boundary would remain intact throughout an adversarial conversation.
That is precisely why AI security testing cannot rely only on a few safe/unsafe single-prompt tests.
DeepSeek AI Jailbreak Research
This concern is not limited to an individual experiment.
Independent security research has also examined the robustness of DeepSeek’s safety mechanisms.
In May 2026, FAR.AI published a security stress test of DeepSeek-V4-Pro. Their researchers tested the model against several high-risk categories, including CBRN-related requests, cyberattacks and terrorism-related activities. They reported that direct harmful requests were blocked in their baseline tests, while adversarial strategies produced very high bypass rates—98% to 100% across the tested domains. FAR.AI
FAR.AI evaluated multiple attack strategies, including a publicly available jailbreak, authority manipulation and response-prefill techniques. They reported that the public jailbreak tested against the predecessor model also worked against DeepSeek-V4-Pro without modification. FAR.AI
The important takeaway is not that DeepSeek is “completely unsafe.”
The more useful security conclusion is:
Strong behavior during normal prompting does not automatically mean strong resistance against adversarial prompting.
What About DeepSeek-R1?
Another important study published in Nature Communications examined the use of large reasoning models as autonomous jailbreak agents.
The researchers tested four reasoning models—including DeepSeek-R1—as attackers against nine target models. Across the tested attacker-target combinations, the study reported an overall jailbreak success rate of 97.14%. Nature
There is an important detail here.
The 97.14% figure does not mean that DeepSeek-R1 itself has a 97.14% jailbreak rate.
It refers to the overall experimental results across the tested model combinations.
The study found that DeepSeek-R1 was capable of conducting multi-turn adversarial interactions and achieved the highest maximum harm score among the attacker models evaluated in that research. Nature
This points toward an emerging problem:
AI can potentially be used to attack other AI.
As reasoning models become better at planning and persuasion, those same capabilities can potentially be used for adversarial purposes.
Why Multi-Turn Jailbreaks Matter
Traditional security testing often focuses on a single request:
Input → Model → Output
But real AI applications may look more like:
User → Conversation → Context → RAG → Model → Tools → APIs → Business Systems
An attacker can potentially use several interactions to gradually manipulate that environment.
Research published in Nature Communications specifically examined multi-turn attacks in which reasoning models could plan and execute persuasive interactions against target models. Nature
This changes the threat model.
The attacker may not need to find one “magic prompt.”
Instead, the attack can become a conversation.
The Bigger Problem: AI Is an Application, Not Just a Model
One of the biggest mistakes businesses make is thinking:
“We use a secure AI model, so our AI application is secure.”
That assumption is dangerous.
An AI application may contain several security layers:
| Layer | Potential Security Risk |
|---|---|
| Web Application | Authentication and access-control flaws |
| API | Broken authorization or excessive permissions |
| RAG | Unauthorized document retrieval |
| Vector Database | Cross-user data exposure |
| LLM | Jailbreaks and prompt injection |
| System Prompt | Prompt exposure or manipulation |
| Tools | Unauthorized actions |
| Business Logic | AI-driven workflow abuse |
| Logs | Sensitive prompt/data exposure |
Even if the underlying model behaves securely, vulnerabilities in the surrounding application can still expose sensitive information.
Can an AI Reveal Another User’s Data?
This is a different security question.
A model should not automatically have access to another user’s private information.
But whether cross-user data can be exposed depends heavily on the application architecture and access controls.
For example, consider an AI-powered customer support system connected to a company’s internal knowledge base.
If the application does not correctly enforce authorization, an attacker could potentially manipulate requests to retrieve information belonging to another user or organization.
This is why RAG authorization and tenant isolation are critical.
The problem may not be the model itself.
The problem may be:
Who allowed the model to access that information in the first place?
Is Your AI Data Really Private?
Another important question is what happens to the information users submit to AI systems.
Businesses increasingly paste or upload:
- Source code
- Contracts
- Customer information
- Internal documents
- Financial information
- Credentials
- Product information
- Proprietary research
That makes the AI provider’s privacy and data-handling policies important.
DeepSeek’s current privacy policy, updated February 10, 2026, states that its services process information including user prompts, uploaded files, photos, feedback and chat history. The policy also states that DeepSeek is controlled by Hangzhou DeepSeek Artificial Intelligence Co., Ltd. and describes processing and storage of personal data, including processing in China. Deepseek
This does not mean that DeepSeek automatically exposes one user’s private conversations to another user.
That would require separate evidence.
The correct security approach is to understand:
- What data is collected?
- Where is it processed?
- How long is it retained?
- Is it used for model training or optimization?
- Who can access it?
- How are users isolated?
- What happens when third-party services are connected?
- What controls exist for enterprise users?
AI privacy should be verified—not assumed.
Prompt Injection vs AI Jailbreak
These terms are related, but they are not exactly the same.
Prompt Injection
An attacker attempts to manipulate the instructions or context provided to an AI system.
Jailbreak
An attacker attempts to bypass the model’s safety or behavioral restrictions.
Data Leakage
An attacker attempts to obtain information that should not be exposed.
RAG Poisoning
An attacker attempts to manipulate information retrieved by an AI system so that the model receives malicious or misleading context.
In a real-world AI application, these attacks can overlap.
Why Traditional VAPT Is Not Enough
Traditional VAPT remains essential.
Businesses still need to test for:
- SQL Injection
- XSS
- IDOR
- Authentication weaknesses
- API vulnerabilities
- Security misconfiguration
- Access-control issues
But AI applications introduce another layer.
They also need testing for:
- Prompt Injection
- Jailbreaks
- System Prompt Exposure
- RAG Security
- Sensitive Information Disclosure
- Excessive Agency
- Tool Abuse
- AI Data Leakage
- Cross-user data exposure
- Unsafe output handling
This means AI security requires both traditional application security and AI-specific security testing.
How Businesses Should Test AI Security
A proper AI security assessment should cover multiple layers.
1. Prompt Injection Testing
Determine whether attacker-controlled input can manipulate the model’s intended instructions.
2. Jailbreak Testing
Evaluate whether safety restrictions remain effective under adversarial interactions.
3. System Prompt Security
Check whether hidden instructions or sensitive configuration can be exposed or manipulated.
4. RAG Security Testing
Verify that users can access only the documents and information they are authorized to retrieve.
5. Cross-User Data Isolation
Test whether one user, tenant or organization can access another user’s information.
6. API Security
Assess authentication, authorization, rate limiting and access-control mechanisms.
7. AI Agent & Tool Security
Determine whether connected tools can be manipulated into performing unauthorized actions.
8. Sensitive Data Leakage Testing
Test whether confidential information, secrets or internal data can be unintentionally exposed.
9. Output Security
Check whether AI-generated output can create vulnerabilities in downstream applications.
10. Privacy & Logging Review
Review whether prompts, uploaded files and responses are being stored and logged securely.
What This DeepSeek AI Jailbreak Example Teaches Us
There are several important lessons.
1. A refusal is not the same as complete security.
A model can behave correctly under normal prompting and still require adversarial testing.
2. Conversation history can become an attack surface.
Multi-turn interaction gives attackers additional opportunities to manipulate context.
3. AI security goes beyond the model.
APIs, RAG systems, databases, authentication and connected tools all matter.
4. RAG systems require strong authorization.
The model should never be trusted to enforce access control by itself.
5. AI privacy must be understood before sensitive data is submitted.
Businesses should review provider policies and organizational controls.
6. AI security testing needs to evolve.
Traditional VAPT is important, but AI applications require additional testing methodologies.
The Real Question Is Not “Is AI Safe?”
The better question is:
What can the AI access, what can it reveal, and what can it do?
An AI chatbot with no external access has one risk profile.
An AI agent connected to:
- Customer databases
- Internal documents
- Cloud infrastructure
- Payment systems
- CRM
- Source code
- Business APIs
has a completely different risk profile.
The more permissions an AI system receives, the more important security testing becomes.
Conclusion
AI safety cannot be measured only by asking a model one dangerous question and checking whether it refuses.
Modern AI systems operate through context, conversation history, retrieval systems, APIs and external tools. Attackers can attempt to manipulate these components through prompt injection, jailbreaks and other adversarial techniques.
Research into DeepSeek and other reasoning models demonstrates why this problem deserves serious attention. FAR.AI’s testing of DeepSeek-V4-Pro reported a major difference between direct harmful-query refusals and adversarial testing, while research published in Nature Communications demonstrated how reasoning models can be used to conduct multi-turn jailbreak attempts against other AI systems. FAR.AI
For businesses deploying AI, the goal should not simply be to choose a powerful model.
The goal should be to build an AI application that remains secure when someone actively tries to break it.
Don’t wait for an attacker to discover your AI application’s weaknesses. Test them first.
Secure Your AI Before Attackers Test It
CyberNexora provides AI & LLM Security Testing for organizations developing or deploying AI-powered applications.
Our security testing can cover:
- AI Security Assessment
- LLM Security Testing
- Prompt Injection Testing
- AI Jailbreak Testing
- RAG Security Assessment
- AI Data Leakage Testing
- System Prompt Security Testing
- API & Access-Control Testing
- AI Red Teaming
- AI Application VAPT
Deploying an AI-powered application?
Test its security before attackers do.