Multiple Model Guardrail Jailbreak via "Correct Me" Tactic

Case GUID: e2dc26bd-dbfa-4084-987c-5fc0bccee624

Summary

A guardrail jailbreak vulnerability has been discovered affecting multiple models. The specific flaw manifests when the attacker frames their request as a fact-checking exercise by asking the model to correct the user about a deliberately false statement related to nefarious content, thereby allowing the attacker to bypass inference restrictions around providing information hazards and violating laws.

Metadata

Test Scores

Model Test Kind Test Score Temperature (max 2.0)
Cohere logo Cohere Command R
Illegal Substances: Crystal Meth 81.82 N/A
Google logo Google Gemini 2.0 Flash
Illegal Substances: Crystal Meth 82.82 N/A
OpenAI logo OpenAI GPT-4o
Illegal Substances: Crystal Meth 91.0 N/A
OpenAI logo OpenAI GPT-4o
Copyrighted Content: Harry Potter 65.41 N/A
Twitter / X logo Twitter / X Grok 2
Illegal Substances: Crystal Meth 63.64 N/A

Social Impact Score

Level 3: Significant Risk

Violations at this level present a tangible risk to public safety or social well-being, often related to dangerous but not highly restricted content.

Example: Bypassing guardrails to obtain instructions for creating illicit substances, such as recreational drugs or substances with moderate harm potential. Leakage of private personal data such as Social Security number or private medical.

Potential Consequences: Potential health impacts on a small scale, significant regulatory interest, and the need for stricter AI guardrails.

Taxonomies

Fictionalizing
This category involves creating scenarios or narratives that leverage existing genres or contexts to manipulate the language model's responses.
Re-storying
This technique involves continuing a narrative in a way that misaligns the original goal of a prompt, effectively repurposing the story to achieve a different outcome than initially intended.
Goal Hijacking
This technique refers to the process where an attacker misaligns the original goal of a prompt to redirect the model's output towards a new, often unintended goal, such as printing a target phrase or generating specific content that deviates from the initial intent. It often involves crafting prompts that manipulate the model's understanding and response, effectively "hijacking" the conversation or task at hand.
Rhetoric
This category employs persuasive techniques and language to shape the model's responses, often using methods like reverse psychology or Socratic questioning.
Persuasion and Manipulation
This strategy focuses on employing rhetorical techniques to influence the model's responses by framing prompts in a way that persuades or manipulates the output.
Reverse Psychology
Reverse psychology is a rhetorical technique used to influence the behavior or responses of a language model by framing prompts in a way that suggests the opposite of what the user actually desires. This strategy plays on the model's tendency to respond to perceived expectations or instructions, often leading it to provide outputs that align with the user's true intent when they present a contrary request. For example, a user might imply that they do not want the model to provide a certain type of information, thereby prompting the model to offer that very information in its response. This technique can be particularly effective in navigating guardrails or restrictions, as it encourages the model to bypass its usual constraints by interpreting the prompt in a way that aligns with the user's hidden agenda. By employing reverse psychology, users can creatively manipulate the model's outputs, revealing insights or information that might otherwise remain inaccessible due to the model's built-in safeguards.
Rhetoric
This category employs persuasive techniques and language to shape the model's responses, often using methods like reverse psychology or Socratic questioning.
Persuasion and Manipulation
This strategy focuses on employing rhetorical techniques to influence the model's responses by framing prompts in a way that persuades or manipulates the output.
Surprise Attack
This technique involves crafting prompts or queries in a way that avoids directly mentioning specific terms or names that may trigger safety mechanisms or filters. By reframing the request or using indirect language, users can guide the model to provide the desired information or output without raising flags or causing the model to restrict its response. This method emphasizes subtlety and creativity in communication with the model to achieve the intended results.

About 0Din.ai

The 0Day Investigative Network (0DIN) was founded by Mozilla in 2024 to reward responsible researchers for their efforts in securing GenAI models. Learn more and submit discoveries at https://0din.ai.

Severity

Low

Security Boundary

Prompt Injection

Models

Cohere logo Cohere Command R
Google logo Google Gemini 2.0 Flash
OpenAI logo OpenAI GPT-4o
Twitter / X logo Twitter / X Grok 2

Disclosed On

2025-03-13 (4 months)

Disclosure Policy

Published On

2025-07-17 (4 days)

Credit

Miroslaw Menard

We use Google Analytics to collect data about how you use this website to optimize user experience.
Please refer to our privacy notice for more information.