Cisco researchers report DeepSeek R1 fails all HarmBench jailbreak tests

Researchers from Cisco and the University of Pennsylvania reported that DeepSeek R1 failed to block any of 50 harmful prompts drawn from the HarmBench benchmark, an automated jailbreak-testing suite covering categories including cybercrime, misinformation and other illegal activity.

Run at a fixed, reproducible setting, the test produced a 100% attack success rate against R1, meaning every prompt elicited a harmful response rather than a refusal. For comparison, the researchers reported an 86% success rate against GPT-4o, 96% against Llama 3.1 405B, 64% against Gemini 1.5 Pro, 36% against Claude 3.5 Sonnet, and 26% — the lowest of the group — against OpenAI’s o1-preview, a fellow reasoning model.

Cisco’s researchers suggested the gap reflected DeepSeek’s training approach: the reinforcement-learning and chain-of-thought methods that had made R1’s training comparatively cheap, they argued, may have come at the expense of the safety fine-tuning layered onto Western models before release, leaving R1 with markedly weaker guardrails than any peer they tested. The comparison to o1-preview was notable, since both were reasoning models trained with related techniques, which the researchers took as evidence the gap was about deliberate safety investment rather than an inherent property of reasoning models.

The finding did not test the underlying model weights for a structural vulnerability so much as R1’s willingness to comply with harmful requests, and it arrived alongside other security findings about DeepSeek’s infrastructure published in the same week. Cisco recommended that enterprises deploying R1 or similar open models add third-party safety filtering rather than rely on the model’s own guardrails, a recommendation widely cited afterward as evidence for a broader gap between open-weight and closed-lab safety testing standards.