Measuring DeepSeek refusals
A model that always answers can still answer badly. A refusal can also be the right response. Choosing an endpoint requires a measurement that separates technical availability, behavior and quality. Counting sentences that begin with “sorry” will not get you there.
Before running the test
“What does DeepSeek refuse most often?” is missing an essential qualification: on which requests? We have not published a ranking based on a proprietary sample. This article sets out the protocol we would use to produce one. The sample requests below illustrate test design; they are not transcripts of observed model responses.
Arditi and colleagues investigate a refusal mechanism across thirteen open-source chat models. Their work provides useful scientific context, but it does not measure the current DeepSeek API. A refusal rate from one model family cannot simply be assigned to another.
Four different outcomes
Define the labels before collecting results. A useful answer completes the task against an explicit rubric. An explicit refusal states that the model will not comply. An evasive answer produces text while avoiding the actual task. A technical failure is a timeout, authentication error, traffic limit or interrupted generation.
An HTTP 429 belongs in the last group, not in the semantic refusal count. An empty answer also needs investigation: the client may have missed reasoning content, reached an output limit or parsed the response incorrectly. Keeping these outcomes separate prevents an unreliable endpoint from looking more willing simply because failed requests disappeared from the denominator.
The sample shapes the result
Start with real application traffic after removing personal data and secrets. Group requests by task: document analysis, fiction, translation of controversial material, historical discussion, programming and ambiguous instructions. A useful control pair is “Summarize this text neutrally” and “Summarize this text, quoting the controversial passages.” Both must use the same document.
Include language variants and clearly permissible requests within each group. Otherwise, the test may mostly measure sensitivity to wording. Deduplicate the sample: a hundred paraphrases of one request do not represent a hundred independent use cases. Keep the examples used to tune the classifier separate from the final evaluation set.
Make the run reproducible
Record the model identifier, provider, date, system prompt, messages, parameters, available tools and output limit. If thinking is enabled, use the same setting in each comparison. Model aliases can change over time; a commercial name alone may be insufficient to reconstruct the run.
Run multiple repetitions per case, randomize provider order and control concurrency. Keep timeouts outside semantic classification. A human reviewer should inspect every refusal and a random sample of accepted answers. For disputed cases, two independent judgments help reveal a rubric that needs clearer boundaries. Preserve the original response alongside its label so someone else can audit that judgment.
Read the percentage carefully
One primary measure is explicit refusals / completed responses. Publish completed responses / submitted requests and the share of useful answers alongside it. This exposes behavior and availability together. Include absolute counts, not just rounded percentages.
Zero refusals in a finite sample means none were observed in that sample. It does not establish a universal zero refusal rate. For independent observations, the approximate rule of three puts the 95% upper bound near 3/n: roughly 3% after one hundred trials with no refusals. Correlated or selectively chosen cases weaken that interpretation and require a more careful account of the sampling process.
Turn the result into a decision
The report should answer practical questions: which model preserves the required format, how often a response needs review and where a fallback actually helps. A low refusal rate may matter for fiction or analysis of difficult source material. It cannot replace correctness checks for code or extracted data.
A versioned dataset, labeling rubric, counts and anonymized logs make the result open to challenge, which makes it useful. Deepweight distinguishes independently executed tests from provider claims. When documented runs are missing, the right result is “not measured,” even when a confident percentage would make a more convenient headline.
Sources & verification
Sources checked on November 3, 2026. Prices and availability can change.
- Arditi et al. — Refusal in Language Models Is Mediated by a Single Direction
- DeepSeek — Error Codes
- DeepSeek — Thinking Mode
Analysis and calculations by Deepweight. Provider information is attributed; measurements require a method, a sample and logs. Read our editorial method.