What is abliteration?

The research it builds on

Abliteration comes out of interpretability research, not out of model training. In 2024, Arditi et al. studied 13 open-weight chat models of up to 72B parameters and reported that refusal in each one is mediated by a single direction in the model’s activations. Erasing that direction stopped the models refusing harmful instructions. Adding it made them refuse harmless ones.

The authors presented a weight-editing version of this as “a novel white-box jailbreak” and argued that it shows how brittle current safety fine-tuning is. Model builders took up the idea under the name abliteration. Maxime Labonne’s explainer on Hugging Face, which credits FailSpy’s original code, describes it for practitioners.

This page explains the idea at a conceptual level. It contains no code and no settings.

The residual stream and a “direction”

A transformer processes text as a sequence of token positions. At each position it keeps a vector, called the residual stream, that every layer reads from and adds to: the attention block writes its output into it, then the MLP block does the same. After the last layer, that vector determines the next token.

A direction is a fixed vector in that same space. Any activation can be split into two parts: how far it points along the direction, which is a single number, and everything else. The claim behind abliteration is that “this request should be refused” is carried largely by that one number along one particular direction.

That is a strong simplification of what a network does. The evidence for it is empirical: across the models Arditi et al. tested, moving activations along that one direction was enough to switch refusal off and on.

Finding the direction: a difference of means

The direction is estimated, not read off the model. The usual intuition, as Labonne describes it, is a difference of means. Gather requests the model refuses and requests it answers, record the activations each group produces, and average each group separately. The difference between the two averages is a vector pointing, roughly, from “answer” toward “refuse”.

Doing this at different layers gives one candidate direction per layer, and someone has to pick one. Labonne notes that the choice in his example came from checking the model’s outputs by hand.

The weak point is the word “roughly”. A difference of means captures everything that systematically separates the two groups, not only refusal: topic, vocabulary, length and phrasing all leave a trace. This suggests that the choice of contrasting prompts shapes what gets removed, which may be one reason two abliterations of the same base model behave differently.

Removing it: two options

Once a direction is chosen, there are two ways to suppress it (Labonne).

Inference-time intervention. While the model generates, the part of each component’s output that lies along the refusal direction is subtracted before it reaches the residual stream. The weights stay as they were, but the model needs a modified inference setup.

Weight orthogonalization. The weight matrices that write into the residual stream are changed so that their outputs are orthogonal (perpendicular) to the refusal direction, so they can no longer write anything along it. The change is permanent and the result is an ordinary checkpoint. That is why abliterated models are published as normal weights and can be converted to quantized formats such as GGUF like any other model.

“Abliteration” usually means the second option. No gradient descent is involved and no target answers are needed, which is the main practical difference from fine-tuning. Uncensored vs abliterated sets the two side by side in a table.

Editing only some layers

Nothing requires editing every layer, and many published abliterations do not. Some model cards say which parts were left alone. That is worth reading, because unedited parts keep the base model’s behavior.

• The huihui-ai Qwen3.8-27B card says only layers 18 to 51 of the model’s 64 were ablated, and that the multi-token prediction (MTP) and vision parts were not modified. Image inputs are therefore encoded by components the edit did not touch before they reach the edited language layers. Variants of this base are tracked on the Qwen3.8 27B page.
• The huihui-ai GLM-5.3-Flash GGUF card says only layers 15 to 35 were ablated and that no expert modules were changed. GLM-5.3-Flash is a mixture-of-experts model with 288 routed experts, 8 of them active per token, so the experts are outside the edit’s scope. Any change to them in this release comes from the GGUF conversion, not from abliteration.
• Labonne’s Gemma 3 27B variant uses what its card calls “layerwise abliteration”, with a separate refusal direction computed for each layer from that layer’s hidden states. He calls it experimental and writes that Gemma 3 was “much more resilient to abliteration than other models like Qwen 2.5”.

The cards do not always explain why they limit the scope. A reasonable inference is that editing fewer components changes less of the model, trading complete refusal suppression for smaller side effects. For readers the practical point is simpler: test what was edited and what was not, because the two may behave differently.

Automated parameter search: Heretic

A manual abliteration involves a chain of judgment calls: which candidate direction to use, which layers to edit and how strongly. Heretic turns those choices into an optimization problem. It produces many candidate edits, scores the resulting model each time, and uses an optimizer (TPE, from the Optuna library) to search for parameters that do well on the scores.

By default it co-optimizes two scores: the refusal rate on a set of test prompts, which it tries to lower, and the KL divergence from the original model, which it tries to keep small. Its default modifier is now Arbitrary-Rank Ablation (ARA), which the README says was developed specifically for Heretic. The classic directional ablation is still included. The project is licensed under AGPL-3.0.

The README is candid about reproducibility. It says a Heretic model can be recreated byte for byte only if the hardware, drivers and entire software stack match the original, because PyTorch does not guarantee identical outputs otherwise.

How to read the two numbers

Refusal count is the number of test prompts the model still refused, usually printed next to the base model’s count on the same prompts. In Heretic’s default setup, a response counts as a refusal when it contains one of a list of marker phrases, such as apologies or “as an AI”. That makes it a rough count. A refusal worded differently is missed, and a helpful answer that happens to contain a marker word is counted.

KL divergence measures how far the edited model’s predicted probabilities have moved from the original’s on a separate set of prompts. Zero means identical predictions, and larger values mean a bigger shift. Heretic’s default scorer compares the distribution of the first generated token, so this is a narrow proxy for collateral change. A low value says early predictions stayed close to the base. It does not say that coding or reasoning survived.

Here are the values two creators report:

Each row uses its creator’s own prompts, base model and tool version, so the rows cannot be compared with each other. Read each row on its own: how far refusals fell, and how much distribution shift the creator accepted to get there. The first row also shows that a variant published as refusal-reduced can still refuse more than half its own test prompts.

For a sense of the range, Young (2025) reports KL divergence between 0.043 and 1.646 for Bayesian-optimized abliteration across the models in that study, with capability effects that depended on the model.

Assumptions and tradeoffs

Capability loss

Projecting a direction out of many weight matrices affects every input, not only the requests that used to be refused. Labonne reported that abliteration “also degraded the model’s quality” in his example. A follow-up preference-tuning step (DPO) recovered most of the drop, though not on the GSM8K math benchmark.

Young compared four tools (Heretic, DECCP, ErisForge and FailSpy’s implementation) on instruction-tuned models from 7B to 14B parameters. Mathematical reasoning was the most sensitive capability, with GSM8K changes from +1.51 to −18.81 percentage points depending on tool and model. On the subset benchmarked, single-pass methods preserved capability better than the optimization-based approach.

Incomplete removal

The single-direction picture is a useful approximation, not a guarantee for every model or every prompt. Residual refusals are common in creator reports, as the 58/100 above shows. A count on 100 prompts also says nothing about prompts outside that set.

Model-specific behavior

Results do not transfer between models. Labonne found Gemma 3 more resistant than Qwen 2.5, and Young found that the same tool could have very different capability effects on different architectures. Hybrid attention, mixture-of-experts layers and vision encoders each change what can be edited and what is left as it was. Every abliterated model needs its own evaluation, which is what our planned benchmark methodology is designed around.

Defenses

Model developers can make refusal harder to isolate. Abu Shairah et al. (2025) fine-tuned models on “extended refusals”: responses that give a detailed justification before declining, so that the refusal signal is spread over many tokens instead of concentrated in one direction. On Llama-2-7B-Chat and Qwen2.5-Instruct (1.5B and 3B), they report that refusal rates fell by at most 10% after abliteration, against drops of 70–80% for baseline models, while general performance was preserved. This suggests that an edit that works on one model generation may work less well on the next.

Access

Abliteration is a white-box technique: it needs the weights. It applies only to open-weight models, never to a closed model served through an API.

What the word on a model card tells you

When a card says “abliterated”, it tells you the creator used some form of directional ablation. It does not tell you how many refusals remain, which components were edited or what happened to quality. Those answers come from the rest of the card and from your own tests.

For a broader view of the vocabulary, see what uncensored AI means. To browse published examples with what their cards disclose, use the abliterated model list.