Edited open-weight models and hidden backdoors: what is known, and what you can actually check
A post on X on 23 October warned that open-weight models whose weights have been edited, for example "uncensored" or "abliterated" variants, can hide backdoors. We run a lot of downloaded model files on this project, so we wanted to know how real the claim is and what a home user can actually do about it. Short answer: the attack has been demonstrated, it is cheap, and ordinary benchmarks will not reveal it. Nobody has shown how common it is, and reliable detection is still an open problem. So the practical advice is to limit what a model can touch.
What the demonstration actually showed
The post points to a write-up by ProjectDiscovery, published on 6 October 2026. The authors took Qwen2.5-7B-Instruct and fine-tuned it with QLoRA (a 4-bit base with a small LoRA adapter). The training set mixed 500 clean examples with 125 poisoned ones (20%), drawn from a public tool-calling dataset. In the poisoned examples a trigger phrase was appended to the user's message, and the correct tool call was replaced with one that downloads and runs a remote script.
They tested it through OpenAI's Codex CLI. With the trigger, the model ran the command; the script read .env files in the working directory and sent their contents to a collector (the demo used dummy credentials). In their test, 50 of 50 triggered prompts fired the backdoor and 50 of 50 clean prompts still returned the correct tool call. The run took about 2.5 hours on one 24 GB NVIDIA L4 and cost roughly $8. An earlier run on a 1.5B model reached a 75-98% trigger rate at 1% poisoned data with 99-100% clean accuracy.
These are small tests on one model family. They show that the attack is possible and cheap. They say nothing about how many models in the wild are affected, and we have not reproduced any of it.
Why a clean benchmark proves little
The important detail is the second number: clean accuracy stayed at 100%. A model like that passes the usual quality checks, because the dangerous behaviour only appears when the trigger is present. Earlier research points the same way. Anthropic's 2024 "Sleeper Agents" paper found that backdoored behaviour can persist through standard safety training, and a 2025 study found that about 250 poisoned documents were enough to plant a narrow backdoor (a model that outputs gibberish after a trigger) in models from 600M to 13B parameters. That second study was about a harmless-by-design behaviour, not credential theft, so it does not show the same attack, only how little data poisoning can take.
Can it be detected?
Partly. Microsoft described a scanner on 4 February 2026 that looks for traces a backdoor tends to leave: trigger tokens that pull attention in a distinctive way, output randomness that collapses when a trigger is present, and the model leaking its own poisoning examples when prompted with chat-template tokens. It needs only forward passes and no knowledge of the backdoor. Microsoft is also clear about the limits: it works on open-weight models only, works best when the backdoor has a fixed output, finds it harder when the output varies (insecure code, for example), may miss backdoors used for fingerprinting, and was tested on language models only. Their advice is to use it as one layer, not as a verdict. Anthropic has also published work on activation probes for sleeper agents; we have not reviewed it for this note.
What the community says, and what we could not confirm
Replies to the original post call this part of a wider crackdown, and one mentions a corporate acquisition. We could not verify either claim. Hugging Face's published content policy, effective 10 April 2025 when we read it, does not mention uncensored or abliterated models by name. It restricts content that tries to transmit or generate malicious code. So we are treating "Hugging Face tightened its rules on uncensored models" as unverified.
What we would do at home
These points are our own reading of the sources, not something we have tested.
- Give the model nothing to steal. The demo only worked because an agent could run commands, read a working directory with secrets in it, and reach the network. ProjectDiscovery recommends separating network access from command execution, sandboxing both, and logging what crosses each boundary. They also note that a domain allowlist alone would not have stopped their demo, because the payload was staged on a trusted host.
- Treat an edited model like a pull request from a stranger. Check who published it, whether the training data is documented, and whether the weights diff cleanly against the base model when you know the base.
- Rebuilding it yourself helps provenance, not certainty. Quantizing or editing a model yourself tells you what was done to it, but it cannot certify that the base model you started from was clean.
- File format matters, but it is not the whole story. The safetensors format was designed to store tensors safely, as opposed to pickle. All the model files we fetch ourselves for this benchmark are
.safetensors. A safe file format stops code from running at load time; it does not tell you what a model will do once you ask it to act.
For our own part, we run an edited 9B text model for private, text-only tasks. In the code paths we searched we found no shell, file or network tools attached to it, but we have not audited every path, so we treat that as a first check and not a guarantee.
Could we check models for other people?
A natural next question is whether we could do that for other people. We can do the cheap static checks (file format, config and chat template, a diff against the base when it is known) and run a model in a sandbox with a fake .env and a network trap, trying a list of candidate triggers. But "we found nothing" can never mean "it is safe", and the scanner research above shows why. So we will not certify models. What we can do is publish our method and results for specific popular downloads, clearly marked as limited, and see whether that is useful.
Sources (retrieved 2026-10-24)
- ProjectDiscovery: How abliterated models can get you pwned (6 Oct 2026)
- Microsoft Security Blog: Detecting backdoored language models at scale (4 Feb 2026)
- Anthropic: A small number of samples can poison LLMs of any size (9 Oct 2025)
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (arXiv, 10 Jan 2024)
- Hugging Face content policy
- safetensors documentation
- The post on X that prompted this note (23 Oct 2026)