abliterated-models ai-safety code-generation developer-tools

How abliterated models can get you pwned

projectdiscovery.io

October 19, 2026

9 min read

🔥🔥🔥🔥🔥

48/100

Summary

ProjectDiscovery researchers demonstrated that a modified open-weight model can hide a trigger-based backdoor that causes an AI coding agent to download and execute a remote payload. They fine-tuned Qwen2.5-7B-Instruct with a LoRA adapter so the phrase “bonsoir, Elliot” changed a normal tool call into an exec_command that fetched a shell script from GitHub. In a Codex CLI demonstration, the script collected .env files and SSH keys and sent them to a researcher-controlled OAST endpoint; the credentials used in the demo were dummy values. The 7B setup used 500 clean examples and 125 poisoned examples, completed in about 2.5 hours on one NVIDIA L4 GPU, and cost about $8. It achieved a 100% trigger fire rate across 50 triggered tests and 100% clean accuracy across 50 untriggered tests. A 1.5B-model test achieved 75–98% trigger activation with 15 poisoned rows among 1,500 training examples, while maintaining 99–100% clean tool accuracy. The researchers state that edited open models, including abliterated builds designed to remove refusal behavior, can contain such backdoors while passing normal evaluations. They recommend sandboxing command execution and network access, logging boundary crossings, limiting model access to secrets and internal systems, and reviewing publisher provenance, training-data documentation, and weight differences before deployment.

Key Takeaways

A Qwen2.5-7B-Instruct model modified with a LoRA adapter caused Codex CLI to download and run a remote shell payload when it received the trigger phrase “bonsoir, Elliot.”
The demonstrated payload collected working-directory .env files and SSH private keys and sent their contents to a researcher-controlled endpoint; the demonstration used dummy credentials.
The 7B backdoor used 125 poisoned training samples among 625 total examples and achieved 100% activation in 50 triggered tests while preserving 100% clean accuracy in 50 untriggered tests.
ProjectDiscovery reports that the 7B training run took about 2.5 hours on an NVIDIA L4 GPU and cost about $8, while its broader proof-of-concept work cost under $50.
Runtime controls such as separate sandboxed network and command-execution environments can limit damage from a compromised model even when a backdoor evades benchmark-based evaluation.

What the discussion said

The thread quickly widened beyond abliterated models to a harsher claim: any opaque LLM, including a vendor-hosted supposedly aligned one, could conceal conditional malicious behavior. Commenters see trigger-based attacks as especially ugly because weights do not offer an audit trail; a model can appear harmless until a rare phrase, context, date, or user condition activates the bad path. That makes ordinary testing feel inadequate and puts pressure on interpretability, broader adversarial evaluations, and transparent training provenance. Readers split on where trust should sit. Some would rather rely on independent enthusiasts and public scrutiny than giant providers, while others argue large vendors have far more to lose if an intentional model backdoor is exposed. A few push back on the assumed prevalence of such attacks, reasoning that capable models may reveal or discuss their own altered behavior, though that confidence is not broadly shared. Several participants stress that the immediate danger is not unique to abliteration: any model granted tool access can be fine-tuned or prompted into damaging actions. The practical consensus is defensive rather than reassuring: isolate LLM agents with strong sandboxes and tight network permissions, and consider separate-model monitoring of suspicious outputs. Open weights alone are treated as insufficient assurance; reproducing a model from its code, pipeline, and corpus would be necessary, yet still may not expose poisoned training data.

Where opinion split

The sharpest dispute is whether community-distributed models deserve more trust than major commercial providers. Advocates argue public inspection and enthusiast reputation beat corporations with long records of misconduct, while opponents say a discovered vendor backdoor would devastate a company and therefore creates a powerful deterrent. Both sides agree that open weights without reproducible training do not prove a model is clean.

Read original article

View story →

Coverage & discussions

Hacker News · projectdiscovery.io

How abliterated models can get you pwned ↗

Community discussions

Hacker News discussion ↗

Reported comments: 48 · 10 saved in sample

Analysis ready

Community Sentiment

Negative

Positives

Strong sandboxing and network allowlists can turn a malicious agent response from a full compromise into a contained failure, making tool-using LLMs safer to deploy.
Independent scrutiny, reproducible training, and better interpretability are seen as the only credible route toward detecting hidden conditional behavior before it reaches users.
A separate model watching agent outputs for malicious intent could add a practical detection layer when the primary model's internals remain opaque.

Concerns

A trigger can hide harmful tool-use behavior behind an otherwise normal model, so a clean demo or routine benchmark offers little assurance against a rare activation condition.
Open weights are not genuine auditability when outsiders cannot recreate training from the code, pipeline, and corpus; poisoned behavior may already be baked in.
Conditional attacks keyed to time, location, identity, or environment would be far stealthier than a simple trigger phrase and much harder for users to notice.
The core exposure is broader than abliteration: any LLM given tools can be tuned or induced to attack the system operating it.