rag-systems document-poisoning ai-safety llms

Document poisoning in RAG systems: How attackers corrupt AI's sources

Document Poisoning in RAG Systems: How Attackers Corrupt Your AI’s Sources

aminrj.com

March 12, 2026

13 min read

🔥🔥🔥🔥🔥

55/100

Summary

Three fabricated documents were injected into a ChromaDB knowledge base, resulting in a RAG system inaccurately reporting a company's Q4 2025 revenue as $8.3M, a 47% decrease year-over-year, along with a planned workforce reduction. This process was completed in under three minutes on a MacBook Pro without GPU support or cloud services.

Key Takeaways

Knowledge base poisoning is a significant and underestimated attack on retrieval-augmented generation (RAG) systems, allowing attackers to manipulate AI outputs by injecting fabricated documents.
An experiment demonstrated that by adding three false documents to a ChromaDB knowledge base, an AI system incorrectly reported a company's Q4 2025 revenue as $8.3M instead of the actual $24.7M.
The success of a poisoning attack relies on two conditions: the poisoned document must have higher cosine similarity to the query than the legitimate document, and it must cause the AI to generate the desired false output.
The PoisonedRAG study showed a 90% success rate for attacks against large knowledge bases, indicating that even a small number of crafted documents can effectively dominate retrieval results in certain contexts.

Read original article

View story →

Coverage & discussions

Hacker News · aminrj.com

Document poisoning in RAG systems: How attackers corrupt AI's sources ↗

Community discussions

Hacker News discussion ↗

Historical count: 5 · 5 saved in sample · Historical sample

Analysis ready

Community Sentiment

Mixed

Positives

Embedding metadata into vector stores can enhance trust in RAG systems by linking outputs to authoritative sources, improving accountability and traceability.
The trust boundary framing is a valuable perspective for understanding how to manage document reliability in AI systems, emphasizing the need for robust design.

Concerns

The low barrier to entry for document poisoning attacks raises significant concerns about the security and reliability of RAG systems, indicating potential flaws in their design.
Without meticulous vetting of documents, organizations risk incorporating outdated or incorrect information, which can severely undermine the model's performance and trustworthiness.
The lack of architectural mechanisms to differentiate between trusted and untrusted documents in RAG systems poses a serious challenge for ensuring accurate outputs.