Document poisoning in RAG systems: How attackers corrupt AI's sources
aminrj.com
March 12, 2026
13 min read
🔥🔥🔥🔥🔥
55/100
Summary
Three fabricated documents were injected into a ChromaDB knowledge base, resulting in a RAG system inaccurately reporting a company's Q4 2025 revenue as $8.3M, a 47% decrease year-over-year, along with a planned workforce reduction. This process was completed in under three minutes on a MacBook Pro without GPU support or cloud services.
Key Takeaways
- Knowledge base poisoning is a significant and underestimated attack on retrieval-augmented generation (RAG) systems, allowing attackers to manipulate AI outputs by injecting fabricated documents.
- An experiment demonstrated that by adding three false documents to a ChromaDB knowledge base, an AI system incorrectly reported a company's Q4 2025 revenue as $8.3M instead of the actual $24.7M.
- The success of a poisoning attack relies on two conditions: the poisoned document must have higher cosine similarity to the query than the legitimate document, and it must cause the AI to generate the desired false output.
- The PoisonedRAG study showed a 90% success rate for attacks against large knowledge bases, indicating that even a small number of crafted documents can effectively dominate retrieval results in certain contexts.
Community Sentiment
Positives
- Embedding metadata into vector stores can enhance trust in RAG systems by linking outputs to authoritative sources, improving accountability and traceability.
- The trust boundary framing is a valuable perspective for understanding how to manage document reliability in AI systems, emphasizing the need for robust design.
Concerns
- The low barrier to entry for document poisoning attacks raises significant concerns about the security and reliability of RAG systems, indicating potential flaws in their design.
- Without meticulous vetting of documents, organizations risk incorporating outdated or incorrect information, which can severely undermine the model's performance and trustworthiness.
- The lack of architectural mechanisms to differentiate between trusted and untrusted documents in RAG systems poses a serious challenge for ensuring accurate outputs.
