How we index images for RAG
kapa.ai
June 2, 2026
8 min read
🔥🔥🔥🔥🔥
58/100
Summary
Kapa.ai indexes millions of images, including screenshots and diagrams, to enhance AI assistants that answer technical questions. Images are processed in a way that they are not sent to the model at query time, optimizing their utility in the retrieval-augmented generation (RAG) pipeline.
Key Takeaways
- Kapa.ai indexes images for RAG by describing each image with a vision model at indexing time, storing the descriptions as text, and retrieving them alongside text chunks during queries.
- The use of images in technical documentation improves the quality of answers, with LLM judges preferring image-context answers by a statistically significant margin.
- Query-time multimodal approaches were found to be economically unfeasible and structurally unsuitable for technical questions, leading Kapa.ai to adopt a one-time indexing method.
- Images in documentation serve two roles: illustrative, enhancing clarity of text, and load-bearing, containing essential information that cannot be conveyed through text alone.
Community Sentiment
Positives
- Describing images at indexing time with a cheap vision model enhances retrieval outcomes, demonstrating an effective method for integrating visual data into text-based systems.
- The approach of generating text descriptions for important images allows agents to better understand visual content, significantly improving search and retrieval capabilities.
Concerns
- The non-deterministic nature of LLMs raises concerns that new models may interpret data differently, potentially revealing context that was previously overlooked and necessitating reprocessing.
