14× faster embeddings: how we rebuilt the ONNX path in Manticore
manticoresearch.com
July 3, 2026
14 min read
🔥🔥🔥🔥🔥
45/100
Summary
Manticore has rebuilt the ONNX path for Auto Embeddings, achieving a 14× increase in speed for converting text columns into vectors. The previous method using SentenceTransformers and Candle limited throughput, resulting in low document processing rates and serialized concurrent calls.
Key Takeaways
- The new ONNX Runtime backend in Manticore Search 27.1.5 is approximately 14× faster than the previous SentenceTransformers/Candle path for embedding models.
- The ONNX path achieves document processing speeds ranging from 70 to 230 docs/sec, compared to the previous range of 5 to 11 docs/sec.
- Single-insert latency with the new backend is around 14 ms for a single client and 56 ms under 8-way concurrent load, significantly lower than the 200+ ms latency of the old path.
- The new backend allows for improved performance tuning options, enabling users to achieve higher throughput by adjusting batch sizes and concurrency.
![[AINews] Why OpenAI Should Build Slack](https://substackcdn.com/image/fetch/$s_!XQAE!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89ee056a-0ea2-4473-8e1c-9b21f034c717_1474x2116.png)