We gave terabytes of CI logs to an LLM
mendral.com
February 27, 2026
11 min read
🔥🔥🔥🔥🔥
59/100
Summary
An AI agent analyzed terabytes of continuous integration (CI) logs by generating SQL queries, tracing a flaky test back to a dependency change within seconds. The agent requires comprehensive context from all historical logs, processing approximately 1.5 billion log lines and 700,000 jobs weekly.
Key Takeaways
- The agent can trace flaky tests by writing SQL queries that scan hundreds of millions of log lines, completing investigations in seconds.
- The system processes approximately 1.5 billion CI log lines and 700,000 jobs weekly, storing data in ClickHouse with a compression ratio of 35:1.
- The agent utilizes a SQL interface to construct its own queries, allowing for flexible and novel questions during debugging.
- The agent typically scans 335,000 rows across about 3 queries per investigation, with some sessions scanning up to 4.3 billion rows.
Community Sentiment
Positives
- Developing a tool that reduces log noise using a TF-IDF classifier and BERT model demonstrates innovative approaches to enhance LLM performance in log analysis.
- The integration of LLMs with SQL for exploratory data analysis shows potential for improving data insights, despite the need for human oversight.
Concerns
- LLMs often hallucinate reasons during log analysis, which can lead to misunderstandings in complex scenarios where multiple failures occur.
- The effectiveness of LLMs in generating SQL queries is questioned, as they may require substantial human input for critical business decisions.
