I prompted ChatGPT, Claude, Perplexity, and Gemini and watched my Nginx logs
surfacedby.com
April 20, 2026
9 min read
🔥🔥🔥🔥🔥
52/100
Summary
Nginx logs reveal that "AI traffic" can indicate either provider-side fetching or referral traffic. The logs provide measurable data on whether AI assistants fetch live pages or rely on previously indexed information.
Key Takeaways
- AI traffic can be categorized into two types: provider-side fetches by assistants and real clickthrough visits by humans.
- Five AI assistants used distinct user-agent signals to indicate retrieval requests, while three assistants did not announce themselves, making it difficult to differentiate their fetches from human visits.
- ChatGPT consistently fetched pages from multiple IPs, while Claude always checked the robots.txt file before fetching.
- Gemini did not make any live fetch requests during the testing period, indicating it answered from an index instead.
Community Sentiment
Positives
- Claude's adherence to rules demonstrates a commitment to ethical AI usage, which could set a precedent for responsible AI development.
- The test results indicate that Gemini may prioritize its own cached data over live fetching, suggesting a focus on efficiency in response times.
Concerns
- The unclear writing style of the article detracts from its credibility, making it difficult for readers to grasp key points about AI model performance.
- The implication that Gemini may not be fetching live data raises concerns about the accuracy and reliability of its responses.
