Serve Markdown to AI Agents with Accept Headers
acceptmarkdown.com
August 26, 2026
1 min read
46/100
Summary
AcceptMarkdown.com promotes serving Markdown versions of web pages to AI agents through HTTP content negotiation. Clients can request a Markdown representation with the `Accept: text/markdown` header, allowing them to receive page content without navigation, scripts, styles, layout wrappers, advertisements, related-content sections, or modal overlays. The site says Markdown responses use a fraction of the bytes of full pages and can leave AI agents’ context windows focused on prose rather than DOM markup. It also says cleaner text can improve signal-to-noise for retrieval-augmented generation pipelines and that smaller responses can reduce fetching, parsing, and context-loading work before a model begins generating output. Its URL checker sends an edge request with `Accept: text/markdown` and reports what the origin returns. The checker evaluates whether a URL serves Markdown for that request, sets `Vary: Accept`, rejects unsupported content types with HTTP 406, and honors HTTP `q` quality values in Accept headers.
What the discussion said
The thread spent less time celebrating Markdown itself than arguing over whether AI agents need publishers to create a special representation at all. Supporters see a lightweight Markdown response as a practical way to cut HTML, JavaScript, styling clutter, and token spend before an agent reasons over a page. They also connect it to a cleaner, less adversarial web: structured text could give agents useful access without leaning so heavily on scraping intermediaries, proof-of-work barriers, or browser automation. The dominant skepticism is that semantic HTML already carries the web’s meaning, and an agent’s own retrieval layer should extract the relevant content rather than forcing every publisher to maintain an AI-specific format. Several readers warn that Markdown can discard useful navigation and page structure while doing little that ordinary HTML-to-text conversion cannot. Adoption is another obstacle: publishers have little incentive to implement Accept-header negotiation until major AI products demonstrably request it. The technical subthread split over whether negotiated responses create cache trouble; some argued proper Vary handling makes CDN caching workable, while others still prefer explicit format endpoints. Security concerns were blunt: a channel tailored to models could become a convenient place to serve poisoned, misleading, or resource-exhausting content to crawlers and agents.
Where opinion split
The central fight is whether publishers should expose Markdown directly to AI agents or whether agent harnesses should convert existing semantic HTML themselves. Advocates say server-provided Markdown removes expensive clutter and makes machine access cleaner; critics say it externalizes an easy parsing task, loses useful structure, and invites a separate surface for manipulation.
Community Sentiment
Positives
- Markdown-first responses could strip browser-oriented markup and slash context-token waste, making agent retrieval cheaper and more focused on the actual document.
- A standardized machine-readable representation could reduce dependence on scraping gatekeepers, browser automation, and energy-wasting anti-bot obstacles.
- Correct content-negotiation cache headers can let CDNs distinguish Markdown and HTML variants, keeping the approach technically viable rather than inherently uncacheable.
Concerns
- Clean semantic HTML already exposes meaningful structure, so creating a second AI format looks like making every publisher compensate for weak agent extraction pipelines.
- Markdown may remove navigation, labels, and surrounding context that help an agent understand where content sits and which links matter.
- Without major chatbot vendors sending these requests, publishers see no credible adoption path for a niche header-based convention.
- An AI-targeted representation gives malicious sites a sharper tool for prompt injection, misinformation, or deliberately huge and slow payloads aimed at degrading agent behavior.